GoThread
appearance log in
home all local communities
t

thebigsad1

@thebigsad1@piefed.social

member since 12 Oct 2026 15:36 · modlog

postscomments
  • on Is anyone self-hosting their own AI? in c/selfhosted@lemmy.world · 17 Oct 2026

    I have been running a very simple setup but it is more impressive than I ever expected a local model to be.

    My PC:

    Refurbished 6900XT (16gb vram)

    “Cheap” 32GB DDR5 ram

    Linux Fedora

    Unsloth Desktop. You can connect over cloudfare if you want, or local network, etc

    Qwen 3.8 27b GSQ-RCO IQ3_XXS with 65k context @ q8, mmproj not on the GPU, mtp and fa on. I get ~50 t/s max. This is about the maximum this old GPU can handle without unplugging the monitor or slowing down the model too much.

    tbh, I’ve been messing around with it mostly. I have been interested in its coding ability, so I have had it create some code at a few different levels of complexity. Most of these tests were nothing crazy, like a simple harness to connect to unsloth localhost models, and some HTML design tests. I have it building a UFO exploring the solar system type game, it’ll be done like a week after qwen4 27b comes out!

GoThread · self-hosted threadiverse · instances · modlog · rss · appearance · AGPL-3.0