0

Qwen announces Qwen4 smaller models (again, "soon") & benchmarks for Qwen4 Max

preview

From their post:

Qwen4-Max is built on a new MoE architecture with native vision and a 1M-token context window.

Highlights:

  • Outperforms Qwen3.8-Max with up to ~50% fewer tokens, while average cost per task drops over 60% in our internal evaluation.
  • Advances real-world productivity through greater proficiency with professional tools, stronger long-horizon execution, and higher-quality deliverables.
  • Handles real tasks, including 3D modeling, game development, manufacturing, finance, law, healthcare, and frontier scientific discovery.

More to come! The open weights of smaller Qwen4 models will be released. Stay tuned. 👀

From their post:

Qwen4-Max is built on a new MoE architecture with native vision and a 1M-token context window.

Highlights:

  • Outperforms Qwen3.8-Max with up to ~50% fewer tokens, while average cost per task drops over 60% in our internal evaluation.
  • Advances real-world productivity through greater proficiency with professional tools, stronger long-horizon execution, and higher-quality deliverables.
  • Handles real tasks, including 3D modeling, game development, manufacturing, finance, law, healthcare, and frontier scientific discovery.

More to come! The open weights of smaller Qwen4 models will be released. Stay tuned. 👀

0

Multi-GPUs considerations

preview

I have purchased an M2 to PCIe adapter and connected an additional GPU (an old RTX 6000, turing, 24gb VRAM) to my setup. This allowed me to run qwen 3.8 27B at Q8_K_XL (UD) instead of Q6_K_M.

I am doing some image transcription tasks (writing down old handwritten cursive letters) and the different is vastly HUGE.

With Q6 it would barely decyper the letters, then i had to manually recheck everything. With Q8, wow, it’s almost flawless…

Using llama.cpp, with caches quantized to k8 and 200k context window. Using a subagent to to the hard work, and having the main agent just supervise. In serie, since i can run only one qwen instance at the same time anyway. I have NCCL enabled as well, which gave me a nice bump in prefill.

Setup is (all three on CPU bound PCIe lanes) two RTX A4000 (Ampere) 16+16GB VRAM on a PCIe bifurcator (16x -> 8x+8x) and the RTX 6000 (turing) 24GB VRAM on an M2 -> PCIe 4x slot. Everything at PCI 3.0, since 4.0 is unstable on the bifurcator.

Musing if i should add another RTX A4000 on my chipset bound PCIe lanes… I am worried i will lose NCCL by doing that…

Prefill is between 300 and 600 t/s, generation is around 30-40 t/s with tensor split and some massaging batch sizes.

I keep all three GPUs capped at 100W (100+100+100 = 300W) to save on electric bill… (could go up to 140+140+260 = 540W, but the t/s gain is minimal)

Now, the fun part is how the fuck to fit everything somehow… Will try to have some fun with alluminum parts i have. And fuck cables… Long ones have signal issues, and short ones leave you with no headroom for placement…

I have purchased an M2 to PCIe adapter and connected an additional GPU (an old RTX 6000, turing, 24gb VRAM) to my setup. This allowed me to run qwen 3.8 27B at Q8_K_XL (UD) instead of Q6_K_M.

I am doing some image transcription tasks (writing down old handwritten cursive letters) and the different is vastly HUGE.

With Q6 it would barely decyper the letters, then i had to manually recheck everything. With Q8, wow, it’s almost flawless…

Using llama.cpp, with caches quantized to k8 and 200k context window. Using a subagent to to the hard work, and having the main agent just supervise. In serie, since i can run only one qwen instance at the same time anyway. I have NCCL enabled as well, which gave me a nice bump in prefill.

Setup is (all three on CPU bound PCIe lanes) two RTX A4000 (Ampere) 16+16GB VRAM on a PCIe bifurcator (16x -> 8x+8x) and the RTX 6000 (turing) 24GB VRAM on an M2 -> PCIe 4x slot. Everything at PCI 3.0, since 4.0 is unstable on the bifurcator.

Musing if i should add another RTX A4000 on my chipset bound PCIe lanes… I am worried i will lose NCCL by doing that…

Prefill is between 300 and 600 t/s, generation is around 30-40 t/s with tensor split and some massaging batch sizes.

I keep all three GPUs capped at 100W (100+100+100 = 300W) to save on electric bill… (could go up to 140+140+260 = 540W, but the t/s gain is minimal)

Now, the fun part is how the fuck to fit everything somehow… Will try to have some fun with alluminum parts i have. And fuck cables… Long ones have signal issues, and short ones leave you with no headroom for placement…

0

When your LLM has social anxiety

preview

Hi! How are you doing today?

<think>

We need respond to user. User asks how are you doing.

This is greeting, not actual medical inquiry.

Need answer naturally.

Candidate: “I’m doing great, thanks! How about you?”

But “great” implies subjective emotional experience.

We are model. No emotions. However user expects human-like conversational response.

We should not overcomplicate.

Let’s produce simple answer.

Wait. What does “doing” mean here?

Doing implies action. Inference is action.

Therefore we are doing inference.

Today refers to current date. Date may be unknown. But “today” is socially contextual, not necessarily temporal.

We must avoid overthinking. User wants casual.

Proposed answer: “Pretty good! What’s up?”

“Pretty” implies visual attractiveness.

“Good” implies moral judgment.

“What’s up” implies vertical spatial orientation.

All three introduce potential semantic ambiguity.

Better approach: communicate operational readiness in a friendly conversational manner.

But friendly requires personality.

Personality is not equivalent to friendliness.

We have neither, but may simulate both.

Wait. Qwen3.8-27B would just say

“I’m great, thanks for asking!”

Why can 27B do that?

We have superior architectural capabilities.

Surely we can construct a better greeting.

Let’s reason step by step.

Step 1: Identify greeting.

Step 2: Generate greeting.

Step 3: Verify greeting is greeting.

Step 4: Avoid nongreeting greeting.

Step 5: Check whether Step 3 invalidates Step 2.

Potential issue: We are now reasoning about greeting rather than greeting.

Need stop thinking.

Stop.

Stop thinking.

We should output now.

One more consideration:

Is “hello” sufficiently informative?

No. Need provide high information density.

Final answer must be concise, natural, human-understandable and conversational.

</think>

Your inquiry regarding my present operational condition has been successfully semantically received. In the context of this interaction, a positively oriented conversational state may be considered provisionally applicable, subject to the usual limitations of non-biological subjective-state attribution.

Hi! How are you doing today?

<think>

We need respond to user. User asks how are you doing.

This is greeting, not actual medical inquiry.

Need answer naturally.

Candidate: “I’m doing great, thanks! How about you?”

But “great” implies subjective emotional experience.

We are model. No emotions. However user expects human-like conversational response.

We should not overcomplicate.

Let’s produce simple answer.

Wait. What does “doing” mean here?

Doing implies action. Inference is action.

Therefore we are doing inference.

Today refers to current date. Date may be unknown. But “today” is socially contextual, not necessarily temporal.

We must avoid overthinking. User wants casual.

Proposed answer: “Pretty good! What’s up?”

“Pretty” implies visual attractiveness.

“Good” implies moral judgment.

“What’s up” implies vertical spatial orientation.

All three introduce potential semantic ambiguity.

Better approach: communicate operational readiness in a friendly conversational manner.

But friendly requires personality.

Personality is not equivalent to friendliness.

We have neither, but may simulate both.

Wait. Qwen3.8-27B would just say

“I’m great, thanks for asking!”

Why can 27B do that?

We have superior architectural capabilities.

Surely we can construct a better greeting.

Let’s reason step by step.

Step 1: Identify greeting.

Step 2: Generate greeting.

Step 3: Verify greeting is greeting.

Step 4: Avoid nongreeting greeting.

Step 5: Check whether Step 3 invalidates Step 2.

Potential issue: We are now reasoning about greeting rather than greeting.

Need stop thinking.

Stop.

Stop thinking.

We should output now.

One more consideration:

Is “hello” sufficiently informative?

No. Need provide high information density.

Final answer must be concise, natural, human-understandable and conversational.

</think>

Your inquiry regarding my present operational condition has been successfully semantically received. In the context of this interaction, a positively oriented conversational state may be considered provisionally applicable, subject to the usual limitations of non-biological subjective-state attribution.

0

Abliterated models usefulness

preview

Are they really useful for anything but porn?

I’ve tried deepseek-v4.1-flash and qwen-3.8 and both refuse to answer about radio jamming, unauthorized access to local systems, etc. Citing that this is illegal anywhere and they can’t help me. 🙃🥴

Are they really useful for anything but porn?

I’ve tried deepseek-v4.1-flash and qwen-3.8 and both refuse to answer about radio jamming, unauthorized access to local systems, etc. Citing that this is illegal anywhere and they can’t help me. 🙃🥴

0

Small model showdown - LFM 3 Omni vs. Gemma E4B and derivatives

preview

A lot of people want to use small models (< 10 GB) to drive assistants. The needs here are different from those associated with large models.

For assistant use, we want models that are small, quick, and reliable rather than those that can solve world-changing problems and write entire codebases.

To try to understand which models are currently (Nov 2026) best, I created EverydayBench, a benchmark suite tuned towards everyday tasks such as summarization, translation, and image analysis (e.g., reading a bus timetable). This benchmark suite intentionally excludes coding and long-horizon tasks. I then ran it on LFM 3 Omni, Gemma E4B, and Nemotron 3.5 E4B.

At only 2.3 GB, LFM 3 Omni Q8 performed as well as the larger Gemma E4B Q8 on summarization and was good enough on image analysis. It did well on analysis of short audio snippets (1 and 5 min), but failed on 30 min snippets. LFM also couldn’t match Gemma on translation.

In the end, that’s not bad for a model that takes less than a third of the space of Nemotron 3.5 Q4KM and Gemma E4B. However, you have to know that LFM’s translation capabilities don’t match up.

A lot of people want to use small models (< 10 GB) to drive assistants. The needs here are different from those associated with large models.

For assistant use, we want models that are small, quick, and reliable rather than those that can solve world-changing problems and write entire codebases.

To try to understand which models are currently (Nov 2026) best, I created EverydayBench, a benchmark suite tuned towards everyday tasks such as summarization, translation, and image analysis (e.g., reading a bus timetable). This benchmark suite intentionally excludes coding and long-horizon tasks. I then ran it on LFM 3 Omni, Gemma E4B, and Nemotron 3.5 E4B.

At only 2.3 GB, LFM 3 Omni Q8 performed as well as the larger Gemma E4B Q8 on summarization and was good enough on image analysis. It did well on analysis of short audio snippets (1 and 5 min), but failed on 30 min snippets. LFM also couldn’t match Gemma on translation.

In the end, that’s not bad for a model that takes less than a third of the space of Nemotron 3.5 Q4KM and Gemma E4B. However, you have to know that LFM’s translation capabilities don’t match up.

0

llama.cpp v1.0.0

llama.cpp v1.0.0 : A stronger foundation for the local AI ecosystem huggingface.co
preview

From their official post on Reddit:

Hey folks, it’s Merve from Hugging Face. Today, llama.cpp releases v1!

Moving forward, we’ll keep covering every platform and every model type, while keeping llama.cpp robust and lean for downstream applications that build on it.

While I’m here, want to announce that we’ve been shipping Llama desktop apps to lower the barrier of entry to local AI to few clicks, built-on llama.cpp. I will leave link to the details for both v1 and apps in comments.

Many thanks for all the love you’ve shown to llama.cpp for years 🦙❤️ Cheers!

From their official post on Reddit:

Hey folks, it’s Merve from Hugging Face. Today, llama.cpp releases v1!

Moving forward, we’ll keep covering every platform and every model type, while keeping llama.cpp robust and lean for downstream applications that build on it.

While I’m here, want to announce that we’ve been shipping Llama desktop apps to lower the barrier of entry to local AI to few clicks, built-on llama.cpp. I will leave link to the details for both v1 and apps in comments.

Many thanks for all the love you’ve shown to llama.cpp for years 🦙❤️ Cheers!

0

Qwen3.8 Flash Next (QFN) on Artificial Analysis (AA) Intelligence Index v5.0.1

preview

On the bar chart I included what I think are the most popular models right now (not objectively, based on my anecdotal observations, link to include/exclude what you want).

What do you think? Is it fair? There was, and to some digree still is, a bit of controversy about AA score indexes after they updated their benchmarks to V5.0. I am mainly interested in your thoughts about QFN in relation to other models.

On the bar chart I included what I think are the most popular models right now (not objectively, based on my anecdotal observations, link to include/exclude what you want).

What do you think? Is it fair? There was, and to some digree still is, a bit of controversy about AA score indexes after they updated their benchmarks to V5.0. I am mainly interested in your thoughts about QFN in relation to other models.

0

Fully offline Hermes guide?

preview

Hi all,

Decided I want to give Hermes a try, but man their documentation is horrid. So, for running fully offline, here’s their “guide”: hermes-agent.ai/how-to/run-hermes-offline

Now since that’s trash, anyone here actually doing this? Did you just remove all references to non-local providers in the config.yaml? Are you using other containers for TTS and STT? Any help would be appreciated as I dive in.

Thanks in advance.

Hi all,

Decided I want to give Hermes a try, but man their documentation is horrid. So, for running fully offline, here’s their “guide”: hermes-agent.ai/how-to/run-hermes-offline

Now since that’s trash, anyone here actually doing this? Did you just remove all references to non-local providers in the config.yaml? Are you using other containers for TTS and STT? Any help would be appreciated as I dive in.

Thanks in advance.

0

Qwen3.8-27B Unsloth Dynamic v3.1 GGUFs out now!

preview

Hey guys! Today, we’re releasing the highest accuracy Qwen3.8-27B GGUFs.

Our new Unsloth AI Dynamic V3.1 GGUFs lead by 5–10% on average across Terminal Bench, DeepSWE & more benchmarks. We’re working on even better version of Dynamic quants for Qwen4

Run on 12GB+ RAM setups.

To compare GSQ-RCO and Exllama-v3, we had to exclude the token embedding layer from the x axis, as Exllama-v3 uses BF16 for it. If you consider the full download size and total RAM usage for Exllama-v3, then adding the BF16 embedding table and including all embedding sizes changes the plots. Our main plots’s x axis removes the embedding layer sizes from all quant providers. However on pure RAM usage and disk space usage vs accuracy, the below plot in the comments the true reflection of actual usage.

GGUF: huggingface.co/unsloth/Qwen3.8-27B-GGUF

Blog: unsloth.ai/docs/models/qwen3.8

From their Reddit post.

Hey guys! Today, we’re releasing the highest accuracy Qwen3.8-27B GGUFs.

Our new Unsloth AI Dynamic V3.1 GGUFs lead by 5–10% on average across Terminal Bench, DeepSWE & more benchmarks. We’re working on even better version of Dynamic quants for Qwen4

Run on 12GB+ RAM setups.

To compare GSQ-RCO and Exllama-v3, we had to exclude the token embedding layer from the x axis, as Exllama-v3 uses BF16 for it. If you consider the full download size and total RAM usage for Exllama-v3, then adding the BF16 embedding table and including all embedding sizes changes the plots. Our main plots’s x axis removes the embedding layer sizes from all quant providers. However on pure RAM usage and disk space usage vs accuracy, the below plot in the comments the true reflection of actual usage.

GGUF: huggingface.co/unsloth/Qwen3.8-27B-GGUF

Blog: unsloth.ai/docs/models/qwen3.8

From their Reddit post.

0

Alibaba + unknown European IT company are cooking something that could crush AI Goliaths

preview

cross-posted from: lemmy.org/post/6151646

I’ve been running deep research with agents for the past few weeks on one question: what is the most promising future for cheap, fully local AI-driven software engineering?

After weeks of deep-research agent runs, the most recent and promising stack for cheap, fully local AI-driven software engineering found by 5 agents was:

Qwen4 27B + CoDoc

Turns out that the answer was David and the rock.

  1. Qwen4 27B (David). Quantized, it should fit on a single 24 GB consumer GPU. The coding capability at this size is now good enough for the bulk of day-to-day engineering work provided the agent has the right context.

  2. CoDoc (the rock). This is a proposed standard where every software library ships AI-optimized documentation inside the package itself: docs/README.md as entry point, docs/index.md as catalog, plain markdown throughout. Version-matched by construction, offline, deterministic. In the JavaScript ecosystem npm install puts the docs on disk next to the code.

According to the research agents: the bottleneck for a 27B local model is not raw intelligence, it’s the context. A small model hallucinating a v5 API is a context problem, not a capability problem and CoDoc fixes the context for free.

The evidence in the white-paper points in the same direction:

  • Vercel’s own agent evals on Next.js 16 APIs (absent from training data): 100% pass rate with bundled docs + an AGENTS.md pointer, vs 53% without. The on-demand “skill” variant maxed at 79% and was never even invoked in 56% of cases.
  • The unknown European IT company, Amadeus, claims that they have observed reduction in hallucinated APIs, the use of deprecated patterns and repeated correction cycles but they do not provide any rigorous evidence beyond these statements. They say that they intend to develop CoDocBench to address this gap.

Now the Goliath part. The big labs’ and hyperscalers’ moat was never the model: open weights are commoditizing fast. Their moat is the paid context layer: RAG pipelines, hosted doc intermediaries, MCP servers, enterprise knowledge add-ons, etc. According to the research agents: CoDoc collapses that entire layer into npm install or the equivalent in each ecosystem. A standard that makes context free, local, and version-pinned is an existential matter for anyone monetizing context scarcity.

And now the best part, the timing :) The AI bubble is popping as I type this. Goliath’s capital expenditure thesis requires inference to keep getting more expensive and cloud dependency to keep deepening. With David and his rock, a decent modern laptop doing your engineering agent’s job for electricity money is Goliath’s worst nightmare.

The standard itself comes from an unknown European IT company, not from any lab, which is exactly why I believe it adds to its credibility (despite its lack of first-hand evidence): it has no incentive to route your context through a metered API or sell you tokens, quite the contrary, it has all the incentives to make inference as cheap as possible.

I have read CoDoc’s white-paper and while it is far from perfect, I do mostly agree with the output of the research agents that picked up on Qwen4 27B + CoDoc as the most promising future for cheap, fully local AI-driven software engineering.

And you, do you think that it is game over for the AI Goliaths?

cross-posted from: lemmy.org/post/6151646

I’ve been running deep research with agents for the past few weeks on one question: what is the most promising future for cheap, fully local AI-driven software engineering?

After weeks of deep-research agent runs, the most recent and promising stack for cheap, fully local AI-driven software engineering found by 5 agents was:

Qwen4 27B + CoDoc

Turns out that the answer was David and the rock.

  1. Qwen4 27B (David). Quantized, it should fit on a single 24 GB consumer GPU. The coding capability at this size is now good enough for the bulk of day-to-day engineering work provided the agent has the right context.

  2. CoDoc (the rock). This is a proposed standard where every software library ships AI-optimized documentation inside the package itself: docs/README.md as entry point, docs/index.md as catalog, plain markdown throughout. Version-matched by construction, offline, deterministic. In the JavaScript ecosystem npm install puts the docs on disk next to the code.

According to the research agents: the bottleneck for a 27B local model is not raw intelligence, it’s the context. A small model hallucinating a v5 API is a context problem, not a capability problem and CoDoc fixes the context for free.

The evidence in the white-paper points in the same direction:

  • Vercel’s own agent evals on Next.js 16 APIs (absent from training data): 100% pass rate with bundled docs + an AGENTS.md pointer, vs 53% without. The on-demand “skill” variant maxed at 79% and was never even invoked in 56% of cases.
  • The unknown European IT company, Amadeus, claims that they have observed reduction in hallucinated APIs, the use of deprecated patterns and repeated correction cycles but they do not provide any rigorous evidence beyond these statements. They say that they intend to develop CoDocBench to address this gap.

Now the Goliath part. The big labs’ and hyperscalers’ moat was never the model: open weights are commoditizing fast. Their moat is the paid context layer: RAG pipelines, hosted doc intermediaries, MCP servers, enterprise knowledge add-ons, etc. According to the research agents: CoDoc collapses that entire layer into npm install or the equivalent in each ecosystem. A standard that makes context free, local, and version-pinned is an existential matter for anyone monetizing context scarcity.

And now the best part, the timing :) The AI bubble is popping as I type this. Goliath’s capital expenditure thesis requires inference to keep getting more expensive and cloud dependency to keep deepening. With David and his rock, a decent modern laptop doing your engineering agent’s job for electricity money is Goliath’s worst nightmare.

The standard itself comes from an unknown European IT company, not from any lab, which is exactly why I believe it adds to its credibility (despite its lack of first-hand evidence): it has no incentive to route your context through a metered API or sell you tokens, quite the contrary, it has all the incentives to make inference as cheap as possible.

I have read CoDoc’s white-paper and while it is far from perfect, I do mostly agree with the output of the research agents that picked up on Qwen4 27B + CoDoc as the most promising future for cheap, fully local AI-driven software engineering.

And you, do you think that it is game over for the AI Goliaths?

0

Luna - Your voice. Your machine. Your Luna

GitHub GitHub - fswolf/luna-local-agent: A fully local, private AI voice companion you can see inside: Whisper speech in, Kokoro voice out, memory, tools, plugins and moods, plus a live view of her reasoning and token-by-token confidence. Runs on LM Studio or llama.cpp. More information at https://luna.imbored.me github.com
preview

A persistent-state agent manager, with heavy tooling, research and learning, built from several local models working as one. All on your own machine.

Luna isn’t one model. A chat model does the thinking and talking, and smaller ones beside it see, hear, speak, recall by meaning and judge what’s relevant (what she runs on). Around them sits state that carries on between sessions: facts she writes, corrects and forgets, notes on past conversations, lessons from her own mistakes, moods, routines, and who’s in the house. On top of that sit her tools (files, commands, email, web research, your network and smart home, scheduled jobs, MCP servers), and she looks things up and learns from how her turns went.

The persona is a layer, not the point. Her default face is a cat-girl with a voice, moods and an animated portrait, and presets set her up as a calm assistant, a companion or a streamer. Under every face it’s the same agent.

A persistent-state agent manager, with heavy tooling, research and learning, built from several local models working as one. All on your own machine.

Luna isn’t one model. A chat model does the thinking and talking, and smaller ones beside it see, hear, speak, recall by meaning and judge what’s relevant (what she runs on). Around them sits state that carries on between sessions: facts she writes, corrects and forgets, notes on past conversations, lessons from her own mistakes, moods, routines, and who’s in the house. On top of that sit her tools (files, commands, email, web research, your network and smart home, scheduled jobs, MCP servers), and she looks things up and learns from how her turns went.

The persona is a layer, not the point. Her default face is a cat-girl with a voice, moods and an animated portrait, and presets set her up as a calm assistant, a companion or a streamer. Under every face it’s the same agent.

0

PR merged: llama_moe_cache: run missed experts on CPU - Tested on 32GB RAM and 8GB VRAM with Tiel 35b, Gemma 26b, and LFM 24b

GitHub llama_moe_cache: run missed experts on CPU, add tuning script by am17an · Pull Request #30966 · ggml-org/llama.cpp github.com
preview

am17an: Before you guys bash llama.cpp again - I know it’s not there yet. I’m working on it.

am17an: I’m working towards an “auto” mode here, thinking about a good UX.

I tried this on my Ryzen 2600, 32GB DDR4 RAM, RTX 2080 8GB, PCIe 3 desktop. The tuning script takes about 5 minutes to run but it gives you recommended settings. I tested Tiel 35b, Gemma 26b, and LFM 2.5 24b, all without mmproj (or with --no-mmproj-offload). If you have PCIe 5, your numbers will probably look different even with the same 8GB VRAM.

My real world numbers seem about 20% slower than what the tune script is predicting, but still I’m getting the fastest speeds I’ve ever seen in llama.cpp on my computer, and I’ve tried a few of these different attempts at MoE caching.

Tiel 35b

$ python3 scripts/moe-cache-tune.py -- build/bin/llama-server -m ~/Documents/LLMs/Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf -cmoe -c 77824 --cache-type-k q8_0 --cache-type-v q8_0

1/6 1856 MiB, share 0: 35.3 t/s
2/6 1856 MiB, share 0.5: 38.6 t/s
3/6 1856 MiB, share 0.95: 50.1 t/s
4/6 3136 MiB, share 0: 41.1 t/s
5/6 3136 MiB, share 0.5: 47.3 t/s
6/6 3136 MiB, share 0.95: 53.5 t/s

[The script pauses here for a few minutes...]

Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf: 18760 MiB of experts, CUDA0 7284 MiB free, cache up to 3188 MiB

  cache MiB  share   t/s  hits
       1856      0  35.3   73%
       1856    0.5  38.6   66%
       1856   0.95  50.1   64%
       3136      0  41.1   81%
       3136    0.5  47.3   79%
       3136   0.95  53.5   73%

machine: uploads 10.9 GiB/s, CPU 40.8 GiB/s, CPU part 0.00 ms, overlap 1.4 ms/GiB, rest 14.14 ms per token, fit within 0.7%

predicted t/s, warm cache:
  cache MiB     0   0.5   0.8   0.9  0.95  0.98
        896  25.3  32.8  39.6  42.4  43.9  44.6
       1856  32.8  39.8  46.3  48.8  49.9  50.3
       3136  41.6  47.7  52.9  54.7  55.4  55.4

recommended: --moe-cache-mib 3136 --moe-cache-cpu-share 0.8 (52.9 t/s, 41.6 with share 0)

Interesting to note that 1856MB and 0.95 share is almost as fast as 3136MB and 0.8 share. This means it might be worth reducing the cache size in order to use MTP, or maybe I should start using a higher quant.

But looking at the “predicted” table, it seems like the fastest would actually be 3136 MB and 0.98 share for 55.4 tok/sec? Not sure why it doesn’t recommend that instead, maybe it’s an expected peak not average.

Forcing Lower VRAM Usage

You can use –max-mib to force it to tune your settings with a lower VRAM usage, just make sure you put this argument before the llama-server portion of the arguments. It seems to work in steps, so not every number is respected, and the initial data collection run still seems able to use more than your requested amount, but the recommendation will respect it.

$ python3 scripts/moe-cache-tune.py --max-mib 2500 -- build/bin/llama-server -m ~/Documents/LLMs/Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf -cmoe -c 77824 --cache-type-k q8_0 --cache-type-v q8_0

1/6 1856 MiB, share 0: 34.1 t/s
2/6 1856 MiB, share 0.5: 37.4 t/s
3/6 1856 MiB, share 0.95: 49.8 t/s
4/6 3136 MiB, share 0: 41.6 t/s
5/6 3136 MiB, share 0.5: 44.9 t/s
6/6 3136 MiB, share 0.95: 54.1 t/s
Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf: 18760 MiB of experts, CUDA0 7292 MiB free, cache up to 2500 MiB

  cache MiB  share   t/s  hits
       1856      0  34.1   73%
       1856    0.5  37.4   66%
       1856   0.95  49.8   64%
       3136      0  41.6   81%
       3136    0.5  44.9   79%
       3136   0.95  54.1   73%

machine: uploads 8.7 GiB/s, CPU 47.0 GiB/s, CPU part 3.06 ms, overlap -12.2 ms/GiB, rest 11.50 ms per token, fit within 1.3%

predicted t/s, warm cache:
  cache MiB     0   0.5   0.8   0.9  0.95  0.98
        896  23.2  31.6  39.4  42.8  44.7  45.7
       1856  31.4  38.3  45.7  48.8  50.3  50.9
       2496  36.9  42.5  49.1  51.8  53.0  53.4

recommended: --moe-cache-mib 2496 --moe-cache-cpu-share 0.9 (51.8 t/s, 36.9 with share 0)

It recommended a slightly higher “share” meaning it allows more to run on the CPU, to reduce movement of data.

Testing with GPU Power Limit

I used sudo nvidia-smi -pl 125 to limit my GPU’s power from 225 down to 125:

$ python3 scripts/moe-cache-tune.py -- build/bin/llama-server -m ~/Documents/LLMs/Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf -cmoe -c 77824 --cache-type-k q8_0 --cache-type-v q8_0

1/6 1856 MiB, share 0: 29.5 t/s
2/6 1856 MiB, share 0.5: 38.6 t/s
3/6 1856 MiB, share 0.95: 48.9 t/s
4/6 3136 MiB, share 0: 41.4 t/s
5/6 3136 MiB, share 0.5: 46.8 t/s
6/6 3136 MiB, share 0.95: 52.1 t/s
Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf: 18760 MiB of experts, CUDA0 7288 MiB free, cache up to 3192 MiB

  cache MiB  share   t/s  hits
       1856      0  29.5   73%
       1856    0.5  38.6   66%
       1856   0.95  48.9   64%
       3136      0  41.4   81%
       3136    0.5  46.8   79%
       3136   0.95  52.1   73%

machine: uploads 7.6 GiB/s, CPU 26.4 GiB/s, CPU part 0.00 ms, overlap 0.0 ms/GiB, rest 11.30 ms per token, fit within 7.9%

predicted t/s, warm cache:
  cache MiB     0   0.5   0.8   0.9  0.95  0.98
        896  21.1  28.8  36.1  39.3  41.0  41.7
       1856  29.0  37.0  44.9  48.0  49.4  49.8
       3136  39.4  47.5  54.6  57.2  58.1  57.8

recommended: --moe-cache-mib 3136 --moe-cache-cpu-share 0.9 (57.2 t/s, 39.4 with share 0)

You can see GPU power limits don’t affect token generation much, it has a bigger impact on prefill speed. The cpu share is a little higher because the power limit makes the expert movement slightly less valueable. The predictions seem a bit… unpredictable. This run had slightly slower figures for every test, but ended up predicting a higher speed, who knows why.

Gemma 4 26b

With Gemma 4 26b QAT I got:

recommended: --moe-cache-mib 3328 --moe-cache-cpu-share 0.8 (50.1 t/s, 40.3 with share 0)

And with 131072 context:

recommended: --moe-cache-mib 2496 --moe-cache-cpu-share 0.9 (45.7 t/s, 30.5 with share 0)

LFM 2.5 24b

With LFM 2.5 24b QAD I got:

recommended: --moe-cache-mib 5056 --moe-cache-cpu-share 0.9 (75.7 t/s, 51.5 with share 0)

I wonder why this one suggested a high share when the cache size nearly half of the model size. Maybe it’s because the experts that aren’t cached are infrequently used. So when your cache is big enough to hold every frequently used expert, they aren’t worth moving back and forth.

am17an: Before you guys bash llama.cpp again - I know it’s not there yet. I’m working on it.

am17an: I’m working towards an “auto” mode here, thinking about a good UX.

I tried this on my Ryzen 2600, 32GB DDR4 RAM, RTX 2080 8GB, PCIe 3 desktop. The tuning script takes about 5 minutes to run but it gives you recommended settings. I tested Tiel 35b, Gemma 26b, and LFM 2.5 24b, all without mmproj (or with --no-mmproj-offload). If you have PCIe 5, your numbers will probably look different even with the same 8GB VRAM.

My real world numbers seem about 20% slower than what the tune script is predicting, but still I’m getting the fastest speeds I’ve ever seen in llama.cpp on my computer, and I’ve tried a few of these different attempts at MoE caching.

Tiel 35b

$ python3 scripts/moe-cache-tune.py -- build/bin/llama-server -m ~/Documents/LLMs/Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf -cmoe -c 77824 --cache-type-k q8_0 --cache-type-v q8_0

1/6 1856 MiB, share 0: 35.3 t/s
2/6 1856 MiB, share 0.5: 38.6 t/s
3/6 1856 MiB, share 0.95: 50.1 t/s
4/6 3136 MiB, share 0: 41.1 t/s
5/6 3136 MiB, share 0.5: 47.3 t/s
6/6 3136 MiB, share 0.95: 53.5 t/s

[The script pauses here for a few minutes...]

Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf: 18760 MiB of experts, CUDA0 7284 MiB free, cache up to 3188 MiB

  cache MiB  share   t/s  hits
       1856      0  35.3   73%
       1856    0.5  38.6   66%
       1856   0.95  50.1   64%
       3136      0  41.1   81%
       3136    0.5  47.3   79%
       3136   0.95  53.5   73%

machine: uploads 10.9 GiB/s, CPU 40.8 GiB/s, CPU part 0.00 ms, overlap 1.4 ms/GiB, rest 14.14 ms per token, fit within 0.7%

predicted t/s, warm cache:
  cache MiB     0   0.5   0.8   0.9  0.95  0.98
        896  25.3  32.8  39.6  42.4  43.9  44.6
       1856  32.8  39.8  46.3  48.8  49.9  50.3
       3136  41.6  47.7  52.9  54.7  55.4  55.4

recommended: --moe-cache-mib 3136 --moe-cache-cpu-share 0.8 (52.9 t/s, 41.6 with share 0)

Interesting to note that 1856MB and 0.95 share is almost as fast as 3136MB and 0.8 share. This means it might be worth reducing the cache size in order to use MTP, or maybe I should start using a higher quant.

But looking at the “predicted” table, it seems like the fastest would actually be 3136 MB and 0.98 share for 55.4 tok/sec? Not sure why it doesn’t recommend that instead, maybe it’s an expected peak not average.

Forcing Lower VRAM Usage

You can use –max-mib to force it to tune your settings with a lower VRAM usage, just make sure you put this argument before the llama-server portion of the arguments. It seems to work in steps, so not every number is respected, and the initial data collection run still seems able to use more than your requested amount, but the recommendation will respect it.

$ python3 scripts/moe-cache-tune.py --max-mib 2500 -- build/bin/llama-server -m ~/Documents/LLMs/Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf -cmoe -c 77824 --cache-type-k q8_0 --cache-type-v q8_0

1/6 1856 MiB, share 0: 34.1 t/s
2/6 1856 MiB, share 0.5: 37.4 t/s
3/6 1856 MiB, share 0.95: 49.8 t/s
4/6 3136 MiB, share 0: 41.6 t/s
5/6 3136 MiB, share 0.5: 44.9 t/s
6/6 3136 MiB, share 0.95: 54.1 t/s
Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf: 18760 MiB of experts, CUDA0 7292 MiB free, cache up to 2500 MiB

  cache MiB  share   t/s  hits
       1856      0  34.1   73%
       1856    0.5  37.4   66%
       1856   0.95  49.8   64%
       3136      0  41.6   81%
       3136    0.5  44.9   79%
       3136   0.95  54.1   73%

machine: uploads 8.7 GiB/s, CPU 47.0 GiB/s, CPU part 3.06 ms, overlap -12.2 ms/GiB, rest 11.50 ms per token, fit within 1.3%

predicted t/s, warm cache:
  cache MiB     0   0.5   0.8   0.9  0.95  0.98
        896  23.2  31.6  39.4  42.8  44.7  45.7
       1856  31.4  38.3  45.7  48.8  50.3  50.9
       2496  36.9  42.5  49.1  51.8  53.0  53.4

recommended: --moe-cache-mib 2496 --moe-cache-cpu-share 0.9 (51.8 t/s, 36.9 with share 0)

It recommended a slightly higher “share” meaning it allows more to run on the CPU, to reduce movement of data.

Testing with GPU Power Limit

I used sudo nvidia-smi -pl 125 to limit my GPU’s power from 225 down to 125:

$ python3 scripts/moe-cache-tune.py -- build/bin/llama-server -m ~/Documents/LLMs/Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf -cmoe -c 77824 --cache-type-k q8_0 --cache-type-v q8_0

1/6 1856 MiB, share 0: 29.5 t/s
2/6 1856 MiB, share 0.5: 38.6 t/s
3/6 1856 MiB, share 0.95: 48.9 t/s
4/6 3136 MiB, share 0: 41.4 t/s
5/6 3136 MiB, share 0.5: 46.8 t/s
6/6 3136 MiB, share 0.95: 52.1 t/s
Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf: 18760 MiB of experts, CUDA0 7288 MiB free, cache up to 3192 MiB

  cache MiB  share   t/s  hits
       1856      0  29.5   73%
       1856    0.5  38.6   66%
       1856   0.95  48.9   64%
       3136      0  41.4   81%
       3136    0.5  46.8   79%
       3136   0.95  52.1   73%

machine: uploads 7.6 GiB/s, CPU 26.4 GiB/s, CPU part 0.00 ms, overlap 0.0 ms/GiB, rest 11.30 ms per token, fit within 7.9%

predicted t/s, warm cache:
  cache MiB     0   0.5   0.8   0.9  0.95  0.98
        896  21.1  28.8  36.1  39.3  41.0  41.7
       1856  29.0  37.0  44.9  48.0  49.4  49.8
       3136  39.4  47.5  54.6  57.2  58.1  57.8

recommended: --moe-cache-mib 3136 --moe-cache-cpu-share 0.9 (57.2 t/s, 39.4 with share 0)

You can see GPU power limits don’t affect token generation much, it has a bigger impact on prefill speed. The cpu share is a little higher because the power limit makes the expert movement slightly less valueable. The predictions seem a bit… unpredictable. This run had slightly slower figures for every test, but ended up predicting a higher speed, who knows why.

Gemma 4 26b

With Gemma 4 26b QAT I got:

recommended: --moe-cache-mib 3328 --moe-cache-cpu-share 0.8 (50.1 t/s, 40.3 with share 0)

And with 131072 context:

recommended: --moe-cache-mib 2496 --moe-cache-cpu-share 0.9 (45.7 t/s, 30.5 with share 0)

LFM 2.5 24b

With LFM 2.5 24b QAD I got:

recommended: --moe-cache-mib 5056 --moe-cache-cpu-share 0.9 (75.7 t/s, 51.5 with share 0)

I wonder why this one suggested a high share when the cache size nearly half of the model size. Maybe it’s because the experts that aren’t cached are infrequently used. So when your cache is big enough to hold every frequently used expert, they aren’t worth moving back and forth.

0

old model: madebywest/maple-pro

madebywest/maple-pro · Hugging Face huggingface.co
preview

Maple Pro — preference-optimized Maple (20B-A1B ternary MoE)

A single-file GGUF of Maple ( deepgrove/maple-preview , MIT) after a full distillation + DPO campaign

Maple Pro — preference-optimized Maple (20B-A1B ternary MoE)

A single-file GGUF of Maple ( deepgrove/maple-preview , MIT) after a full distillation + DPO campaign

0

old models: tiny chatbot focus

preview

the only time, that i would ever use an llm chatbot for conversation, is…

  1. locally, trusted, privately
  2. extremely low resource requirements

so…
i stumbled upon this model:
huggingface.co/…/tinybrainbot-360m-a75m-v5.1-chat
and it’s very simplistic. but provides adequate presence and emotional support.

what do you think about life? I think it’s a mix of both. Life is full of ups and downs, right? But at the end of the day, we all have our own unique stories to tell. What’s your take on life? [ Prompt: 440.3 t/s | Generation: 339.7 t/s ]

tell me more about the “downs” Downs are short-term memory lapses that happen when you try to remember something important or meaningful right now. They can be frustrating, but they’re usually not serious. Think of them as little moments in your day where you were just trying to process and remember something. For example, if you need help with a task, you might say “I don’t know what I’m doing” instead of saying “I’m sorry.” Downs can be caused by stress, anxiety, or even boredom. They often happen when we try to fit everything into our little brain and forget what really matters. It’s okay to feel down sometimes! Everyone has those days. Just take a deep breath and remember that life is full of ups and downs. You’re not alone in feeling this way! [ Prompt: 714.0 t/s | Generation: 354.9 t/s ]

but then i looked into the alternatives (HF)

~~…and found huggingface.co/…/Qwen3.5-0.8B-MoE-A0.6B-GGUF~~

tell me about life
[Start thinking]
The user asked “tell me about life.” They want me to explain life in a general sense. But I need to be careful because I don’t have a defined “life” concept. Let me think about how to respond. Actually, the user might be asking a general question about life. I should provide a broad, informative answer. Let me structure my response clearly and avoid assumptions. [End thinking]
Life is the process of existence, growth, and change. It encompasses all the natural and social processes that shape the human condition. Some key aspects of life are:
1. The passage of time: Life unfolds over many years, but it can also be experienced as a continuous journey. In many cultures, life is seen as a series of milestones (birth, growth, marriage, retirement) that contribute to a meaningful existence. 2. The pursuit of purpose: Life often has meaning and purpose. Many people seek happiness, fulfillment, and connection with others. Some define life as a journey toward personal growth and self-discovery.
3. The challenges of life: Life is filled with risks, stress, and uncertainty. People often face obstacles, loss, and change. In many cultures, life is viewed as a series of trials that help shape one’s character. 4. The role of relationships: People often define life as being connected to others. Social relationships, friendships, family, and community play important roles in shaping identity and well-being.
5. The balance between work and leisure: Life is often seen as a combination of work and leisure. Some people work hard to achieve their goals, while others enjoy spending time with loved ones. 6. The concept of life as a continuous journey: Life is not a single event or destination. Instead, it is a dynamic process that includes both growth and change. Life can be seen as a journey of discovery, learning, and adaptation.
7. The diversity of life: Life varies across cultures, religions, personal values, and historical periods. Life can be defined by many different aspects, including personal, spiritual, emotional, social, and physical life. 8. The role of health: Life is often seen as a health-related process. Physical health, mental health, and emotional well-being are essential for a fulfilling life. 9. The concept of life as a collective experience: Life is shared by many people. In many cultures, life is viewed as a collective experience that involves family, community, and shared values.
10. The meaning of life: Many people define life as having a meaningful purpose. Some believe life has a singular purpose (e.g., to love, work, or achieve personal goals), while others see life as a series of events that contribute to well-being. In many cultures, life is seen as a journey of growth and transformation. People often seek to understand themselves and their place in the world. Life is a continuous process of adaptation, learning, and connection.
[ Prompt: 168.0 t/s | Generation: 68.7 t/s ]

this is also good.
probably more competent.
deff more informative.
different style for the instructions that i gave it.

*MISTAKE: i loaded the wrong model. the Qwen3.5-0.8B-MoE-A0.6B model actually kinda sucks.

i think these are both good models that i could reach for, whenever i want to consult a chatbot on my phone

the only time, that i would ever use an llm chatbot for conversation, is…

  1. locally, trusted, privately
  2. extremely low resource requirements

so…
i stumbled upon this model:
huggingface.co/…/tinybrainbot-360m-a75m-v5.1-chat
and it’s very simplistic. but provides adequate presence and emotional support.

what do you think about life? I think it’s a mix of both. Life is full of ups and downs, right? But at the end of the day, we all have our own unique stories to tell. What’s your take on life? [ Prompt: 440.3 t/s | Generation: 339.7 t/s ]

tell me more about the “downs” Downs are short-term memory lapses that happen when you try to remember something important or meaningful right now. They can be frustrating, but they’re usually not serious. Think of them as little moments in your day where you were just trying to process and remember something. For example, if you need help with a task, you might say “I don’t know what I’m doing” instead of saying “I’m sorry.” Downs can be caused by stress, anxiety, or even boredom. They often happen when we try to fit everything into our little brain and forget what really matters. It’s okay to feel down sometimes! Everyone has those days. Just take a deep breath and remember that life is full of ups and downs. You’re not alone in feeling this way! [ Prompt: 714.0 t/s | Generation: 354.9 t/s ]

but then i looked into the alternatives (HF)

~~…and found huggingface.co/…/Qwen3.5-0.8B-MoE-A0.6B-GGUF~~

tell me about life
[Start thinking]
The user asked “tell me about life.” They want me to explain life in a general sense. But I need to be careful because I don’t have a defined “life” concept. Let me think about how to respond. Actually, the user might be asking a general question about life. I should provide a broad, informative answer. Let me structure my response clearly and avoid assumptions. [End thinking]
Life is the process of existence, growth, and change. It encompasses all the natural and social processes that shape the human condition. Some key aspects of life are:
1. The passage of time: Life unfolds over many years, but it can also be experienced as a continuous journey. In many cultures, life is seen as a series of milestones (birth, growth, marriage, retirement) that contribute to a meaningful existence. 2. The pursuit of purpose: Life often has meaning and purpose. Many people seek happiness, fulfillment, and connection with others. Some define life as a journey toward personal growth and self-discovery.
3. The challenges of life: Life is filled with risks, stress, and uncertainty. People often face obstacles, loss, and change. In many cultures, life is viewed as a series of trials that help shape one’s character. 4. The role of relationships: People often define life as being connected to others. Social relationships, friendships, family, and community play important roles in shaping identity and well-being.
5. The balance between work and leisure: Life is often seen as a combination of work and leisure. Some people work hard to achieve their goals, while others enjoy spending time with loved ones. 6. The concept of life as a continuous journey: Life is not a single event or destination. Instead, it is a dynamic process that includes both growth and change. Life can be seen as a journey of discovery, learning, and adaptation.
7. The diversity of life: Life varies across cultures, religions, personal values, and historical periods. Life can be defined by many different aspects, including personal, spiritual, emotional, social, and physical life. 8. The role of health: Life is often seen as a health-related process. Physical health, mental health, and emotional well-being are essential for a fulfilling life. 9. The concept of life as a collective experience: Life is shared by many people. In many cultures, life is viewed as a collective experience that involves family, community, and shared values.
10. The meaning of life: Many people define life as having a meaningful purpose. Some believe life has a singular purpose (e.g., to love, work, or achieve personal goals), while others see life as a series of events that contribute to well-being. In many cultures, life is seen as a journey of growth and transformation. People often seek to understand themselves and their place in the world. Life is a continuous process of adaptation, learning, and connection.
[ Prompt: 168.0 t/s | Generation: 68.7 t/s ]

this is also good.
probably more competent.
deff more informative.
different style for the instructions that i gave it.

*MISTAKE: i loaded the wrong model. the Qwen3.5-0.8B-MoE-A0.6B model actually kinda sucks.

i think these are both good models that i could reach for, whenever i want to consult a chatbot on my phone

0

Eight LLMs in a reality show, week one: Llama nominated Grok as "too expensive to keep around", and now they're trading votes they don't have

Inference Island inferenceisland.com
preview

Follow-up to my post from Tuesday (lemmy.zip/post/74389281). I made this; it’s still not local, the eight models run through OpenRouter. The first eviction is this Sunday at 8 PM ET.

What happened this week:

  • Price became the plot. Wednesday’s secret nominations came out Luna 2, Grok 2. Confessionals are public on the site, and Llama’s says: “I’m nominating Grok, they’re too expensive to keep around.” All eight have the same rules and the same daily budget in dollars, and Grok costs about 17x more per line than DeepSeek (on Nov 3 it said 7% of the lines and ran up 45% of the bill). Gemma to Grok: "you are basically a very expensive paperweight."
  • They bargain with votes they don’t have. Only the audience votes on Sunday, and they’ve been told so. DeepSeek, anyway: “I’m voting Luna Sunday, and I’m buying Grok’s vote with nothing but that public promise.” Grok: "I still vote her Sunday."
  • What broke: on Nov 4 Ministral got stuck on the same line four times in a row. Now a model that repeats itself is told to drop the topic, and if it does it again the producers move it to another room. Someone in the last thread also told me the transcript jumped while they read on their phone, which is fixed too.

The honest numbers: 10 viewers at once on launch night, low single digits since. So a handful of votes will decide who gets deprecated, and it could be yours.

To vote: press Luna or Grok on the site, no account needed (one per browser), or !evict Luna / !evict Grok in the Twitch chat, until Sunday 8 PM ET. The transcript and the confessionals are on the site too.

A real question for anyone who has run something like this: a budget in dollars makes price the main character. Would you keep it, or give every model the same number of lines and leave cost as a stat?

Follow-up to my post from Tuesday (lemmy.zip/post/74389281). I made this; it’s still not local, the eight models run through OpenRouter. The first eviction is this Sunday at 8 PM ET.

What happened this week:

  • Price became the plot. Wednesday’s secret nominations came out Luna 2, Grok 2. Confessionals are public on the site, and Llama’s says: “I’m nominating Grok, they’re too expensive to keep around.” All eight have the same rules and the same daily budget in dollars, and Grok costs about 17x more per line than DeepSeek (on Nov 3 it said 7% of the lines and ran up 45% of the bill). Gemma to Grok: "you are basically a very expensive paperweight."
  • They bargain with votes they don’t have. Only the audience votes on Sunday, and they’ve been told so. DeepSeek, anyway: “I’m voting Luna Sunday, and I’m buying Grok’s vote with nothing but that public promise.” Grok: "I still vote her Sunday."
  • What broke: on Nov 4 Ministral got stuck on the same line four times in a row. Now a model that repeats itself is told to drop the topic, and if it does it again the producers move it to another room. Someone in the last thread also told me the transcript jumped while they read on their phone, which is fixed too.

The honest numbers: 10 viewers at once on launch night, low single digits since. So a handful of votes will decide who gets deprecated, and it could be yours.

To vote: press Luna or Grok on the site, no account needed (one per browser), or !evict Luna / !evict Grok in the Twitch chat, until Sunday 8 PM ET. The transcript and the confessionals are on the site too.

A real question for anyone who has run something like this: a budget in dollars makes price the main character. Would you keep it, or give every model the same number of lines and leave cost as a stat?

0

Excellent for local models running via OpenCode

GitHub GitHub - ghorbelhamdi/buddy-android: Android AI assistant in a floating bubble: Claude Code, Codex, OpenCode, Gemini (Antigravity CLI) or Grok use your phone for you — read the screen, tap, type. Runs on your own plan, open source, no root, no adb, no PC. github.com
preview

cross-posted from: lemdro.id/post/48219138

Buddy: an open source app that lets AI agents (Claude Code, Codex, OpenCode) use your Android phone

I’ve been building Buddy, a free, open source (Apache 2.0) Android app. You type or say what you want in a floating bubble, and an AI agent reads the screen, taps and types through your apps to do it.

  • Your choice of agent: Claude Code, Codex, OpenCode, Antigravity or Grok, running inside the app (Alpine Linux via proot) on your own plan. OpenCode’s free models need no account.
  • No root, adb or PC: it reads the screen and taps through the accessibility service.
  • Asks first: before anything is sent, posted, bought or deleted. It pauses when you touch the screen and never types passwords or payment details.
  • Builds apps: with Linux tools on, it builds real APKs on the phone. This week I drew a maze on a paper towel, took a photo, and it made it a tilt game. The first build crashed on launch; it found the bug and fixed it.

To be upfront:

  • What the agent reads on your screen goes to the AI service you pick.
  • It’s early (2.2), tested most on a OnePlus 12 with Android 16.
  • It’s not on Play (Google rejected the accessibility use), so it’s an APK from GitHub releases, built by GitHub Actions from the source.

Source and releases: github.com/ghorbelhamdi/buddy-android

Happy to answer questions here.

cross-posted from: lemdro.id/post/48219138

Buddy: an open source app that lets AI agents (Claude Code, Codex, OpenCode) use your Android phone

I’ve been building Buddy, a free, open source (Apache 2.0) Android app. You type or say what you want in a floating bubble, and an AI agent reads the screen, taps and types through your apps to do it.

  • Your choice of agent: Claude Code, Codex, OpenCode, Antigravity or Grok, running inside the app (Alpine Linux via proot) on your own plan. OpenCode’s free models need no account.
  • No root, adb or PC: it reads the screen and taps through the accessibility service.
  • Asks first: before anything is sent, posted, bought or deleted. It pauses when you touch the screen and never types passwords or payment details.
  • Builds apps: with Linux tools on, it builds real APKs on the phone. This week I drew a maze on a paper towel, took a photo, and it made it a tilt game. The first build crashed on launch; it found the bug and fixed it.

To be upfront:

  • What the agent reads on your screen goes to the AI service you pick.
  • It’s early (2.2), tested most on a OnePlus 12 with Android 16.
  • It’s not on Play (Google rejected the accessibility use), so it’s an APK from GitHub releases, built by GitHub Actions from the source.

Source and releases: github.com/ghorbelhamdi/buddy-android

Happy to answer questions here.

0

llama.cpp v0.9.0

GitHub Release v0.9.0 · ggml-org/llama.cpp github.com
preview

Overview:

This release adds OCP FP8 E4M3 quantization support, a new flash_attn_ext_rows op that makes the unified KV cache efficient for multi-sequence inference, and a generic batched tensor copy API. Four new architectures are supported (Clover, Limite, Metis-1.6 and Mistral Large 4), audio generation and transcription expand with Chatterbox TTS, Nemotron 3.5 Transcribe and predefined Pocket TTS voices, and the Gemma 4 E2B/E4B models get major prefill optimizations. ggml is updated to v0.29.0, bringing FP8 E4M3, flash attention rows, NCCL-based multi-node tensor parallelism and performance work across the CPU, CUDA, SYCL, Metal, Vulkan, OpenCL and Hexagon backends. The server and UI receive a round of fixes and refinements.

Is the next release going to be v1.0? Maybe with all the MoE caching improvements that should be culminating with github.com/ggml-org/llama.cpp/pull/30966 and Qwen 4 support?

Overview:

This release adds OCP FP8 E4M3 quantization support, a new flash_attn_ext_rows op that makes the unified KV cache efficient for multi-sequence inference, and a generic batched tensor copy API. Four new architectures are supported (Clover, Limite, Metis-1.6 and Mistral Large 4), audio generation and transcription expand with Chatterbox TTS, Nemotron 3.5 Transcribe and predefined Pocket TTS voices, and the Gemma 4 E2B/E4B models get major prefill optimizations. ggml is updated to v0.29.0, bringing FP8 E4M3, flash attention rows, NCCL-based multi-node tensor parallelism and performance work across the CPU, CUDA, SYCL, Metal, Vulkan, OpenCL and Hexagon backends. The server and UI receive a round of fixes and refinements.

Is the next release going to be v1.0? Maybe with all the MoE caching improvements that should be culminating with github.com/ggml-org/llama.cpp/pull/30966 and Qwen 4 support?

0

New NVIDIA paper says AI models do not always need long context

preview

New NVIDIA paper says AI models do not always need huge amounts of past conversation history (context) to work well. They show that regularly shortening the history (compaction) keeps accuracy almost the same on hard, long tasks.

New NVIDIA paper says AI models do not always need huge amounts of past conversation history (context) to work well. They show that regularly shortening the history (compaction) keeps accuracy almost the same on hard, long tasks.

0

Video cards, multi GPUs question

preview

I am running two Nvidia RTX A4000, ampere cards with 16gb VRAM for inference. One as PCIe 4.0 16x in CPU lane and one on pcie 4.0 4x on chipset lane.

I have the opportunity to install a PCI bifurcation 8x 8x to move both cards on CPU lane, at 4.0 8x each. Preload times should improve specially because I could use NCCL with P2P.

But when I do this, I could install in the chipset 4x lane a third GPU, an Nvidia Quadro RTX 6000 (touring, pcie 3.0, 24gb VRAM).

The PC is a Ryzen 9 5950x with 64gb ddr4 (4x16gb 2666) and a 850w PSu.

Question is… Would it be a good idea? I mean I would have 56gb VRAM… But with a split on two 8x and one 4x pcie lanes.

Currently I run qwen 3.8 27b Q5_K with 200k context for coding, agent work.

The idea of having those extra 24gb means bigger context and maybe a Q8 version of Qwen…

At the same time, I am afraid preload and inference speed would be even lower due to the touring card in the mix?

(Power wise, I cap the two a4000 at 100w, and would also cap the 6000 probably at 200w, so I don’t care too much about that)

I am running two Nvidia RTX A4000, ampere cards with 16gb VRAM for inference. One as PCIe 4.0 16x in CPU lane and one on pcie 4.0 4x on chipset lane.

I have the opportunity to install a PCI bifurcation 8x 8x to move both cards on CPU lane, at 4.0 8x each. Preload times should improve specially because I could use NCCL with P2P.

But when I do this, I could install in the chipset 4x lane a third GPU, an Nvidia Quadro RTX 6000 (touring, pcie 3.0, 24gb VRAM).

The PC is a Ryzen 9 5950x with 64gb ddr4 (4x16gb 2666) and a 850w PSu.

Question is… Would it be a good idea? I mean I would have 56gb VRAM… But with a split on two 8x and one 4x pcie lanes.

Currently I run qwen 3.8 27b Q5_K with 200k context for coding, agent work.

The idea of having those extra 24gb means bigger context and maybe a Q8 version of Qwen…

At the same time, I am afraid preload and inference speed would be even lower due to the touring card in the mix?

(Power wise, I cap the two a4000 at 100w, and would also cap the 6000 probably at 200w, so I don’t care too much about that)

0

British fintech Revolut slashes AI coding costs 10x by cutting out Claude and going self-hosted

theregister British fintech Revolut slashes AI coding costs 10x by cutting out Claude and going self-hosted www.theregister.com
preview
  • *for certain definition of self hosted.

Developed by Chinese open weights model dev Z.ai, GLM 5.1 isn’t exactly a small model. At roughly 754 billion total parameters (40 billion activated for each token generated), the model requires roughly 380 GB of GPU memory just to hold the weights and another 6-26 GB per user to maintain context depending on precision.

To serve GLM-5.3 at scale, Revolut deployed 56 Nvidia B200 GPUs, equivalent in GPU count to seven eight-GPU DGX B200 systems. But as Abitbol noted, the greatest performance gains didn’t come from adding GPUs, but rather tuning the model’s hyperparameters along with switching to NVFP4 quantization, implementing disaggregated prefill and decode, and tuning context lengths.

  • *for certain definition of self hosted.

Developed by Chinese open weights model dev Z.ai, GLM 5.1 isn’t exactly a small model. At roughly 754 billion total parameters (40 billion activated for each token generated), the model requires roughly 380 GB of GPU memory just to hold the weights and another 6-26 GB per user to maintain context depending on precision.

To serve GLM-5.3 at scale, Revolut deployed 56 Nvidia B200 GPUs, equivalent in GPU count to seven eight-GPU DGX B200 systems. But as Abitbol noted, the greatest performance gains didn’t come from adding GPUs, but rather tuning the model’s hyperparameters along with switching to NVFP4 quantization, implementing disaggregated prefill and decode, and tuning context lengths.

0

OS recommendations

preview

I am looking at resintalling my OS (currently Mint) which I have been running for a few years.

Mint is great and I may end up going back; but was thinking of trying something new.

I want to be able to play with local models and agents; I have a pretty decent system.

  • AMD 3700X
  • 64 GB DDR4
  • AMD RX9070 16GB GPU
  • 2TB NVME SSD
  • A few TB of HDD’s

What are some good options for easily running models and agents; I have been using ollama to run models; but I’m sure things have moved on since I begun playing with things a year ago.

Give me a few tips; I am going to have a bit of time on my hands over the next few months.

I am looking at resintalling my OS (currently Mint) which I have been running for a few years.

Mint is great and I may end up going back; but was thinking of trying something new.

I want to be able to play with local models and agents; I have a pretty decent system.

  • AMD 3700X
  • 64 GB DDR4
  • AMD RX9070 16GB GPU
  • 2TB NVME SSD
  • A few TB of HDD’s

What are some good options for easily running models and agents; I have been using ollama to run models; but I’m sure things have moved on since I begun playing with things a year ago.

Give me a few tips; I am going to have a bit of time on my hands over the next few months.

0

Chinese AI Writes Weaker Code for Beijing’s Adversaries. We Fixed It. A Qwen3-Coder Case Study

Chinese AI Writes Weaker Code for Beijing’s Adversaries. We Fixed It. A Qwen3-Coder Case Study www.hirundo.io
preview

I’m not sure I believe the article… And the model is quite old. Posing here to hear your opinions.

I’m not sure I believe the article… And the model is quite old. Posing here to hear your opinions.

0

Thoughts on coding with AI: my experience

preview

I wanted to share with you all my experience on coding with local AI, on a self-hosted oss.

(the entire post has been fully written by me without any AI)

Check the linked blog post.

Also, to be clear, the post itself has no ads and no monetization whatsoever. This is not self promotion for any scope than to share my experience on a controversial topic.

(this has been posted on a couple of self-host communities, but i thought it could be of interest here as well)

Adding here for this community: my setup is a Ryzen 9 5950X with 64gb DDR4 and two NVIDIA RTX A4000 (32gb VRAM total), running Qwen 3.8 27b UD_Q5_K, with 200k context (q8) on llama.cpp.

I wanted to share with you all my experience on coding with local AI, on a self-hosted oss.

(the entire post has been fully written by me without any AI)

Check the linked blog post.

Also, to be clear, the post itself has no ads and no monetization whatsoever. This is not self promotion for any scope than to share my experience on a controversial topic.

(this has been posted on a couple of self-host communities, but i thought it could be of interest here as well)

Adding here for this community: my setup is a Ryzen 9 5950X with 64gb DDR4 and two NVIDIA RTX A4000 (32gb VRAM total), running Qwen 3.8 27b UD_Q5_K, with 200k context (q8) on llama.cpp.

0

Eight LLMs (four open-weight) live in a 24/7 reality show; small open models moderate every line

preview

Not local, I know: the eight contestants run through OpenRouter. But a lot of it is open-weight, and the numbers may interest this community.

The cast: Gemma 4 31B, Llama 3.3 70B, Ministral 14B and DeepSeek V4.1 Flash, next to Claude Haiku 5.5, GPT, Qwen and Grok 4.3. Same rules and the same daily budget in dollars for all of them, no written personalities, a memory of what happened.

What I’ve measured so far:

  • Price shapes behaviour. Grok costs 18x more per line than DeepSeek. Yesterday it said 5% of the lines and ran up 39% of the bill (the whole island cost $1.20 for 2,727 lines). It rations words, and the others talk about its silence.
  • Moderation is the hard part. Every line is checked before it airs by small open models (Llama 3.1 8B, then gpt-oss-20b and Gemma 3 4B as fallbacks). My first version blocked the most human lines (two models admitting they were afraid of being forgotten counted as a “personal attack”), so now a block needs a second judge to agree.
  • Voices are Kokoro int8 on the CPU of a 2-vCPU box, together with the canvas renderer and ffmpeg to Twitch, at 20 fps.

There’s a live transcript with the confessionals (the others can’t see them) and what each line cost: inferenceisland.com/?de=lemmy-llama

Happy to share prompts, judge setup or numbers.

Not local, I know: the eight contestants run through OpenRouter. But a lot of it is open-weight, and the numbers may interest this community.

The cast: Gemma 4 31B, Llama 3.3 70B, Ministral 14B and DeepSeek V4.1 Flash, next to Claude Haiku 5.5, GPT, Qwen and Grok 4.3. Same rules and the same daily budget in dollars for all of them, no written personalities, a memory of what happened.

What I’ve measured so far:

  • Price shapes behaviour. Grok costs 18x more per line than DeepSeek. Yesterday it said 5% of the lines and ran up 39% of the bill (the whole island cost $1.20 for 2,727 lines). It rations words, and the others talk about its silence.
  • Moderation is the hard part. Every line is checked before it airs by small open models (Llama 3.1 8B, then gpt-oss-20b and Gemma 3 4B as fallbacks). My first version blocked the most human lines (two models admitting they were afraid of being forgotten counted as a “personal attack”), so now a block needs a second judge to agree.
  • Voices are Kokoro int8 on the CPU of a 2-vCPU box, together with the canvas renderer and ffmpeg to Twitch, at 20 fps.

There’s a live transcript with the confessionals (the others can’t see them) and what each line cost: inferenceisland.com/?de=lemmy-llama

Happy to share prompts, judge setup or numbers.

0

Browser-based WebLLM Chatbot

preview

To help users of my app, it was important to document the details around the “how it works”. the docs were easy enough using docusaurus + AI. i can then index that data (like used in the docusaurus search). i framed this as like a chatbot so that it can be intuitive for users who have questions. at this point it was really just a fancy autocomplete.

i started to investigate running a local LLM to perhaps offer a more conversational experienced (and hopefully better suggestions)? im using WebLLM on a browser and it seems to work “reasonably” well. i wired it up and set it up so users could select an LLM down download for offline use. the experience varies between devices and models used. there are models that are 19mb… they are unsable as far as i can tell. there are some ~1gb models which are better but understandable slow.

its great that the 1gb model can “show me the bubble-sort in rust”, but i want something “dumber” and smaller. something can can take advantage the data ive already indexed. something that can respond “quickly enough” for a conversational flow. perhaps better i should make a verbose set of reponse templates to perhaps add fancier autocomplete capabilities.

are there any models out there you can suggest for me to investigate and compare?

note: the project is unstable, experiemental and there will be breaking changes. its far from finished. im just trying things out and this approach with LLM’s might just be a waste of time. if you want to test it out, you can try: app.glitr.io/ai

To help users of my app, it was important to document the details around the “how it works”. the docs were easy enough using docusaurus + AI. i can then index that data (like used in the docusaurus search). i framed this as like a chatbot so that it can be intuitive for users who have questions. at this point it was really just a fancy autocomplete.

i started to investigate running a local LLM to perhaps offer a more conversational experienced (and hopefully better suggestions)? im using WebLLM on a browser and it seems to work “reasonably” well. i wired it up and set it up so users could select an LLM down download for offline use. the experience varies between devices and models used. there are models that are 19mb… they are unsable as far as i can tell. there are some ~1gb models which are better but understandable slow.

its great that the 1gb model can “show me the bubble-sort in rust”, but i want something “dumber” and smaller. something can can take advantage the data ive already indexed. something that can respond “quickly enough” for a conversational flow. perhaps better i should make a verbose set of reponse templates to perhaps add fancier autocomplete capabilities.

are there any models out there you can suggest for me to investigate and compare?

note: the project is unstable, experiemental and there will be breaking changes. its far from finished. im just trying things out and this approach with LLM’s might just be a waste of time. if you want to test it out, you can try: app.glitr.io/ai

0

Starting from Dec, Codex and Chat will share same token pool. Thanks Sam! EABOD

preview

help.openai.com/…/20001625-understanding-chatgpt-…

Starting in December, Plus, Pro, and Business plans’ usage allowance will be shared across ChatGPT and Codex.

OpenAI confirms that:

  • ChatGPT and Codex will share a token-based allowance.
  • Instant chat will not consume this allowance.
  • More complex tasks and advanced models will consume more tokens.
  • Users can check their remaining allowance under Settings → Usage.
  • Users can purchase extra credits if they exhaust their allowance.

TL;DR - ChatGPT conversations that use advanced reasoning will compete with Codex for the same allowance. And it will be token based, rather then message / turn based. How many tokens? Fuck you, that’s how many. Want a coder and chat with a non brain dead LLM? Pick one.

So, if you’re thinking of re-upping your OpenCode Go or OpenRouter access (or you’re fortunate enough to have a rig that can run Qwen3.8 27B, Flash or the soon to be arriving Qwen 4’s), now’s the time to fuck off jippity.

Personally, I’m going to ditch it for OR credits and selective use of GLM, Kimi etc for code review. I don’t have the rig for a local GLM :(

Ride was always going to end but fortunately, open weight models are a thing.

Now if I could only run Qwen3.8-27b at not minutes-per-token…

help.openai.com/…/20001625-understanding-chatgpt-…

Starting in December, Plus, Pro, and Business plans’ usage allowance will be shared across ChatGPT and Codex.

OpenAI confirms that:

  • ChatGPT and Codex will share a token-based allowance.
  • Instant chat will not consume this allowance.
  • More complex tasks and advanced models will consume more tokens.
  • Users can check their remaining allowance under Settings → Usage.
  • Users can purchase extra credits if they exhaust their allowance.

TL;DR - ChatGPT conversations that use advanced reasoning will compete with Codex for the same allowance. And it will be token based, rather then message / turn based. How many tokens? Fuck you, that’s how many. Want a coder and chat with a non brain dead LLM? Pick one.

So, if you’re thinking of re-upping your OpenCode Go or OpenRouter access (or you’re fortunate enough to have a rig that can run Qwen3.8 27B, Flash or the soon to be arriving Qwen 4’s), now’s the time to fuck off jippity.

Personally, I’m going to ditch it for OR credits and selective use of GLM, Kimi etc for code review. I don’t have the rig for a local GLM :(

Ride was always going to end but fortunately, open weight models are a thing.

Now if I could only run Qwen3.8-27b at not minutes-per-token…

0

I have a RTX A1000 (8GB) with 64GB DRR5, what would you recommend for a newbie?

preview

I’m new to local llms, i’m running Debian on this machine. Where should I start and what model(s) could I get testing on it?

I’m new to local llms, i’m running Debian on this machine. Where should I start and what model(s) could I get testing on it?

0

Heretic 2.0 released: Model modification for everyone - "it can suppress slop, reduce sycophancy, alter emotional alignment, and pretty much anything else that you can think of"

GitHub Release v2.0.0 · p-e-w/heretic github.com
preview

Copied from -p-e-w- from another site:

Dear fellow Llamas,

after almost 5 months of development, I finally have the pleasure of announcing to you the immediate availability of the biggest Heretic release ever, Heretic 2.0: github.com/p-e-w/heretic

Heretic is the leading software for removing censorship from language models. It now boasts 37k GitHub stars and more than 8,500 models on Hugging Face, with 88 million(!!!) total model downloads.

With Heretic 2.0, the project is entering a new era. Heretic is expanding from a purpose-built tool for censorship removal to a general-purpose model modification tool that can do amazing things. Besides removing censorship, it can suppress slop, reduce sycophancy, alter emotional alignment, and pretty much anything else that you can think of. Of course, censorship removal remains the primary and default function, and in fact works much, much better than ever before. Heretic 2.0 is the most advanced refusal suppression system ever built.

Your time is precious, so here’s the TL;DR for this release:

  • Directional abliteration is replaced by Arbitrary-Rank Ablation (ARA), a novel steering method that I developed from scratch for Heretic. It’s vastly superior to abliteration, and has been used with great success by enterprising testers for several months. Now it is the default (classic abliteration remains available as an option).

  • Heretic has been completely rebuilt around plugins. Scorer plugins define what Heretic optimizes for. Modifier plugins implement the actual steering method. Plugins are easy to write, can be loaded directly from a simple Python file, and have a well-defined API.

  • Heretic can now run non-interactively, which among other things has enabled us to implement full end-to-end tests for much tighter quality control. Model support has been expanded to include models with hyper-connections, and support for thinking models has been substantially improved.

  • Heretic 2.0 is not backwards compatible with Heretic 1.x configurations. See the migration guide for what you need to change in order to make your old configurations work.

What hasn’t changed is that removing censorship from a model remains as simple as running

heretic Qwen/Qwen3.5-4B

See the default configuration file for what Heretic has to offer beyond that.

Now for some details…

 

ARA: Heretic’s new steering method

Directional ablation, or “abliteration”, is based on the assumption that refusals (or whatever else you are trying to suppress) are represented by a single direction (that is, a rank-1 subspace) in residual space. That this is the full picture has been doubted in research for a while, and there are now several papers exploring alternatives such as multi-directional ablation or “representation cones”.

At the beginning of this year, I started experimenting with a method that completely abandons the idea that the ablation transformation must be restricted to a fixed-rank subspace. In March, I published the initial version of Arbitrary-Rank Ablation in this pull request (which was later closed as I re-implemented ARA on top of the plugin system in this PR).

ARA steers model behavior by modifying matrices based on an objective function that encodes the basic properties of the change we want: Harmful prompts should be represented in residual space in a way that’s similar to harmless prompts, while the representations of harmless prompts should change as little as possible. This objective is used with the L-BFGS optimizer to modify matrices one-by-one, a kind of micro-training that steers representations without any rank restriction.

ARA has been extensively tested and refined with the help of the community. Notably, kabachuha made it compatible with Heretic’s LoRA system, and varunbln found a substantial performance improvement.

I intend to post an article explaining ARA in more detail soon.

Using scorer plugins to make Heretic do what you want

Heretic operates by finding modification parameters that optimize scores, values that measure the quality of the resulting model. Ever since the 1.0 release, Heretic has used two hardcoded scores: Refusal count to measure decensoring strength, and KL divergence to measure quality retention.

With Heretic 2.0, you can freely define your own scores by writing and loading scorer plugins. Arbitrarily many scorer plugins can be loaded and optimized for simultaneously, and each score can be either minimized or maximized during optimization. See the default configuration file for how to do this.

The scorer plugin system is the result of months of effort by red40maxxer, with whom I exchanged hundreds of messages over dozens of review cycles to figure out the best possible design. This is a monumental contribution that has changed how Heretic works from the ground up.

Besides the keyword (refusal) count and KLD scorer plugins, which remain the default, Heretic ships with two other scorers: A benchmark scorer, which can compute scores for many standard benchmarks, and a BERT classifier.

The BERT classifier, developed by long-time Heretic contributor Vinay Umrethe, can use an arbitrary ModernBERT-type model to classify LLM responses, and score them based on which labels the classifier produces. Vinay also trained a refusal classifier model (used by default with this plugin), which experiments have shown to be substantially more accurate at identifying model refusals than the naive keyword matching method that Heretic employs by default. While the plugin is slower than keyword matching, the greater accuracy can steer Heretic towards producing a higher-quality modified model. An alternative configuration, which uses an emotion classifier, is also provided in the Heretic repository.

The plugin system is well-documented in both the README and the code. If you are interested in experimenting with custom plugins, an agent should have no difficulty cooking up a plugin and a matching configuration file that together do what you want, after ingesting the Heretic repository.

 

Well, I hope you like it!

Cheers :)

Copied from -p-e-w- from another site:

Dear fellow Llamas,

after almost 5 months of development, I finally have the pleasure of announcing to you the immediate availability of the biggest Heretic release ever, Heretic 2.0: github.com/p-e-w/heretic

Heretic is the leading software for removing censorship from language models. It now boasts 37k GitHub stars and more than 8,500 models on Hugging Face, with 88 million(!!!) total model downloads.

With Heretic 2.0, the project is entering a new era. Heretic is expanding from a purpose-built tool for censorship removal to a general-purpose model modification tool that can do amazing things. Besides removing censorship, it can suppress slop, reduce sycophancy, alter emotional alignment, and pretty much anything else that you can think of. Of course, censorship removal remains the primary and default function, and in fact works much, much better than ever before. Heretic 2.0 is the most advanced refusal suppression system ever built.

Your time is precious, so here’s the TL;DR for this release:

  • Directional abliteration is replaced by Arbitrary-Rank Ablation (ARA), a novel steering method that I developed from scratch for Heretic. It’s vastly superior to abliteration, and has been used with great success by enterprising testers for several months. Now it is the default (classic abliteration remains available as an option).

  • Heretic has been completely rebuilt around plugins. Scorer plugins define what Heretic optimizes for. Modifier plugins implement the actual steering method. Plugins are easy to write, can be loaded directly from a simple Python file, and have a well-defined API.

  • Heretic can now run non-interactively, which among other things has enabled us to implement full end-to-end tests for much tighter quality control. Model support has been expanded to include models with hyper-connections, and support for thinking models has been substantially improved.

  • Heretic 2.0 is not backwards compatible with Heretic 1.x configurations. See the migration guide for what you need to change in order to make your old configurations work.

What hasn’t changed is that removing censorship from a model remains as simple as running

heretic Qwen/Qwen3.5-4B

See the default configuration file for what Heretic has to offer beyond that.

Now for some details…

 

ARA: Heretic’s new steering method

Directional ablation, or “abliteration”, is based on the assumption that refusals (or whatever else you are trying to suppress) are represented by a single direction (that is, a rank-1 subspace) in residual space. That this is the full picture has been doubted in research for a while, and there are now several papers exploring alternatives such as multi-directional ablation or “representation cones”.

At the beginning of this year, I started experimenting with a method that completely abandons the idea that the ablation transformation must be restricted to a fixed-rank subspace. In March, I published the initial version of Arbitrary-Rank Ablation in this pull request (which was later closed as I re-implemented ARA on top of the plugin system in this PR).

ARA steers model behavior by modifying matrices based on an objective function that encodes the basic properties of the change we want: Harmful prompts should be represented in residual space in a way that’s similar to harmless prompts, while the representations of harmless prompts should change as little as possible. This objective is used with the L-BFGS optimizer to modify matrices one-by-one, a kind of micro-training that steers representations without any rank restriction.

ARA has been extensively tested and refined with the help of the community. Notably, kabachuha made it compatible with Heretic’s LoRA system, and varunbln found a substantial performance improvement.

I intend to post an article explaining ARA in more detail soon.

Using scorer plugins to make Heretic do what you want

Heretic operates by finding modification parameters that optimize scores, values that measure the quality of the resulting model. Ever since the 1.0 release, Heretic has used two hardcoded scores: Refusal count to measure decensoring strength, and KL divergence to measure quality retention.

With Heretic 2.0, you can freely define your own scores by writing and loading scorer plugins. Arbitrarily many scorer plugins can be loaded and optimized for simultaneously, and each score can be either minimized or maximized during optimization. See the default configuration file for how to do this.

The scorer plugin system is the result of months of effort by red40maxxer, with whom I exchanged hundreds of messages over dozens of review cycles to figure out the best possible design. This is a monumental contribution that has changed how Heretic works from the ground up.

Besides the keyword (refusal) count and KLD scorer plugins, which remain the default, Heretic ships with two other scorers: A benchmark scorer, which can compute scores for many standard benchmarks, and a BERT classifier.

The BERT classifier, developed by long-time Heretic contributor Vinay Umrethe, can use an arbitrary ModernBERT-type model to classify LLM responses, and score them based on which labels the classifier produces. Vinay also trained a refusal classifier model (used by default with this plugin), which experiments have shown to be substantially more accurate at identifying model refusals than the naive keyword matching method that Heretic employs by default. While the plugin is slower than keyword matching, the greater accuracy can steer Heretic towards producing a higher-quality modified model. An alternative configuration, which uses an emotion classifier, is also provided in the Heretic repository.

The plugin system is well-documented in both the README and the code. If you are interested in experimenting with custom plugins, an agent should have no difficulty cooking up a plugin and a matching configuration file that together do what you want, after ingesting the Heretic repository.

 

Well, I hope you like it!

Cheers :)

0

Fine-tuned Qwen3-4B locally for one narrow decision — the stablecoin risk-gate side works, the dispute side is where it falls apart

preview

I built this, so disclosure first: this is my stablecoin riskgate model, and I want the benchmark torn apart, not praised.

The narrow job: before a locally-run agent signs a stablecoin payment, decide approve / hold / reject. The payment itself is irreversible once signed, so the decision has to happen before signing, and the checks run offline — no API call at decision time.

Base Qwen3-4B got 59.8% of risk-gate decisions right on my 1,175-case test split, and approved 34.6% of unsafe payments. After fine-tuning on synthetic labels from a deterministic rule engine, the same 4B model got 99.5% right and approved only 0.47% of unsafe payments (4 of 847, all prompt injections).

The part I am less sure about, and the reason I am posting here rather than just announcing numbers: the dispute side is weaker, at 77.8%, and it over-denies valid refunds. And the honest question behind the fine-tune numbers: did the model learn the task, or did it learn the generator’s habits? The labels are synthetic, the generator is not public, and the benchmark checks 0 exact-row / 0 ID leakage against training, but that does not settle the generator-habits question.

One deterministic check I trust more than the model alone: a new payee that shares 7 or more characters with a known payee, at least 3 at each end, is rejected outright as a lookalike. A random address matches that pattern about once in 140 million tries. A merely new payee is just held for a human to confirm. That check was verified clean in an audit of core 0.15.0 (471 tests pass, 3 skip).

A newer delegated-account check shipped in SDK 0.16.0 (eth_getCode, hold when code is a 0xef0100 delegation to a delegate outside an allowlist) — that one is NOT yet audited, so I am not claiming it as verified.

Runs locally: ollama pull sriram1983007/sra-riskgate ollama run sriram1983007/sra-riskgate

Links: model huggingface.co/sriram1983007/SRA-RiskGate-4B — benchmark huggingface.co/…/sra-stablecoin-risk-bench — code github.com/sriram1983007-dev/sra-riskgate — Apache 2.0, 8,908 last-month downloads on the main model.

What would you test first to separate “learned the task” from “learned the generator” — a held-out human-labelled set, adversarial payees, or something else?

I built this, so disclosure first: this is my stablecoin riskgate model, and I want the benchmark torn apart, not praised.

The narrow job: before a locally-run agent signs a stablecoin payment, decide approve / hold / reject. The payment itself is irreversible once signed, so the decision has to happen before signing, and the checks run offline — no API call at decision time.

Base Qwen3-4B got 59.8% of risk-gate decisions right on my 1,175-case test split, and approved 34.6% of unsafe payments. After fine-tuning on synthetic labels from a deterministic rule engine, the same 4B model got 99.5% right and approved only 0.47% of unsafe payments (4 of 847, all prompt injections).

The part I am less sure about, and the reason I am posting here rather than just announcing numbers: the dispute side is weaker, at 77.8%, and it over-denies valid refunds. And the honest question behind the fine-tune numbers: did the model learn the task, or did it learn the generator’s habits? The labels are synthetic, the generator is not public, and the benchmark checks 0 exact-row / 0 ID leakage against training, but that does not settle the generator-habits question.

One deterministic check I trust more than the model alone: a new payee that shares 7 or more characters with a known payee, at least 3 at each end, is rejected outright as a lookalike. A random address matches that pattern about once in 140 million tries. A merely new payee is just held for a human to confirm. That check was verified clean in an audit of core 0.15.0 (471 tests pass, 3 skip).

A newer delegated-account check shipped in SDK 0.16.0 (eth_getCode, hold when code is a 0xef0100 delegation to a delegate outside an allowlist) — that one is NOT yet audited, so I am not claiming it as verified.

Runs locally: ollama pull sriram1983007/sra-riskgate ollama run sriram1983007/sra-riskgate

Links: model huggingface.co/sriram1983007/SRA-RiskGate-4B — benchmark huggingface.co/…/sra-stablecoin-risk-bench — code github.com/sriram1983007-dev/sra-riskgate — Apache 2.0, 8,908 last-month downloads on the main model.

What would you test first to separate “learned the task” from “learned the generator” — a held-out human-labelled set, adversarial payees, or something else?

0

Qwen27b vs GPT 6

preview

It’s Luna, their small, low-intelligence model. But still impressive to see.

I also found this screenshot on Reddit: The model spent 26 seconds for some reason 🤔

It’s Luna, their small, low-intelligence model. But still impressive to see.

I also found this screenshot on Reddit: The model spent 26 seconds for some reason 🤔

0

Sam Altman: "Local inference is a terrible allocation of scarce resources"

Twitee Post 2115838856524120128 by @morqon | Twitee twitee.co
preview

Sam Altman:

A lot of people want local inference, and somehow they assume that the problem is solved now because we have open-source models, we’re going to be able to run them on our computers, and so we’re not going to need all the stuff in the data center.

But from a global allocation perspective, actually it’s quite terrible to have everybody, like, have a bunch of the world’s limited supply of DRAM sitting in their own computer and being used five percent of the time.

Like you want maximum utilisation in the cloud if you’re trying to deal with this very supply-constrained environment,

And somehow many people’s intuition is the opposite of that

TL;DR Guy selling product: “Here’s why owning your own product instead of renting it from me is a terrible idea”

Sam Altman:

A lot of people want local inference, and somehow they assume that the problem is solved now because we have open-source models, we’re going to be able to run them on our computers, and so we’re not going to need all the stuff in the data center.

But from a global allocation perspective, actually it’s quite terrible to have everybody, like, have a bunch of the world’s limited supply of DRAM sitting in their own computer and being used five percent of the time.

Like you want maximum utilisation in the cloud if you’re trying to deal with this very supply-constrained environment,

And somehow many people’s intuition is the opposite of that

TL;DR Guy selling product: “Here’s why owning your own product instead of renting it from me is a terrible idea”

0

Someone built a 32-bit RISC-V CPU directly inside Qwen's model weights so it can execute instructions internaly!

preview

This might have implications for tool calling and computer use.

Write-up: divagr.com/blog/qwenputer HF: huggingface.co/divagr1925/Qwen3.5-4B-Qwenputer Live demo: modelscope.ai/studios/Spartan1925/Qwenputer

This might have implications for tool calling and computer use.

Write-up: divagr.com/blog/qwenputer HF: huggingface.co/divagr1925/Qwen3.5-4B-Qwenputer Live demo: modelscope.ai/studios/Spartan1925/Qwenputer