Your quantized model might be broken

Numbers come from arXiv 2609.05881v3 and the author's quantcheck audit data. Whether the bad files are still being served is our own check; the method is at the end.

A scene

You are on a laptop. You run ollama pull qwen2.5-coder:3b, choosing a 2-bit tag because memory is tight.

The model starts. It answers, and it is fast. Then you ask for a sorting function and the code does not compile. You try another task; same thing.

Your first instinct is to blame yourself: the prompt? the temperature? maybe this model is just weak?

A paper turned that doubt into a census: of 305 code-capable tags in the official Ollama library, seven solved at most six of 164 HumanEval+ tasks. Six solved none.

The counter-intuitive part: the files are fine

When a model misbehaves, you suspect a corrupt download.

The author rebuilt five of the seven from the original weights with llama.cpp. Comparing tensor by tensor, all five matched exactly — 434 tensors for the Qwen2.5-Coder-3B tags.

So these are not corrupt files. They are correctly reproduced broken artifacts.

Checksums pass. File sizes match. ollama show looks normal. Because the file really is exactly what it claims to be. What is broken is the quantization recipe: same weights, same toolchain, same tensor mix, but without calibration the model collapsed into a shape that cannot write code.

Why this happens

Quantization squeezes weights from 16 bits down to 2–4 bits. Every step has to decide where the rounding error lands.

The default approach spreads it evenly. The better approach uses calibration data — llama.cpp's importance matrix (imatrix) — to push error away from the weights that matter, based on activation statistics.

Without imatrix, small models at low bit are the ones that fall apart. All seven suspects sit in q2_K through q3_K_L, in models of at most 4B parameters. That subgroup contains 48 tags, so the failure rate inside it is 14.6%.

But the recipe alone does not predict failure: the same uncalibrated recipe left Qwen2.5-Coder-1.5B functional (13–15 of 15). The problem is not "low bit always breaks". It is that when it breaks, you cannot tell.

Calibration rescues most of them

Holding weights, toolchain and tensor mix fixed, adding only an imatrix:

(The paper's abstract says "0 → 76–113"; the per-level numbers above come from the v3.4 audit data, which we treat as authoritative.)

Same model, same bit width, one calibration table: from zero tasks to three digits.

Another failure mode: a missing chat template

Beyond the seven, the author also tested 22 popular Hugging Face community files. Two of them (TheBloke's deepseek-coder-6.7B Q3_K_M and Q4_K_M) scored 5 and 6 of 15 in the census.

The cause was not quantization but a missing chat template. With no template, Ollama falls back to a default ChatML format that does not match the model's instruction format: 28 of 30 answers ran on into invented chat turns.

With the right template: 98 and 101 of 164.

So "it runs" and "it answers properly" are separated by a template.

We checked: the bad tags are still being served

A September incident is only interesting if it is still downloadable. We compared the blob digests in the audit data against the current Ollama registry manifests:

Seven out of seven are still there, with identical digests. This is not history; it is something you can pull today.

(Method: fetch https://registry.ollama.ai/v2/library/<model>/manifests/<tag> and compare the application/vnd.ollama.image.model layer digest. Script at the end.)

Three things you can do now

Screen. The author released quantcheck, a 15-task screen per tag. It costs about 30 seconds of generation (on a 4090, excluding download). None of 66 sampled tags that passed the screen turned out to be non-functional, so it works as a pre-filter.

If you publish quantizations: add an imatrix, and run a functional test before publishing. Registries will not do it for you.

Check the chat template. ollama show <model> --template; if it prints {{ .Prompt }}, there is no template and the model may be answering in the wrong format.

Do not over-read this

• 2.3% is the share of official, code-capable, ≤8 GB tags — not of all quantized models
• Only coding ability was tested (HumanEval+); perplexity is a pre-check, not proof of function
• Seven is a lower bound: all 12 deepseek-coder-1.3B tags, including fp16, score 1–6 of 15, so a group baseline cannot judge them and the author does not count them
• The author is an independent researcher; data and tools are public, but this is one registry and one runtime version

Conclusion

Exact reproducibility is not function.

Identical tensors, a passing checksum, a normal ollama show — all of that proves the file is the file. None of it proves the model can do the job.

So the question is not whether quantized models are usable. It is whether anyone tested whether this one is. Right now the registries have not, which leaves it to you.

Sources: arXiv 2609.05881v3, quantcheck, Ollama issue #18252. Our digest check: scripts/check-ollama-tags.ts; run npm run check:ollama -- --list.