Quantization Speed Is Often a Kernel Problem

Every number here comes from arXiv 2605.05699v1 (May 7, 2026). The author is Mohamed Amine Bergach at Illumina; this is a single-author preprint, not peer-reviewed. We did not rerun it on an Apple M1. Where I add a judgement, I say so.

A scene

You are running a 1B model on a laptop; push the context from 256 to 4K and decoding slows as the KV cache (the K/V state kept during decoding) grows with every token. This essay is about two ledgers hidden in the usual advice — "quantize, but it slows you down" — and when to keep an fp16 fallback.

The implementation tax: the same algorithm written as four eager dispatches or two fused kernels. The deployment bill: an int4 cache against fp16, end to end, including where it breaks. By the end you should know which number is being compared to what, and four questions to ask of any benchmark that claims quantization is faster.

The paper runs the same int4 KV-cache method both ways. The eager version is 17% slower on a 360M model, measured against SRHT (a sign-randomized Hadamard rotation used as the control). Fuse the four steps into two kernels and SmolLM comes out 1% faster; on a 1.7B model it is 5% faster. Qwen is still 1% slower. End to end against fp16, Gemma-3 1B at a 256-token prefix has 8.2% lower latency.

Then the warning. On Qwen2.5-1.5B at a 2K prefix, the int4 cache emits two tokens and stops. fp16 emits 64.

First, the 17%

The method is rotate-then-quantize. Each K and V vector in the KV cache is rotated, compressed to int4 (a 4-bit integer), then dequantized and rotated back when attention reads it.

The rotation here is SRFT (sign-randomized FFT): multiply by a fixed random ±1 pattern, then take an FFT. The point is to spread out outliers. In a K/V vector, a few large values stretch the quantization scale and squash the rest toward zero. Rotate first and the coordinates look close to Gaussian; uniform quantization no longer has to survive a handful of spikes.

The problem is how the four steps are written. The eager version calls rotate, quantize, dequantize and inverse-rotate as four separate PyTorch/MPS operations. Every layer, every decoding step, pays four kernel dispatches.

Dispatch is the queueing and data movement around a GPU kernel launch. The arithmetic may take microseconds; four launches and four round trips through memory can cost more than the arithmetic itself.

The paper's measured tax: 17% higher latency on SmolLM2-360M, 16% on 1.7B, 12% on Qwen2.5-1.5B. The baseline is SRHT (sign-randomized Hadamard transform), another rotation, not fp16.

Why SRHT? Because SRFT and SRHT produce quantization quality within noise of each other; the difference is whether the rotation is an FFT or a Hadamard transform. Using SRHT as the control isolates the cost of the implementation from the cost of the rotation.

The paper puts the mechanism in one line, which I paraphrase:

The cost is dispatch, not compute. The underlying rfft and irfft calls take single-digit microseconds, but the four-step pipeline multiplies that overhead.

That is why the same algorithm turns from +17% into −1% — not because quantization improved, but because four launches became two (Figure 1).

Fusing four steps into two kernels is not enough

The fusion puts the first two steps and the last two into one Metal kernel each: forward does rotate plus quantize; inverse does dequantize plus inverse-rotate. Data stays in fast on-chip memory instead of round-tripping to device memory.

The same comparison flips for SmolLM, but Qwen stays 1% slower:

(Δ against SRHT; percentages rounded; 256-token prompt plus 128 generated tokens; Apple M1. Not against fp16.)

Why is the gap larger on 1.7B? The eager penalty is roughly the same (+17% vs +16%). The difference appears after fusion: 1.7B has a 2048-dim KV per layer, 360M only 320, so the fused kernel's fixed launch cost is spread across more work. That is why it moves from −1% to −5%.

At this point the story looks clean: four steps become two kernels, and 17% slower becomes 1% faster (on 360M). But if you stop here and conclude "quantization is always faster", you will miss the next section.

The end-to-end −8.2% against fp16 is not fusion alone. It needs three conditions.

First, the quantizer. With a single per-token scale (one scale per token), Qwen's 4-bit ΔPPL explodes to +7975. The author switches to per-channel λ plus per-group abs-max (one scale per channel, one abs-max per 32 values) and pulls the same number down to +638.6. That recipe runs inside the fused kernel.

Second, a dequant-prefix cache. The dequantized prefix is cached between steps instead of recomputed; the last 16 tokens stay in fp16 and the cache is invalidated when that window fills. Without it, Qwen at a 1024-token prefix is 2.4% slower than fp16; with it, it is 1.6% faster. The paper's text says −0.7%; its own ms/tok, 137.1 → 134.9, implies −1.6%. We use the table's numbers and note the discrepancy.

Third, the model architecture. Gemma-3 1B has sliding-window layers: most layers see a fixed window and stay in fp16; only a few full-attention layers (which see the whole prefix) pay the dequant cost. That is why its advantage survives to 4096 tokens. Qwen has 28 full-attention layers, so the cost grows linearly with prefix length.

Put those three back into the end-to-end table:

(Δ against fp16; Apple M1; 256-token prompt plus 128 generated tokens. Qwen at 2048 is not in the table; see the next section. Qwen 1024's −1.6% is the arithmetic from 137.1 → 134.9; the paper's text says −0.7%. We use the table's numbers.)

On memory: Qwen is 3.0–3.1×; Gemma's 19.5× is not a like-for-like comparison (an apples-to-apples comparison inside the full-attention layers is about 3.2×), so do not compare it directly with Qwen.

The −8.2% is the best case, at a 256-token prefix. The same model at 4096 is −3.2%; Qwen at 1024 is −1.6%. "Quantization is 8% faster" is not a constant; it is a curve that moves with prefix length.

Before you celebrate, check the quality bill

Speed is one ledger. Quality is the other.

Quantization quality is usually measured as perplexity (PPL). ΔPPL is the change from the fp16 baseline; lower is better.

The 4-bit per-token ΔPPL on Cosmopedia (8192 held-out tokens):

(The first two rows are the mean ± 1σ over three seeds; 1.7B ran one seed because of compute cost. 6-bit and 8-bit are lossless in this test.)

Rotation is not optional. At 135M, no rotation is +1.96; with rotation it is +0.38, roughly a 5× reduction. That is also why "quantization breaks" and "quantization is just slower" are often two sides of the same problem: if the rotation is wrong, the 4-bit error lands directly in PPL.

6-bit and 8-bit are lossless; 4-bit is the trade-off. 3-bit is worse in absolute terms; the paper does not sell 3-bit, and I would not start there.

Qwen's 4-bit is a different story. The Python reference's per-token ΔPPL is +7975.3; the Metal per-token version is +7727.9. With per-channel λ plus per-group g32, the fused version lands at +638.6 — a 12.5× improvement, in the same range as the Python reference's +654.6.

But +638.6 is still +638.6. A 12.5× improvement is not the same as deployable. The paper calls this "closing the catastrophe"; I think it is closer to moving a fire from the whole house into one room. If you enable 4-bit on Qwen, validate with task metrics, not just the relative PPL gain.

Gemma looks better: per-token +18.7, fused +3.6. The Python reference is −5.2; the +8.8 gap comes from the kernel's fp32 reduction and rounding order, not the algorithm. The author rules that out with two ablations.

It really breaks: Qwen at 2K emits two tokens

Back to the warning.

Qwen2.5-1.5B at a 2048-token prefix, int4, emits two tokens and hits EOS (end-of-sequence); fp16 emits 64.

There is more than one cause. The dequant-prefix cache plus int4 storage pushes peak memory from 4447 to 5527 MB (+24%); combined with quantization noise across 28 full-attention layers, the logits tip into an early EOS.

I think the design that makes 1024 faster is also part of what makes 2048 break. The prefix cache turns dequantization from "slower as the prefix grows" into a fixed cost, but it also occupies memory; on a full-attention stack like Qwen, both happen at once.

Why is Gemma fine? Its sliding-window layers stay in fp16; only a few full-attention layers pay the dequant cost. Longer prefixes grow the cost more slowly. Same cache, different architecture, different curve.

If you want to enable this cache on your own model, test 2K first. Not "does it generate", but "is the output still correct, and does it stop early". The author's fallback is explicit: above about 1.5K, go back to fp16.

Four questions for any quantization benchmark

The most useful part of the paper is not "Apple Silicon is fast". It is a way to split a benchmark into questions.

First, the platform. Unified memory or separate HBM? The paper says the conventional trade-off returns on GPUs with separate HBM and high dispatch overhead. The same kernel does not necessarily stay faster when you move it. Ask where the number was measured.

Second, the kernel. Fused, or four separate steps? If the benchmark does not say, the number may be dominated by dispatch tax. Ask: how many dispatches per layer per step?

Third, the prefix and the architecture. The −8% at 256 tokens is not the −3.2% at 4K; full-attention and sliding-window stacks curve differently. Ask for 256, 1K and 2K, not just the best chart.

Fourth, the quality. What is the ΔPPL baseline, and which quantizer? +638.6 is better than +7975, but it is still +638.6. Per-token and per-channel+group are different things; do not compare them as if they were the same.

What this does not prove

• One platform: Apple M1 only; no discrete-HBM GPU.
• Small models: 135M–1.7B; nothing above 7B.
• A single-author preprint, not peer-reviewed; we did not reproduce it.
• Gemma's 19.5× is not like-for-like; an apples-to-apples comparison inside the full-attention layers is about 3.2×.
• Qwen at 2K breaks; the author's own fallback is 1.5K.
• This is not the only path. It only addresses the cost of writing the four steps separately.

Conclusion: the question is not "is quantization fast?"

Quantization is not a switch. It is a stack: platform, kernel, quantizer, prefix length, architecture. Change one and the answer changes.

The two ledgers:

• Implementation tax. The same method, eager four steps vs two fused kernels: 11–21 percentage points.
• Deployment bill. int4 cache vs fp16: up to −8.2% at a short prefix, single digits at long prefixes (Qwen 1024 is −1.6%, Gemma 4096 is −3.2%), and it can stop early.

One thing to do tomorrow: run your model at 256, 1K and 2K, record ms/tok and output quality. If it is a full-attention stack, keep an fp16 fallback above 1.5K; 2K is the test point.

If you want help running those three prefixes on your own model, our AI readiness audit is exactly that: what can be handed to AI now, and what should wait.

The series so far: broken-on-arrival on the recipe, and this essay on the implementation. The earlier memory-trade-off essay is available in Chinese. The question moves from "should we quantize?" to "how is that kernel written?"

Paper: arXiv 2605.05699v1 (Mohamed Amine Bergach, Illumina). Previous essay: the quantized model may be broken.