CALM, or why token-by-token generation hit a wall
CALM compresses K discrete tokens into one continuous vector, cutting generation steps to 1/K. Notes on the autoencoder, the likelihood-free head, and what it actually buys you.
From — Tencent WeChat AI × Tsinghua University Paper — arXiv:2510.27688
The wall
Every large language model today predicts one token at a time. Each token costs a full forward pass, so cost grows linearly — and with attention, worse — as sequences get longer.
Text representation has already moved from characters to subwords (BPE), but the discrete representation itself is running out of room:
- Vocabulary inflation: to cover more meaning, vocabularies grew from 32k to 256k. The final layer's compute and memory cost exploded with them.
- Low information density: each discrete token carries roughly 15–18 bits. The model has enormous compute and spends it on the labour-intensive job of predicting one low-information token at a time.
CALM's premise: stop picking from a vocabulary, and predict a point in continuous space instead.
The idea
An autoencoder compresses K discrete tokens into one high-dimensional continuous vector, so sequence length drops from T to T/K and the number of generation steps drops by the same factor.
It trades more semantic bandwidth per step for fewer steps.
| Discrete | CALM (continuous) | |
|---|---|---|
| Scaling | vocabulary grows exponentially | vector dimension grows smoothly |
| Objective | pick one of hundreds of thousands | predict coordinates in a continuous space |
| Sequence | long T | short T/K |
A useful mental image: conventional generation is searching a warehouse with hundreds of thousands of drawers. CALM draws coordinates.
Four pillars
1. A high-fidelity autoencoder. Compresses K tokens into 128+ dimensions at roughly 75M parameters of overhead, with reconstruction accuracy above 99.9%. Without this, everything downstream is garbage.
2. An energy-based head, no likelihood. Instead of softmax over a vocabulary, the model regresses the next target vector directly — a single step, not the dozens or hundreds of denoising iterations a diffusion model needs.
3. Three tricks for robustness in continuous space.
- Variational regularisation, to smooth folds and cliffs in the latent space
- KL clipping, to stop dimensions collapsing
- Double dropout, for extra noise tolerance
Together they build a smooth, elastic landscape where a small prediction error does not turn into gibberish.
4. Evaluation without probabilities.
- BrierLM: perplexity does not apply when there is no likelihood. CALM uses a Brier score (accuracy minus collision rate) as its yardstick, which correlates strongly and negatively with cross-entropy loss.
- Black-box temperature: with no probability denominator to tune, batch size N becomes the temperature knob. Small N is cold and deterministic, large N is hot and diverse. With rejection sampling, it behaves much like conventional temperature.
The loop
- Input: discrete tokens
- Compress: the encoder turns every K tokens into one vector
- Process: the transformer runs attention over a much shorter sequence
- Predict: the head emits the next continuous vector in one step
- Restore: the decoder expands it back into K tokens
Results
- With K=4: training compute down 44%, inference compute down 34%, generation steps from T to T/4
- Larger but cheaper: 371M parameters (slightly larger than the baseline) with equal or better quality and less total compute
- A new scaling axis: information density. As semantic bandwidth grows, CALM reaches a higher Brier score at lower compute — pushing past the old compute–quality frontier
- An unusual training curve: performance climbs sharply after ~200k steps and then overtakes conventional models. Learning continuous structure seems to need a long runway before it pays off
My take
The framing is the contribution: the vocabulary was never the point. Discrete tokens are a compression artifact of text, and once you accept that, predicting a continuous vector is the obvious next move.
Two things to watch before believing it replaces anything:
- Reconstruction fidelity is a hard ceiling. 99.9% sounds great until you multiply it across a long document.
- Sampling control moves to a different knob. Batch-size-as-temperature works, but it means your serving stack has to be redesigned around it.
For deployment on constrained hardware, a 4× reduction in generation steps is the most interesting number here — that is the kind of win that changes what fits on a device.