[{"data":1,"prerenderedAt":2512},["ShallowReactive",2],{"post-quantization-kernel-dispatch":3,"site-info":1305,"related-en-quantization-kernel-dispatch":1400},{"zh":4,"en":705},{"id":5,"title":6,"author":7,"body":8,"cover":685,"coverAlt":686,"date":687,"description":688,"draft":689,"extension":690,"featured":689,"lang":691,"meta":692,"navigation":81,"ogImage":693,"path":694,"seo":695,"slug":696,"stem":697,"tags":698,"translationKey":696,"updated":703,"__hash__":704},"blog\u002Fblog\u002Fzh-TW\u002Fquantization-kernel-dispatch.md","量化會不會加速，取決於你怎麼寫 kernel","Jecision",{"type":9,"value":10,"toc":673},"minimark",[11,18,22,25,28,31,34,37,41,44,47,50,53,56,59,62,67,70,87,91,94,97,165,168,171,174,177,184,190,196,199,295,298,301,304,308,311,314,384,387,390,393,396,403,406,410,413,416,466,469,472,475,478,482,485,491,497,503,509,570,573,595,599,602,605,619,622,631,649,652],[12,13,14],"blockquote",{},[15,16,17],"p",{},"數字全部來自 arXiv 2605.05699v1（2026-05-07）。作者是 Illumina 的 Mohamed Amine Bergach；這是單人預印本，未同儕審查。我們沒有在 Apple M1 上獨立重跑；「我認為」的地方會標出來。",[19,20,21],"h2",{"id":21},"先講一個畫面",[15,23,24],{},"你在筆電上跑 1B 模型，context 從 256 拉到 4K，解碼變慢、KV cache（解碼時累積的 K\u002FV 快取）越長越吃記憶體。這篇要幫你看懂兩組數字：那個「慢 17%」跟「快 8%」分別在比什麼。還有下次看 benchmark 要問哪四件事，以及什麼時候該留 fp16 退路。",[15,26,27],{},"上網查，答案通常是「量化」。但下一句總是「會變慢」。",[15,29,30],{},"有一篇論文把這句話拆開了。同一份 int4（4-bit 整數）KV cache 量化，eager 版要四次 kernel 啟動（dispatch）。在 360M 模型上，這讓延遲比 SRHT（用 Hadamard 做隨機旋轉的對照法）高 17%。把四步融成兩顆 kernel 後，SmolLM 的同一組對照變成低 1%，1.7B 低 5%；Qwen 仍慢 1%。端到端對 fp16，Gemma-3 1B 在 256 前綴低 8.2%。",[15,32,33],{},"但先別高興。換到 Qwen2.5-1.5B 的 2K 前綴，它只吐兩個 token 就 EOS（提前結束）。這 8% 有平台、前綴、架構的前提，後面會把邊界畫出來。",[15,35,36],{},"這篇要把兩個常被混在一起的問題拆開。一個是實作稅：同一個演算法，eager 版四次 dispatch、融合版兩顆 kernel。另一個是部署帳：int4 cache 對 fp16，端到端到底快多少、什麼時候會壞。",[19,38,40],{"id":39},"先看那個慢-17","先看那個「慢 17%」",[15,42,43],{},"論文的方法叫 rotate-then-quantize。KV cache 裡的每個 K\u002FV 向量，先做一個隨機旋轉，再壓成 int4；attention 讀取時，再解量化、反旋轉回來。",[15,45,46],{},"這裡的旋轉是 SRFT（sign-randomized FFT）：先乘上一組固定的隨機正負號，再做 FFT。目的是打散離群值。模型的 K\u002FV 向量裡，少數很大的值會把量化尺度撐大，其他值就被壓成 0；先旋轉，讓每個座標看起來接近高斯分布，均勻量化才不會被少數大值拖垮。",[15,48,49],{},"問題在「四步」怎麼寫。eager 版把旋轉、量化、解量化、反旋轉拆成四個 PyTorch\u002FMPS 呼叫；每一層、每一個解碼步，就是四次 kernel dispatch。",[15,51,52],{},"dispatch 是什麼？就是 GPU 每啟動一顆 kernel 的排隊與搬資料成本。計算本身可能只有幾微秒，但四次啟動、四次把資料搬進搬出，加起來就變成主成本。",[15,54,55],{},"論文量到的代價是：SmolLM2-360M 延遲高 17%、1.7B 高 16%、Qwen2.5-1.5B 高 12%。基準是 SRHT（sign-randomized Hadamard transform），另一種隨機旋轉，不是 fp16。",[15,57,58],{},"為什麼拿 SRHT 當基準？因為 SRFT 和 SRHT 的量化品質在誤差內，差別只在「旋轉用 FFT 還是 Hadamard」。拿它對照，才能把「旋轉本身的成本」和「四步拆開的實作稅」分開。",[15,60,61],{},"論文有一句話是整段的骨架，我照原意翻：",[12,63,64],{},[15,65,66],{},"成本在 dispatch，不在計算。底層的 rfft 和 irfft 各只要個位數微秒，但四步分開呼叫，把這個開銷乘了上去。",[15,68,69],{},"這就是為什麼同樣的演算法，換一個寫法會從 +17% 變成 −1%。不是量化變好了，是四次排隊變成兩次（如圖 1）。",[71,72,75,76,75,83],"figure",{"className":73},[74],"article-figure","\n  ",[77,78],"video",{"src":79,"poster":80,"autoPlay":81,"loop":81,"muted":81,"playsInline":81,"ariaLabel":82},"\u002Fimages\u002Fblog\u002Fquantization-kernel-dispatch\u002Fkernel-fusion.mp4","\u002Fimages\u002Fblog\u002Fquantization-kernel-dispatch\u002Fkernel-fusion-poster.jpg",true,"左邊是 eager 版四次 kernel dispatch，右邊是融合成 forward 與 inverse 兩顆 kernel",[84,85,86],"figcaption",{},"圖 1：同一份量化，eager 版每層每步四次 dispatch；融合版把前兩步與後兩步各收成一顆 kernel。",[19,88,90],{"id":89},"把四步融成兩顆還不夠","把四步融成兩顆，還不夠",[15,92,93],{},"融合的做法，是把前兩步和後兩步各自塞進一顆 Metal kernel（Apple 的 GPU 程式）：forward 做旋轉加量化，inverse 做解量化加反旋轉。資料在中間不用回 device memory，留在 GPU 內部的快速記憶體。",[15,95,96],{},"同一組對照，SmolLM 反轉，Qwen 仍慢 1%：",[98,99,100,119],"table",{},[101,102,103],"thead",{},[104,105,106,110,113,116],"tr",{},[107,108,109],"th",{},"模型",[107,111,112],{},"SRHT 基準",[107,114,115],{},"eager Δ",[107,117,118],{},"融合 SRFT（Δ）",[120,121,122,137,151],"tbody",{},[104,123,124,128,131,134],{},[125,126,127],"td",{},"SmolLM2-360M",[125,129,130],{},"37.4 ms\u002Ftok",[125,132,133],{},"+17%",[125,135,136],{},"37.0 ms\u002Ftok（−1%）",[104,138,139,142,145,148],{},[125,140,141],{},"SmolLM2-1.7B",[125,143,144],{},"222.2 ms\u002Ftok",[125,146,147],{},"+16%",[125,149,150],{},"211.9 ms\u002Ftok（−5%）",[104,152,153,156,159,162],{},[125,154,155],{},"Qwen2.5-1.5B",[125,157,158],{},"84.9 ms\u002Ftok",[125,160,161],{},"+12%",[125,163,164],{},"85.7 ms\u002Ftok（+1%）",[15,166,167],{},"（Δ 相對 SRHT，百分比取整；256 token 前綴加 128 token 生成；Apple M1。不是對 fp16。）",[15,169,170],{},"1.7B 的差距比 360M 大。eager 的罰則在兩個模型上差不多（+17% 對 +16%）。差別在融合後：1.7B 每層的 KV 維度是 2048，360M 只有 320。融合 kernel 的固定啟動成本被更多工作攤薄，所以它從 −1% 變成 −5%。",[15,172,173],{},"到這裡，故事看起來很乾淨：四步變兩顆，慢 17% 變快 1%（360M）。但如果你看到這裡就下結論「量化一定快」，會漏掉下一段。",[15,175,176],{},"端到端對 fp16 的 −8.2%，不是只有融合。它還需要三個條件。",[15,178,179,183],{},[180,181,182],"strong",{},"第一，量化配方。"," 4-bit 如果只用 per-token（每個 token 一個尺度），Qwen 的 ΔPPL 會炸到 +7975。作者改成 per-channel λ 加 per-group abs-max（每個 channel 一個縮放係數，每 32 個值一組取絕對值最大），把同一個數字壓到 +638.6。這個配方是在同一顆融合 kernel 裡做的。",[15,185,186,189],{},[180,187,188],{},"第二，dequant-prefix cache。"," 解量化過的前綴先存起來，不要每一步重算；最近 16 個 token 保留 fp16，只有這個視窗滿了才失效。沒有它，Qwen 在 1024 前綴是 +2.4% 慢；有了它，才變成 −1.6% 快（表內回算；論文文字寫 −0.7%）。",[15,191,192,195],{},[180,193,194],{},"第三，模型架構。"," Gemma-3 1B 有 sliding-window 層：大部分層只看固定視窗、用 fp16，只有少數 full-attention 層（看完整前綴）要付 dequant 成本。所以它的優勢可以撐到 4096；Qwen 是 28 層 full-attention，成本隨前綴線性成長。",[15,197,198],{},"把這三件事放回去，端到端的表長這樣：",[98,200,201,216],{},[101,202,203],{},[104,204,205,207,210,213],{},[107,206,109],{},[107,208,209],{},"前綴",[107,211,212],{},"fp16 → int4（ms\u002Ftok）",[107,214,215],{},"延遲差",[120,217,218,232,245,258,271,283],{},[104,219,220,223,226,229],{},[125,221,222],{},"Gemma-3 1B",[125,224,225],{},"256",[125,227,228],{},"71.2 → 65.4",[125,230,231],{},"−8.2%",[104,233,234,236,239,242],{},[125,235,222],{},[125,237,238],{},"1024",[125,240,241],{},"83.3 → 82.5",[125,243,244],{},"−0.9%",[104,246,247,249,252,255],{},[125,248,222],{},[125,250,251],{},"2048",[125,253,254],{},"129.5 → 121.3",[125,256,257],{},"−6.3%",[104,259,260,262,265,268],{},[125,261,222],{},[125,263,264],{},"4096",[125,266,267],{},"188.5 → 182.4",[125,269,270],{},"−3.2%",[104,272,273,275,277,280],{},[125,274,155],{},[125,276,225],{},[125,278,279],{},"90.7 → 88.3",[125,281,282],{},"−2.6%",[104,284,285,287,289,292],{},[125,286,155],{},[125,288,238],{},[125,290,291],{},"137.1 → 134.9",[125,293,294],{},"−1.6%",[15,296,297],{},"（Δ 相對 fp16；Apple M1；256 token 前綴加 128 token 生成。Qwen 2048 不在表裡，原因見下一節。Qwen 1024 的 −1.6% 是用表內 137.1 → 134.9 回算；論文文字寫 −0.7%，我們以表內數字為準。）",[15,299,300],{},"記憶體方面，Qwen 是 3.0–3.1×；Gemma 的 19.5× 不是同層比較（full-attention 層內約 3.2×），不能直接跟 Qwen 比。",[15,302,303],{},"−8.2% 是 256 前綴的最佳情況。同一個模型拉到 4096，只剩 −3.2%。Qwen 拉到 1024，只剩 −1.6%。「量化快 8%」不是一個常數，是一條會隨前綴長度變化的曲線。",[19,305,307],{"id":306},"快之前先看它掉了多少","快之前，先看它掉了多少",[15,309,310],{},"速度之外，還有一本帳：品質。量化對品質的影響，通常看困惑度（perplexity，PPL）的變化；ΔPPL 是量化後減 fp16 的差。",[15,312,313],{},"4-bit per-token 的 ΔPPL 如下（Cosmopedia，8192 個 held-out token）：",[98,315,316,334],{},[101,317,318],{},[104,319,320,322,325,328,331],{},[107,321,109],{},[107,323,324],{},"FP16 PPL",[107,326,327],{},"不旋轉（identity）",[107,329,330],{},"SRHT",[107,332,333],{},"SRFT",[120,335,336,353,369],{},[104,337,338,341,344,347,350],{},[125,339,340],{},"SmolLM2-135M",[125,342,343],{},"6.61",[125,345,346],{},"+1.96 ± 0.15",[125,348,349],{},"+0.34 ± 0.01",[125,351,352],{},"+0.38 ± 0.06",[104,354,355,357,360,363,366],{},[125,356,127],{},[125,358,359],{},"4.87",[125,361,362],{},"+0.97 ± 0.08",[125,364,365],{},"+0.20 ± 0.03",[125,367,368],{},"+0.21 ± 0.02",[104,370,371,373,376,379,382],{},[125,372,141],{},[125,374,375],{},"3.77",[125,377,378],{},"+1.04",[125,380,381],{},"+0.16",[125,383,381],{},[15,385,386],{},"（前兩列是 3 個 seed 的平均 ± 1σ；1.7B 因為計算成本只跑 1 個 seed。6-bit 與 8-bit 在這組測試裡無損。）",[15,388,389],{},"旋轉不是可選項。135M 不旋轉是 +1.96，旋轉後是 +0.38；差了大約 5 倍。這也解釋了為什麼「量化會壞」和「量化只是變慢」常常是同一件事的兩面：如果你的旋轉沒做對，4-bit 的誤差會直接反映在 PPL 上。",[15,391,392],{},"6-bit 和 8-bit 無損，4-bit 才是取捨點。3-bit 的絕對誤差更大，但這篇沒有把 3-bit 當賣點，我也不建議讀者從 3-bit 開始。",[15,394,395],{},"Qwen 的 4-bit 是另一個故事。Python reference 的 per-token 是 +7975.3，Metal 的 per-token 是 +7727.9；加上 per-channel λ 加 per-group g32 之後，融合版是 +638.6。這是 12.5 倍的改善，跟 Python reference 的 +654.6 同級。",[15,397,398,399,402],{},"但 +638.6 仍然是 +638.6。",[180,400,401],{},"12.5 倍的改善，不等於可以上線。"," 論文把這個叫做「關掉災難」；我認為，它比較像是把火從整棟房子縮到一個房間，離安全還很遠。如果你要在 Qwen 上開 4-bit，請用任務指標驗收，不要只看 PPL 的相對改善。",[15,404,405],{},"Gemma 的結果比較好：per-token +18.7，融合後 +3.6。Python reference 是 −5.2，差距 +8.8 來自 kernel 的 fp32 reduction 與 rounding ordering，不是演算法。這一點作者自己做了兩個 ablation 排除。",[19,407,409],{"id":408},"它真的會壞qwen-2k-只吐兩個-token","它真的會壞：Qwen 2K 只吐兩個 token",[15,411,412],{},"現在回到那個「先別高興」。",[15,414,415],{},"Qwen2.5-1.5B 在 2048 前綴，int4 只生成 2 個 token 就 EOS；fp16 是 64 個。",[98,417,418,431],{},[101,419,420],{},[104,421,422,425,428],{},[107,423,424],{},"項目",[107,426,427],{},"fp16",[107,429,430],{},"int4",[120,432,433,444,455],{},[104,434,435,438,441],{},[125,436,437],{},"生成 token 數（2048 前綴）",[125,439,440],{},"64",[125,442,443],{},"2（EOS）",[104,445,446,449,452],{},[125,447,448],{},"峰值記憶體",[125,450,451],{},"4447 MB",[125,453,454],{},"5527 MB",[104,456,457,460,463],{},[125,458,459],{},"作者退路",[125,461,462],{},"—",[125,464,465],{},"約 1.5K 以上退回 fp16",[15,467,468],{},"原因不只一個。dequant-prefix cache 加上 int4 storage，讓峰值記憶體從 4447 拉到 5527 MB（+24%）；再加上 28 層 full-attention 的量化噪聲累積，logits 被推去提前 EOS。",[15,470,471],{},"我認為，讓 1024 變快的那個設計，也是讓 2048 變壞的設計之一。prefix cache 把解量化成本從「前綴越長越慢」攤成固定成本，但它同時佔記憶體；在 Qwen 這種全注意力堆疊上，兩件事一起發生，就爆了。",[15,473,474],{},"Gemma 為什麼沒事？它的 sliding-window 層用 fp16，只有少數 full-attention 層付 dequant 成本；所以前綴拉長，成本成長得比較慢。架構不同，同一份量化 cache 的結果就不同。",[15,476,477],{},"如果你要在自己的模型上開這個 cache，先測 2K。不是測「能不能生成」，是測「生成出來的東西對不對、會不會提前停」。作者給的退路很明確：超過約 1.5K，就退回 fp16。",[19,479,481],{"id":480},"看-benchmark-要問的四件事","看 benchmark 要問的四件事",[15,483,484],{},"這篇論文最有用的地方，不是「Apple Silicon 比較快」，是它把一張 benchmark 拆成幾個要問的問題。",[15,486,487,490],{},[180,488,489],{},"第一，平台。"," 這是 unified memory（CPU 與 GPU 共用記憶體）還是獨立 HBM（顯示卡自己的記憶體）？論文說：在獨立 HBM、dispatch 開銷高的 GPU 上，慣常的取捨會回復。也就是說，同樣的 kernel 搬過去，不一定還快。看到「量化加速」先問跑在哪。",[15,492,493,496],{},[180,494,495],{},"第二，kernel。"," 這是 fused 還是四步分開？如果 benchmark 沒寫，數字可能被 dispatch 稅主導。你可以直接問：一層一步幾次 dispatch？",[15,498,499,502],{},[180,500,501],{},"第三，前綴與架構。"," 256 前綴的 −8% 不是 4K 的 −3.2%；full-attention 堆疊和 sliding-window 的曲線也不一樣。要求對方給 256、1K、2K 三組數字，不要只看一張最好的圖。",[15,504,505,508],{},[180,506,507],{},"第四，品質。"," ΔPPL 的 baseline 與配方是什麼？+638.6 比 +7975 好，但仍然是 +638.6。per-token 和 per-channel+group 是兩種不同的東西，不能拿來互相比。",[98,510,511,524],{},[101,512,513],{},[104,514,515,518,521],{},[107,516,517],{},"問題",[107,519,520],{},"為什麼要問",[107,522,523],{},"怎麼查",[120,525,526,537,548,559],{},[104,527,528,531,534],{},[125,529,530],{},"平台是 unified memory 還是獨立 HBM？",[125,532,533],{},"取捨是否反轉的邊界",[125,535,536],{},"看硬體規格；要求分平台數字",[104,538,539,542,545],{},[125,540,541],{},"Kernel 是 fused 還是四步分開？",[125,543,544],{},"相對 SRHT 可能慢 12–17%",[125,546,547],{},"看 benchmark 是否寫「4 dispatches」或「fused」",[104,549,550,553,556],{},[125,551,552],{},"前綴多長、模型是 full-attention 還是 sliding-window？",[125,554,555],{},"長 context 可能失效",[125,557,558],{},"要求 256／1K／2K 三組；看架構",[104,560,561,564,567],{},[125,562,563],{},"ΔPPL 的 baseline 與量化配方是什麼？",[125,565,566],{},"+638.6 比 +7975 好，但仍是 +638.6",[125,568,569],{},"看 baseline、per-token 或 per-channel+group",[19,571,572],{"id":572},"別過度解讀",[574,575,576,580,583,586,589,592],"ul",{},[577,578,579],"li",{},"單一平台：只有 Apple M1；獨立 HBM 的 GPU 沒測。",[577,581,582],{},"模型都是小模型（135M–1.7B），沒有 7B 以上。",[577,584,585],{},"單人預印本，未同儕審查；我們沒有獨立複現。",[577,587,588],{},"Gemma 的 19.5× 不是同層比較；full-attention 層內約 3.2×。",[577,590,591],{},"Qwen 2K 會壞；作者自己寫了 1.5K 的退路。",[577,593,594],{},"這篇不是說「融合是唯一解」；它只處理四步拆開這一種成本。",[19,596,598],{"id":597},"結論問題不是量化快不快","結論：問題不是「量化快不快」",[15,600,601],{},"量化不是一個開關，是一條工程鏈：平台、kernel、量化配方、前綴長度、模型架構，任何一個換了，答案就換了。",[15,603,604],{},"這篇的兩本帳，可以這樣記：",[574,606,607,613],{},[577,608,609,612],{},[180,610,611],{},"實作稅"," 是同一份方法，eager 四步對融合兩顆，差距 11–21 個百分點。",[577,614,615,618],{},[180,616,617],{},"部署帳"," 是 int4 cache 對 fp16，短前綴最多 −8.2%，長前綴會縮到個位數百分比（Qwen 1024 是 −1.6%，Gemma 4096 是 −3.2%），而且可能提前 EOS。",[15,620,621],{},"明天可以先做一件事：用你的模型跑 256、1K、2K 三個前綴，記錄 ms\u002Ftok 和輸出品質；如果模型是 full-attention 堆疊，1.5K 以上先留 fp16 退路；2K 是測試點。",[15,623,624,625,630],{},"如果你要的是有人陪你把自己的模型跑過這三個前綴，我們的",[626,627,629],"a",{"href":628},"\u002Fservices","企業 AI 導入健檢","做的就是這件事。先判斷哪些工作現在就能交給 AI，哪些還不成熟。",[15,632,633,634,641,642,648],{},"量化系列走到這裡，三篇的弧線是：",[626,635,637],{"href":636},"\u002Fblog\u002Fstrata-125b-on-12gb",[638,639,640],"code",{},"strata"," 講取捨，",[626,643,645],{"href":644},"\u002Fblog\u002Fbroken-on-arrival-quantized-models",[638,646,647],{},"broken-on-arrival"," 講配方，這篇講實作。同一個問題，從「要不要量化」走到「怎麼寫那個 kernel」。",[650,651],"hr",{},[15,653,654],{},[655,656,657,658,664,665,668,669,672],"em",{},"論文：",[626,659,663],{"href":660,"rel":661},"https:\u002F\u002Farxiv.org\u002Fabs\u002F2605.05699",[662],"nofollow","arXiv 2605.05699v1","（Mohamed Amine Bergach, Illumina）。前作：",[626,666,667],{"href":636},"125B 模型跑在 12GB 顯卡","、",[626,670,671],{"href":644},"你下載的量化模型可能是壞的","。",{"title":674,"searchDepth":675,"depth":675,"links":676},"",2,[677,678,679,680,681,682,683,684],{"id":21,"depth":675,"text":21},{"id":39,"depth":675,"text":40},{"id":89,"depth":675,"text":90},{"id":306,"depth":675,"text":307},{"id":408,"depth":675,"text":409},{"id":480,"depth":675,"text":481},{"id":572,"depth":675,"text":572},{"id":597,"depth":675,"text":598},"\u002Fimages\u002Fblog\u002Fquantization-kernel-dispatch.svg","標題「量化會不會加速，取決於你怎麼寫 kernel」的技術筆記封面","2026-11-05","同一份 int4 KV cache 量化，eager 版相對 SRHT 慢 12–17%，把四步融成兩顆 Metal kernel 後追平；端到端對 fp16，Gemma 短前綴快 8.2%，但 Qwen 2K 前綴只吐兩個 token。差別在四步拆開呼叫，還是融成兩顆——以及平台、前綴與架構的前提。",false,"md","zh-TW",{},"\u002Fimages\u002Fblog\u002Fquantization-kernel-dispatch.jpg","\u002Fblog\u002Fzh-tw\u002Fquantization-kernel-dispatch",{"title":6,"description":688},"quantization-kernel-dispatch","blog\u002Fzh-TW\u002Fquantization-kernel-dispatch",[699,700,701,702],"quantization","kv-cache","apple-silicon","inference",null,"nxxvpD8Ugm7w7mX3s1VM_GxUtkB58VVj0Hd2cm51owY",{"id":706,"title":707,"author":7,"body":708,"cover":1294,"coverAlt":1295,"date":687,"description":1296,"draft":689,"extension":690,"featured":689,"lang":1297,"meta":1298,"navigation":81,"ogImage":1299,"path":1300,"seo":1301,"slug":696,"stem":1302,"tags":1303,"translationKey":696,"updated":703,"__hash__":1304},"blog\u002Fblog\u002Fen\u002Fquantization-kernel-dispatch.md","Quantization Speed Is Often a Kernel Problem",{"type":9,"value":709,"toc":1284},[710,715,719,722,725,728,731,735,738,741,744,747,750,753,756,761,764,775,779,782,785,838,841,844,847,850,856,862,868,871,950,953,956,959,963,966,969,972,1027,1030,1033,1036,1039,1042,1045,1049,1052,1055,1099,1102,1105,1108,1111,1115,1118,1124,1130,1136,1142,1203,1207,1227,1231,1234,1237,1251,1254,1262,1269,1271],[12,711,712],{},[15,713,714],{},"Every number here comes from arXiv 2605.05699v1 (May 7, 2026). The author is Mohamed Amine Bergach at Illumina; this is a single-author preprint, not peer-reviewed. We did not rerun it on an Apple M1. Where I add a judgement, I say so.",[19,716,718],{"id":717},"a-scene","A scene",[15,720,721],{},"You are running a 1B model on a laptop; push the context from 256 to 4K and decoding slows as the KV cache (the K\u002FV state kept during decoding) grows with every token. This essay is about two ledgers hidden in the usual advice — \"quantize, but it slows you down\" — and when to keep an fp16 fallback.",[15,723,724],{},"The implementation tax: the same algorithm written as four eager dispatches or two fused kernels. The deployment bill: an int4 cache against fp16, end to end, including where it breaks. By the end you should know which number is being compared to what, and four questions to ask of any benchmark that claims quantization is faster.",[15,726,727],{},"The paper runs the same int4 KV-cache method both ways. The eager version is 17% slower on a 360M model, measured against SRHT (a sign-randomized Hadamard rotation used as the control). Fuse the four steps into two kernels and SmolLM comes out 1% faster; on a 1.7B model it is 5% faster. Qwen is still 1% slower. End to end against fp16, Gemma-3 1B at a 256-token prefix has 8.2% lower latency.",[15,729,730],{},"Then the warning. On Qwen2.5-1.5B at a 2K prefix, the int4 cache emits two tokens and stops. fp16 emits 64.",[19,732,734],{"id":733},"first-the-17","First, the 17%",[15,736,737],{},"The method is rotate-then-quantize. Each K and V vector in the KV cache is rotated, compressed to int4 (a 4-bit integer), then dequantized and rotated back when attention reads it.",[15,739,740],{},"The rotation here is SRFT (sign-randomized FFT): multiply by a fixed random ±1 pattern, then take an FFT. The point is to spread out outliers. In a K\u002FV vector, a few large values stretch the quantization scale and squash the rest toward zero. Rotate first and the coordinates look close to Gaussian; uniform quantization no longer has to survive a handful of spikes.",[15,742,743],{},"The problem is how the four steps are written. The eager version calls rotate, quantize, dequantize and inverse-rotate as four separate PyTorch\u002FMPS operations. Every layer, every decoding step, pays four kernel dispatches.",[15,745,746],{},"Dispatch is the queueing and data movement around a GPU kernel launch. The arithmetic may take microseconds; four launches and four round trips through memory can cost more than the arithmetic itself.",[15,748,749],{},"The paper's measured tax: 17% higher latency on SmolLM2-360M, 16% on 1.7B, 12% on Qwen2.5-1.5B. The baseline is SRHT (sign-randomized Hadamard transform), another rotation, not fp16.",[15,751,752],{},"Why SRHT? Because SRFT and SRHT produce quantization quality within noise of each other; the difference is whether the rotation is an FFT or a Hadamard transform. Using SRHT as the control isolates the cost of the implementation from the cost of the rotation.",[15,754,755],{},"The paper puts the mechanism in one line, which I paraphrase:",[12,757,758],{},[15,759,760],{},"The cost is dispatch, not compute. The underlying rfft and irfft calls take single-digit microseconds, but the four-step pipeline multiplies that overhead.",[15,762,763],{},"That is why the same algorithm turns from +17% into −1% — not because quantization improved, but because four launches became two (Figure 1).",[71,765,75,767,75,772],{"className":766},[74],[77,768],{"src":769,"poster":770,"autoPlay":81,"loop":81,"muted":81,"playsInline":81,"ariaLabel":771},"\u002Fimages\u002Fblog\u002Fen\u002Fquantization-kernel-dispatch\u002Fkernel-fusion.mp4","\u002Fimages\u002Fblog\u002Fen\u002Fquantization-kernel-dispatch\u002Fkernel-fusion-poster.jpg","Left: four eager kernel dispatches. Right: two fused kernels, forward and inverse.",[84,773,774],{},"Figure 1: The same quantization, eager with four dispatches per layer per step; fused into a forward and an inverse kernel.",[19,776,778],{"id":777},"fusing-four-steps-into-two-kernels-is-not-enough","Fusing four steps into two kernels is not enough",[15,780,781],{},"The fusion puts the first two steps and the last two into one Metal kernel each: forward does rotate plus quantize; inverse does dequantize plus inverse-rotate. Data stays in fast on-chip memory instead of round-tripping to device memory.",[15,783,784],{},"The same comparison flips for SmolLM, but Qwen stays 1% slower:",[98,786,787,803],{},[101,788,789],{},[104,790,791,794,797,800],{},[107,792,793],{},"Model",[107,795,796],{},"SRHT baseline",[107,798,799],{},"Eager Δ",[107,801,802],{},"Fused SRFT (Δ)",[120,804,805,816,827],{},[104,806,807,809,811,813],{},[125,808,127],{},[125,810,130],{},[125,812,133],{},[125,814,815],{},"37.0 ms\u002Ftok (−1%)",[104,817,818,820,822,824],{},[125,819,141],{},[125,821,144],{},[125,823,147],{},[125,825,826],{},"211.9 ms\u002Ftok (−5%)",[104,828,829,831,833,835],{},[125,830,155],{},[125,832,158],{},[125,834,161],{},[125,836,837],{},"85.7 ms\u002Ftok (+1%)",[15,839,840],{},"(Δ against SRHT; percentages rounded; 256-token prompt plus 128 generated tokens; Apple M1. Not against fp16.)",[15,842,843],{},"Why is the gap larger on 1.7B? The eager penalty is roughly the same (+17% vs +16%). The difference appears after fusion: 1.7B has a 2048-dim KV per layer, 360M only 320, so the fused kernel's fixed launch cost is spread across more work. That is why it moves from −1% to −5%.",[15,845,846],{},"At this point the story looks clean: four steps become two kernels, and 17% slower becomes 1% faster (on 360M). But if you stop here and conclude \"quantization is always faster\", you will miss the next section.",[15,848,849],{},"The end-to-end −8.2% against fp16 is not fusion alone. It needs three conditions.",[15,851,852,855],{},[180,853,854],{},"First, the quantizer."," With a single per-token scale (one scale per token), Qwen's 4-bit ΔPPL explodes to +7975. The author switches to per-channel λ plus per-group abs-max (one scale per channel, one abs-max per 32 values) and pulls the same number down to +638.6. That recipe runs inside the fused kernel.",[15,857,858,861],{},[180,859,860],{},"Second, a dequant-prefix cache."," The dequantized prefix is cached between steps instead of recomputed; the last 16 tokens stay in fp16 and the cache is invalidated when that window fills. Without it, Qwen at a 1024-token prefix is 2.4% slower than fp16; with it, it is 1.6% faster. The paper's text says −0.7%; its own ms\u002Ftok, 137.1 → 134.9, implies −1.6%. We use the table's numbers and note the discrepancy.",[15,863,864,867],{},[180,865,866],{},"Third, the model architecture."," Gemma-3 1B has sliding-window layers: most layers see a fixed window and stay in fp16; only a few full-attention layers (which see the whole prefix) pay the dequant cost. That is why its advantage survives to 4096 tokens. Qwen has 28 full-attention layers, so the cost grows linearly with prefix length.",[15,869,870],{},"Put those three back into the end-to-end table:",[98,872,873,888],{},[101,874,875],{},[104,876,877,879,882,885],{},[107,878,793],{},[107,880,881],{},"Prefix",[107,883,884],{},"fp16 → int4 (ms\u002Ftok)",[107,886,887],{},"Latency Δ",[120,889,890,900,910,920,930,940],{},[104,891,892,894,896,898],{},[125,893,222],{},[125,895,225],{},[125,897,228],{},[125,899,231],{},[104,901,902,904,906,908],{},[125,903,222],{},[125,905,238],{},[125,907,241],{},[125,909,244],{},[104,911,912,914,916,918],{},[125,913,222],{},[125,915,251],{},[125,917,254],{},[125,919,257],{},[104,921,922,924,926,928],{},[125,923,222],{},[125,925,264],{},[125,927,267],{},[125,929,270],{},[104,931,932,934,936,938],{},[125,933,155],{},[125,935,225],{},[125,937,279],{},[125,939,282],{},[104,941,942,944,946,948],{},[125,943,155],{},[125,945,238],{},[125,947,291],{},[125,949,294],{},[15,951,952],{},"(Δ against fp16; Apple M1; 256-token prompt plus 128 generated tokens. Qwen at 2048 is not in the table; see the next section. Qwen 1024's −1.6% is the arithmetic from 137.1 → 134.9; the paper's text says −0.7%. We use the table's numbers.)",[15,954,955],{},"On memory: Qwen is 3.0–3.1×; Gemma's 19.5× is not a like-for-like comparison (an apples-to-apples comparison inside the full-attention layers is about 3.2×), so do not compare it directly with Qwen.",[15,957,958],{},"The −8.2% is the best case, at a 256-token prefix. The same model at 4096 is −3.2%; Qwen at 1024 is −1.6%. \"Quantization is 8% faster\" is not a constant; it is a curve that moves with prefix length.",[19,960,962],{"id":961},"before-you-celebrate-check-the-quality-bill","Before you celebrate, check the quality bill",[15,964,965],{},"Speed is one ledger. Quality is the other.",[15,967,968],{},"Quantization quality is usually measured as perplexity (PPL). ΔPPL is the change from the fp16 baseline; lower is better.",[15,970,971],{},"The 4-bit per-token ΔPPL on Cosmopedia (8192 held-out tokens):",[98,973,974,989],{},[101,975,976],{},[104,977,978,980,982,985,987],{},[107,979,793],{},[107,981,324],{},[107,983,984],{},"No rotation (identity)",[107,986,330],{},[107,988,333],{},[120,990,991,1003,1015],{},[104,992,993,995,997,999,1001],{},[125,994,340],{},[125,996,343],{},[125,998,346],{},[125,1000,349],{},[125,1002,352],{},[104,1004,1005,1007,1009,1011,1013],{},[125,1006,127],{},[125,1008,359],{},[125,1010,362],{},[125,1012,365],{},[125,1014,368],{},[104,1016,1017,1019,1021,1023,1025],{},[125,1018,141],{},[125,1020,375],{},[125,1022,378],{},[125,1024,381],{},[125,1026,381],{},[15,1028,1029],{},"(The first two rows are the mean ± 1σ over three seeds; 1.7B ran one seed because of compute cost. 6-bit and 8-bit are lossless in this test.)",[15,1031,1032],{},"Rotation is not optional. At 135M, no rotation is +1.96; with rotation it is +0.38, roughly a 5× reduction. That is also why \"quantization breaks\" and \"quantization is just slower\" are often two sides of the same problem: if the rotation is wrong, the 4-bit error lands directly in PPL.",[15,1034,1035],{},"6-bit and 8-bit are lossless; 4-bit is the trade-off. 3-bit is worse in absolute terms; the paper does not sell 3-bit, and I would not start there.",[15,1037,1038],{},"Qwen's 4-bit is a different story. The Python reference's per-token ΔPPL is +7975.3; the Metal per-token version is +7727.9. With per-channel λ plus per-group g32, the fused version lands at +638.6 — a 12.5× improvement, in the same range as the Python reference's +654.6.",[15,1040,1041],{},"But +638.6 is still +638.6. A 12.5× improvement is not the same as deployable. The paper calls this \"closing the catastrophe\"; I think it is closer to moving a fire from the whole house into one room. If you enable 4-bit on Qwen, validate with task metrics, not just the relative PPL gain.",[15,1043,1044],{},"Gemma looks better: per-token +18.7, fused +3.6. The Python reference is −5.2; the +8.8 gap comes from the kernel's fp32 reduction and rounding order, not the algorithm. The author rules that out with two ablations.",[19,1046,1048],{"id":1047},"it-really-breaks-qwen-at-2k-emits-two-tokens","It really breaks: Qwen at 2K emits two tokens",[15,1050,1051],{},"Back to the warning.",[15,1053,1054],{},"Qwen2.5-1.5B at a 2048-token prefix, int4, emits two tokens and hits EOS (end-of-sequence); fp16 emits 64.",[98,1056,1057,1068],{},[101,1058,1059],{},[104,1060,1061,1064,1066],{},[107,1062,1063],{},"Item",[107,1065,427],{},[107,1067,430],{},[120,1069,1070,1080,1089],{},[104,1071,1072,1075,1077],{},[125,1073,1074],{},"Generated tokens (2048 prefix)",[125,1076,440],{},[125,1078,1079],{},"2 (EOS)",[104,1081,1082,1085,1087],{},[125,1083,1084],{},"Peak memory",[125,1086,451],{},[125,1088,454],{},[104,1090,1091,1094,1096],{},[125,1092,1093],{},"Author's fallback",[125,1095,462],{},[125,1097,1098],{},"fp16 above about 1.5K",[15,1100,1101],{},"There is more than one cause. The dequant-prefix cache plus int4 storage pushes peak memory from 4447 to 5527 MB (+24%); combined with quantization noise across 28 full-attention layers, the logits tip into an early EOS.",[15,1103,1104],{},"I think the design that makes 1024 faster is also part of what makes 2048 break. The prefix cache turns dequantization from \"slower as the prefix grows\" into a fixed cost, but it also occupies memory; on a full-attention stack like Qwen, both happen at once.",[15,1106,1107],{},"Why is Gemma fine? Its sliding-window layers stay in fp16; only a few full-attention layers pay the dequant cost. Longer prefixes grow the cost more slowly. Same cache, different architecture, different curve.",[15,1109,1110],{},"If you want to enable this cache on your own model, test 2K first. Not \"does it generate\", but \"is the output still correct, and does it stop early\". The author's fallback is explicit: above about 1.5K, go back to fp16.",[19,1112,1114],{"id":1113},"four-questions-for-any-quantization-benchmark","Four questions for any quantization benchmark",[15,1116,1117],{},"The most useful part of the paper is not \"Apple Silicon is fast\". It is a way to split a benchmark into questions.",[15,1119,1120,1123],{},[180,1121,1122],{},"First, the platform."," Unified memory or separate HBM? The paper says the conventional trade-off returns on GPUs with separate HBM and high dispatch overhead. The same kernel does not necessarily stay faster when you move it. Ask where the number was measured.",[15,1125,1126,1129],{},[180,1127,1128],{},"Second, the kernel."," Fused, or four separate steps? If the benchmark does not say, the number may be dominated by dispatch tax. Ask: how many dispatches per layer per step?",[15,1131,1132,1135],{},[180,1133,1134],{},"Third, the prefix and the architecture."," The −8% at 256 tokens is not the −3.2% at 4K; full-attention and sliding-window stacks curve differently. Ask for 256, 1K and 2K, not just the best chart.",[15,1137,1138,1141],{},[180,1139,1140],{},"Fourth, the quality."," What is the ΔPPL baseline, and which quantizer? +638.6 is better than +7975, but it is still +638.6. Per-token and per-channel+group are different things; do not compare them as if they were the same.",[98,1143,1144,1157],{},[101,1145,1146],{},[104,1147,1148,1151,1154],{},[107,1149,1150],{},"Question",[107,1152,1153],{},"Why it matters",[107,1155,1156],{},"How to check",[120,1158,1159,1170,1181,1192],{},[104,1160,1161,1164,1167],{},[125,1162,1163],{},"Unified memory or separate HBM?",[125,1165,1166],{},"Where the trade-off flips",[125,1168,1169],{},"Hardware spec; ask for per-platform numbers",[104,1171,1172,1175,1178],{},[125,1173,1174],{},"Fused kernel or four dispatches?",[125,1176,1177],{},"Dispatch tax can be 12–17% vs SRHT",[125,1179,1180],{},"Does the benchmark say \"4 dispatches\" or \"fused\"?",[104,1182,1183,1186,1189],{},[125,1184,1185],{},"Prefix length and attention pattern?",[125,1187,1188],{},"Long context may break",[125,1190,1191],{},"Ask for 256 \u002F 1K \u002F 2K; check full-attention vs sliding-window",[104,1193,1194,1197,1200],{},[125,1195,1196],{},"ΔPPL baseline and quantizer?",[125,1198,1199],{},"+638.6 is better than +7975 but still +638.6",[125,1201,1202],{},"Check the baseline; per-token or per-channel+group",[19,1204,1206],{"id":1205},"what-this-does-not-prove","What this does not prove",[574,1208,1209,1212,1215,1218,1221,1224],{},[577,1210,1211],{},"One platform: Apple M1 only; no discrete-HBM GPU.",[577,1213,1214],{},"Small models: 135M–1.7B; nothing above 7B.",[577,1216,1217],{},"A single-author preprint, not peer-reviewed; we did not reproduce it.",[577,1219,1220],{},"Gemma's 19.5× is not like-for-like; an apples-to-apples comparison inside the full-attention layers is about 3.2×.",[577,1222,1223],{},"Qwen at 2K breaks; the author's own fallback is 1.5K.",[577,1225,1226],{},"This is not the only path. It only addresses the cost of writing the four steps separately.",[19,1228,1230],{"id":1229},"conclusion-the-question-is-not-is-quantization-fast","Conclusion: the question is not \"is quantization fast?\"",[15,1232,1233],{},"Quantization is not a switch. It is a stack: platform, kernel, quantizer, prefix length, architecture. Change one and the answer changes.",[15,1235,1236],{},"The two ledgers:",[574,1238,1239,1245],{},[577,1240,1241,1244],{},[180,1242,1243],{},"Implementation tax."," The same method, eager four steps vs two fused kernels: 11–21 percentage points.",[577,1246,1247,1250],{},[180,1248,1249],{},"Deployment bill."," int4 cache vs fp16: up to −8.2% at a short prefix, single digits at long prefixes (Qwen 1024 is −1.6%, Gemma 4096 is −3.2%), and it can stop early.",[15,1252,1253],{},"One thing to do tomorrow: run your model at 256, 1K and 2K, record ms\u002Ftok and output quality. If it is a full-attention stack, keep an fp16 fallback above 1.5K; 2K is the test point.",[15,1255,1256,1257,1261],{},"If you want help running those three prefixes on your own model, our ",[626,1258,1260],{"href":1259},"\u002Fen\u002Fservices","AI readiness audit"," is exactly that: what can be handed to AI now, and what should wait.",[15,1263,1264,1265,1268],{},"The series so far: ",[626,1266,647],{"href":1267},"\u002Fen\u002Fblog\u002Fbroken-on-arrival-quantized-models"," on the recipe, and this essay on the implementation. The earlier memory-trade-off essay is available in Chinese. The question moves from \"should we quantize?\" to \"how is that kernel written?\"",[650,1270],{},[15,1272,1273],{},[655,1274,1275,1276,1279,1280,1283],{},"Paper: ",[626,1277,663],{"href":660,"rel":1278},[662]," (Mohamed Amine Bergach, Illumina). Previous essay: ",[626,1281,1282],{"href":1267},"the quantized model may be broken",".",{"title":674,"searchDepth":675,"depth":675,"links":1285},[1286,1287,1288,1289,1290,1291,1292,1293],{"id":717,"depth":675,"text":718},{"id":733,"depth":675,"text":734},{"id":777,"depth":675,"text":778},{"id":961,"depth":675,"text":962},{"id":1047,"depth":675,"text":1048},{"id":1113,"depth":675,"text":1114},{"id":1205,"depth":675,"text":1206},{"id":1229,"depth":675,"text":1230},"\u002Fimages\u002Fblog\u002Fen\u002Fquantization-kernel-dispatch.svg","Cover for the English essay, Quantization Speed Is Often a Kernel Problem","Against SRHT, eager int4 KV-cache quantization ran 12–17% slower; fusing the four steps into Metal kernels cut most of the tax. The 8.2% win at 256 tokens has conditions—and Qwen breaks at 2K.","en",{},"\u002Fimages\u002Fblog\u002Fen\u002Fquantization-kernel-dispatch.jpg","\u002Fblog\u002Fen\u002Fquantization-kernel-dispatch",{"title":707,"description":1296},"blog\u002Fen\u002Fquantization-kernel-dispatch",[699,700,701,702],"VrMGy2tQ__ko07oo7IbI-0WNfg5nBCIyvoC2Snzani0",{"id":1306,"company":1307,"extension":1318,"meta":1319,"services":1320,"social":1397,"stem":1398,"__hash__":1399},"site\u002Fsite.yml",{"name":1308,"nameEn":1309,"email":1310,"address":1311,"addressEn":1312,"lineUrl":1313,"lineId":1314,"duns":1315,"youtube":1316,"youtubeId":1317},"銡策數位有限公司","Jecision Digital Co., Ltd.","contact@jecision.com","231 新北市新店區北新路二段 262 號 3 樓之 3","3F.-3, No. 262, Sec. 2, Beixin Rd., Xindian Dist., New Taipei City 231, Taiwan","https:\u002F\u002Fline.me\u002FR\u002Fti\u002Fp\u002F@374qnumi","@374qnumi","963905705","https:\u002F\u002Fwww.youtube.com\u002Fshorts\u002F6UE0WTmyjsg","6UE0WTmyjsg","yml",{},[1321,1344,1370],{"key":1322,"zh":1323,"en":1333},"audit",{"title":629,"tagline":1324,"summary":1325,"points":1326,"deliverable":1330,"detail":1331,"duration":1332},"先看清楚，再花錢","60 分鐘線上訪談，盤點最耗時、最依賴人力的營運環節；我們會告訴你哪些工作現在就能交給 AI、哪些還不成熟。",[1327,1328,1329],"現況盤點：梳理日常營運中最耗時、最依賴人力的環節","痛點診斷：判斷哪些工作現在就能交給 AI，哪些建議先保留人力","可行性評估：依你的業務特性與預算，篩選適合的模型與工具","《企業 AI 導入建議報告書》（PDF）","含 1–2 個「開發成本最低、成效最明顯」的優先項目、技術棧建議、預期節省的人力時間與下一步","訪談 60 分鐘，報告書 3 個工作天內交付",{"title":1334,"tagline":1335,"summary":1336,"points":1337,"deliverable":1341,"detail":1342,"duration":1343},"AI Readiness Audit","Look before you spend","A 60-minute remote audit of the workflows that eat the most time and people-power — and an honest answer on what AI can take over today, and what it cannot.",[1338,1339,1340],"Audit the daily workflows that consume the most time and headcount","Triage which tasks AI can own today, and which should stay human","Shortlist the models and tools that fit your budget and constraints","AI Adoption Recommendation Report (PDF)","One or two highest-ROI entry points, a recommended tech stack, expected hours saved, and the concrete next step","60-minute call, report delivered within 3 business days",{"key":1345,"zh":1346,"en":1357},"prompt",{"title":1347,"tagline":1348,"summary":1349,"points":1350,"deliverable":1354,"detail":1355,"duration":1356},"提示詞工程","讓 AI 穩定產出，而不是碰運氣","針對你的實際任務設計提示詞與多步驟工作流，並做極限測試，確保換了資料、換了人操作也不會走鐘。",[1351,1352,1353],"客製提示詞：ChatGPT、Claude、Gemini、Midjourney 等主流工具的專屬模板","除錯與優化：修掉幻覺、格式跑掉、時好時壞的問題","工作流建置：把複雜任務拆成多步驟提示詞與可重複的 SOP","提示詞文件或設定檔","附使用說明與範例，並提供 1 對 1 教學，確保你與團隊能自己維護","依任務複雜度報價，通常 3–7 個工作天",{"title":1358,"tagline":1359,"summary":1360,"points":1361,"deliverable":1367,"detail":1368,"duration":1369},"Prompt Engineering","Make AI produce reliably, not by luck","Prompt sets and multi-step workflows designed for your actual tasks, stress-tested so they keep working when the data or the operator changes.",[1362,1363,1365],"Custom prompts for ChatGPT, Claude, Gemini, Midjourney and friends",{"Debugging and tuning":1364},"kill hallucinations, broken formats and flaky output",{"Workflow design":1366},"break complex work into repeatable multi-step SOPs","Prompt library or exportable config","Ships with usage notes and examples, plus a 1:1 walkthrough so your team can maintain it","Quoted by complexity, typically 3–7 business days",{"key":1371,"zh":1372,"en":1383},"poc",{"title":1373,"tagline":1374,"summary":1375,"points":1376,"deliverable":1380,"detail":1381,"duration":1382},"極速 PoC","五天，看到能跑的東西","不談架構藍圖，直接做一個能動的雛形。五個工作天內交付線上可操作的智能體，用你真實的資料驗證。",[1377,1378,1379],"Day 1：15 分鐘對焦，鎖定單一痛點與資料盤點","Day 2–4：模型串接、邏輯設計與雛形開發","Day 5：Demo 展示與 AI 擴展建議書","可線上操作的智能體雛形","交付範圍為概念驗證與展示，不含後續維運或大規模伺服器部署；要進階成完整系統會另行報價","5 個工作天",{"title":1384,"tagline":1385,"summary":1386,"points":1387,"deliverable":1394,"detail":1395,"duration":1396},"Rapid PoC","Five days to something that runs","No architecture decks — a working prototype built in five business days, validated against your real data.",[1388,1390,1392],{"Day 1":1389},"15-minute scoping call, one pain point, data inventory",{"Day 2–4":1391},"model integration, logic design, prototype build",{"Day 5":1393},"live demo plus an AI expansion recommendation","A runnable online agent prototype","Proof of concept and demo scope only; ongoing operations and large-scale deployment are quoted separately","5 business days",[],"site","XTwARfVAOcrXNiKehsM6V6cdkd2HXy4d5yZVrSWrI3Q",[1401,1859],{"id":1402,"title":1403,"author":7,"body":1404,"cover":1845,"coverAlt":1846,"date":1847,"description":1848,"draft":689,"extension":690,"featured":689,"lang":1297,"meta":1849,"navigation":81,"ogImage":1850,"path":1851,"seo":1852,"slug":1853,"stem":1854,"tags":1855,"translationKey":1853,"updated":703,"__hash__":1858},"blog\u002Fblog\u002Fen\u002Fbroken-on-arrival-quantized-models.md","Your quantized model might be broken",{"type":9,"value":1405,"toc":1834},[1406,1415,1417,1424,1427,1430,1436,1440,1443,1450,1456,1467,1471,1474,1481,1491,1497,1501,1504,1562,1565,1571,1575,1578,1585,1591,1594,1598,1601,1706,1712,1723,1727,1740,1746,1760,1764,1782,1786,1791,1797,1804,1806],[12,1407,1408],{},[15,1409,1410,1411,1414],{},"Numbers come from arXiv 2609.05881v3 and the author's ",[638,1412,1413],{},"quantcheck"," audit data. Whether the bad files are still being served is our own check; the method is at the end.",[19,1416,718],{"id":717},[15,1418,1419,1420,1423],{},"You are on a laptop. You run ",[638,1421,1422],{},"ollama pull qwen2.5-coder:3b",", choosing a 2-bit tag because memory is tight.",[15,1425,1426],{},"The model starts. It answers, and it is fast. Then you ask for a sorting function and the code does not compile. You try another task; same thing.",[15,1428,1429],{},"Your first instinct is to blame yourself: the prompt? the temperature? maybe this model is just weak?",[15,1431,1432,1433],{},"A paper turned that doubt into a census: ",[180,1434,1435],{},"of 305 code-capable tags in the official Ollama library, seven solved at most six of 164 HumanEval+ tasks. Six solved none.",[19,1437,1439],{"id":1438},"the-counter-intuitive-part-the-files-are-fine","The counter-intuitive part: the files are fine",[15,1441,1442],{},"When a model misbehaves, you suspect a corrupt download.",[15,1444,1445,1446,1449],{},"The author rebuilt five of the seven from the original weights with llama.cpp. Comparing tensor by tensor, ",[180,1447,1448],{},"all five matched exactly"," — 434 tensors for the Qwen2.5-Coder-3B tags.",[15,1451,1452,1453,1283],{},"So these are not corrupt files. They are ",[180,1454,1455],{},"correctly reproduced broken artifacts",[15,1457,1458,1459,1462,1463,1466],{},"Checksums pass. File sizes match. ",[638,1460,1461],{},"ollama show"," looks normal. Because the file really is exactly what it claims to be. What is broken is the ",[180,1464,1465],{},"quantization recipe",": same weights, same toolchain, same tensor mix, but without calibration the model collapsed into a shape that cannot write code.",[19,1468,1470],{"id":1469},"why-this-happens","Why this happens",[15,1472,1473],{},"Quantization squeezes weights from 16 bits down to 2–4 bits. Every step has to decide where the rounding error lands.",[15,1475,1476,1477,1480],{},"The default approach spreads it evenly. The better approach uses ",[180,1478,1479],{},"calibration data"," — llama.cpp's importance matrix (imatrix) — to push error away from the weights that matter, based on activation statistics.",[15,1482,1483,1484,1487,1488,1283],{},"Without imatrix, small models at low bit are the ones that fall apart. All seven suspects sit in ",[180,1485,1486],{},"q2_K through q3_K_L, in models of at most 4B parameters",". That subgroup contains 48 tags, so the failure rate inside it is ",[180,1489,1490],{},"14.6%",[15,1492,1493,1494,1283],{},"But the recipe alone does not predict failure: the same uncalibrated recipe left Qwen2.5-Coder-1.5B functional (13–15 of 15). The problem is not \"low bit always breaks\". It is that ",[180,1495,1496],{},"when it breaks, you cannot tell",[19,1498,1500],{"id":1499},"calibration-rescues-most-of-them","Calibration rescues most of them",[15,1502,1503],{},"Holding weights, toolchain and tensor mix fixed, adding only an imatrix:",[98,1505,1506,1519],{},[101,1507,1508],{},[104,1509,1510,1513,1516],{},[107,1511,1512],{},"Tag",[107,1514,1515],{},"Before",[107,1517,1518],{},"After imatrix (of 164)",[120,1520,1521,1532,1542,1552],{},[104,1522,1523,1526,1529],{},[125,1524,1525],{},"Qwen2.5-Coder-3B q2_K",[125,1527,1528],{},"0",[125,1530,1531],{},"106–113",[104,1533,1534,1537,1539],{},[125,1535,1536],{},"Qwen2.5-Coder-3B q3_K_M",[125,1538,1528],{},[125,1540,1541],{},"76–89",[104,1543,1544,1547,1549],{},[125,1545,1546],{},"Qwen2.5-Coder-3B q3_K_L",[125,1548,1528],{},[125,1550,1551],{},"80–86",[104,1553,1554,1557,1559],{},[125,1555,1556],{},"Qwen2.5-Coder-3B q3_K_S",[125,1558,1528],{},[125,1560,1561],{},"13–30 (still weak)",[15,1563,1564],{},"(The paper's abstract says \"0 → 76–113\"; the per-level numbers above come from the v3.4 audit data, which we treat as authoritative.)",[15,1566,1567,1568],{},"Same model, same bit width, one calibration table: ",[180,1569,1570],{},"from zero tasks to three digits.",[19,1572,1574],{"id":1573},"another-failure-mode-a-missing-chat-template","Another failure mode: a missing chat template",[15,1576,1577],{},"Beyond the seven, the author also tested 22 popular Hugging Face community files. Two of them (TheBloke's deepseek-coder-6.7B Q3_K_M and Q4_K_M) scored 5 and 6 of 15 in the census.",[15,1579,1580,1581,1584],{},"The cause was not quantization but a ",[180,1582,1583],{},"missing chat template",". With no template, Ollama falls back to a default ChatML format that does not match the model's instruction format: 28 of 30 answers ran on into invented chat turns.",[15,1586,1587,1588,1283],{},"With the right template: ",[180,1589,1590],{},"98 and 101 of 164",[15,1592,1593],{},"So \"it runs\" and \"it answers properly\" are separated by a template.",[19,1595,1597],{"id":1596},"we-checked-the-bad-tags-are-still-being-served","We checked: the bad tags are still being served",[15,1599,1600],{},"A September incident is only interesting if it is still downloadable. We compared the blob digests in the audit data against the current Ollama registry manifests:",[98,1602,1603,1618],{},[101,1604,1605],{},[104,1606,1607,1609,1612,1615],{},[107,1608,1512],{},[107,1610,1611],{},"Audited digest",[107,1613,1614],{},"Today",[107,1616,1617],{},"Status",[120,1619,1620,1634,1646,1658,1670,1682,1694],{},[104,1621,1622,1625,1628,1631],{},[125,1623,1624],{},"qwen2.5-coder:3b-instruct-q2_K",[125,1626,1627],{},"14c4247b8ce4…",[125,1629,1630],{},"same",[125,1632,1633],{},"still served",[104,1635,1636,1639,1642,1644],{},[125,1637,1638],{},"qwen2.5-coder:3b-instruct-q3_K_S",[125,1640,1641],{},"81d167af7fa3…",[125,1643,1630],{},[125,1645,1633],{},[104,1647,1648,1651,1654,1656],{},[125,1649,1650],{},"qwen2.5-coder:3b-instruct-q3_K_M",[125,1652,1653],{},"9bb5d28b842a…",[125,1655,1630],{},[125,1657,1633],{},[104,1659,1660,1663,1666,1668],{},[125,1661,1662],{},"qwen2.5-coder:3b-instruct-q3_K_L",[125,1664,1665],{},"7a88dc11faae…",[125,1667,1630],{},[125,1669,1633],{},[104,1671,1672,1675,1678,1680],{},[125,1673,1674],{},"phi3.5:3.8b-mini-instruct-q2_K",[125,1676,1677],{},"54d47caa8bf3…",[125,1679,1630],{},[125,1681,1633],{},[104,1683,1684,1687,1690,1692],{},[125,1685,1686],{},"llama3.2:1b-instruct-q2_K",[125,1688,1689],{},"d1ede02a2111…",[125,1691,1630],{},[125,1693,1633],{},[104,1695,1696,1699,1702,1704],{},[125,1697,1698],{},"llama3.2:1b-instruct-q3_K_S",[125,1700,1701],{},"81881dcc12ec…",[125,1703,1630],{},[125,1705,1633],{},[15,1707,1708,1711],{},[180,1709,1710],{},"Seven out of seven are still there, with identical digests."," This is not history; it is something you can pull today.",[15,1713,1714,1715,1718,1719,1722],{},"(Method: fetch ",[638,1716,1717],{},"https:\u002F\u002Fregistry.ollama.ai\u002Fv2\u002Flibrary\u002F\u003Cmodel>\u002Fmanifests\u002F\u003Ctag>"," and compare the ",[638,1720,1721],{},"application\u002Fvnd.ollama.image.model"," layer digest. Script at the end.)",[19,1724,1726],{"id":1725},"three-things-you-can-do-now","Three things you can do now",[15,1728,1729,1732,1733,1735,1736,1739],{},[180,1730,1731],{},"Screen."," The author released ",[638,1734,1413],{},", a 15-task screen per tag. It costs about ",[180,1737,1738],{},"30 seconds of generation"," (on a 4090, excluding download). None of 66 sampled tags that passed the screen turned out to be non-functional, so it works as a pre-filter.",[15,1741,1742,1745],{},[180,1743,1744],{},"If you publish quantizations:"," add an imatrix, and run a functional test before publishing. Registries will not do it for you.",[15,1747,1748,1751,1752,1755,1756,1759],{},[180,1749,1750],{},"Check the chat template."," ",[638,1753,1754],{},"ollama show \u003Cmodel> --template","; if it prints ",[638,1757,1758],{},"{{ .Prompt }}",", there is no template and the model may be answering in the wrong format.",[19,1761,1763],{"id":1762},"do-not-over-read-this","Do not over-read this",[574,1765,1766,1769,1772,1779],{},[577,1767,1768],{},"2.3% is the share of official, code-capable, ≤8 GB tags — not of all quantized models",[577,1770,1771],{},"Only coding ability was tested (HumanEval+); perplexity is a pre-check, not proof of function",[577,1773,1774,1775,1778],{},"Seven is a ",[180,1776,1777],{},"lower bound",": all 12 deepseek-coder-1.3B tags, including fp16, score 1–6 of 15, so a group baseline cannot judge them and the author does not count them",[577,1780,1781],{},"The author is an independent researcher; data and tools are public, but this is one registry and one runtime version",[19,1783,1785],{"id":1784},"conclusion","Conclusion",[15,1787,1788],{},[180,1789,1790],{},"Exact reproducibility is not function.",[15,1792,1793,1794,1796],{},"Identical tensors, a passing checksum, a normal ",[638,1795,1461],{}," — all of that proves the file is the file. None of it proves the model can do the job.",[15,1798,1799,1800,1803],{},"So the question is not whether quantized models are usable. It is ",[180,1801,1802],{},"whether anyone tested whether this one is",". Right now the registries have not, which leaves it to you.",[650,1805],{},[15,1807,1808],{},[655,1809,1810,1811,1816,1817,1816,1821,1826,1827,1830,1831,1283],{},"Sources: ",[626,1812,1815],{"href":1813,"rel":1814},"https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.05881",[662],"arXiv 2609.05881v3",", ",[626,1818,1413],{"href":1819,"rel":1820},"https:\u002F\u002Fgithub.com\u002Faditi-p31\u002Fquantcheck",[662],[626,1822,1825],{"href":1823,"rel":1824},"https:\u002F\u002Fgithub.com\u002Follama\u002Follama\u002Fissues\u002F18252",[662],"Ollama issue #18252",". Our digest check: ",[638,1828,1829],{},"scripts\u002Fcheck-ollama-tags.ts","; run ",[638,1832,1833],{},"npm run check:ollama -- --list",{"title":674,"searchDepth":675,"depth":675,"links":1835},[1836,1837,1838,1839,1840,1841,1842,1843,1844],{"id":717,"depth":675,"text":718},{"id":1438,"depth":675,"text":1439},{"id":1469,"depth":675,"text":1470},{"id":1499,"depth":675,"text":1500},{"id":1573,"depth":675,"text":1574},{"id":1596,"depth":675,"text":1597},{"id":1725,"depth":675,"text":1726},{"id":1762,"depth":675,"text":1763},{"id":1784,"depth":675,"text":1785},"\u002Fimages\u002Fblog\u002Fen\u002Fbroken-on-arrival-quantized-models.svg","A quantized model file that passes its checksum but fails every coding task","2026-11-02","Someone tested every code-capable tag in Ollama's library. Seven low-bit quantizations failed nearly all tasks, yet the files were exact rebuilds. Checksums pass; the recipe is broken.",{},"\u002Fimages\u002Fblog\u002Fen\u002Fbroken-on-arrival-quantized-models.jpg","\u002Fblog\u002Fen\u002Fbroken-on-arrival-quantized-models",{"title":1403,"description":1848},"broken-on-arrival-quantized-models","blog\u002Fen\u002Fbroken-on-arrival-quantized-models",[699,1856,1857],"local-llm","gguf","tOVVVr1HmeG95b9UhxSxh_uy4dZQvyUwEripBKmKjww",{"id":1860,"title":1861,"author":7,"body":1862,"cover":2497,"coverAlt":2498,"date":2499,"description":2500,"draft":689,"extension":690,"featured":689,"lang":1297,"meta":2501,"navigation":81,"ogImage":2502,"path":2503,"seo":2504,"slug":2505,"stem":2506,"tags":2507,"translationKey":2505,"updated":703,"__hash__":2511},"blog\u002Fblog\u002Fen\u002Frunning-is-not-done.md","A running model is not a finished job",{"type":9,"value":1863,"toc":2485},[1864,1869,1871,1874,1877,1880,1883,1891,1895,1922,1925,1930,1933,1994,1997,2000,2003,2006,2017,2021,2024,2027,2030,2033,2048,2051,2054,2057,2060,2063,2067,2070,2073,2076,2079,2082,2085,2088,2091,2094,2097,2114,2117,2120,2131,2135,2138,2141,2144,2147,2150,2153,2156,2159,2209,2212,2215,2218,2221,2224,2227,2230,2234,2237,2240,2243,2246,2249,2252,2255,2264,2267,2270,2274,2277,2288,2291,2294,2297,2300,2362,2365,2368,2371,2375,2389,2392,2395,2399,2402,2434,2437,2448,2456,2463,2469,2473,2476,2479,2482],[12,1865,1866],{},[15,1867,1868],{},"Every number here comes from arXiv 2610.34241v1, a single-author engineering report. It is not a controlled experiment. Where I add a judgement, I say so.",[19,1870,718],{"id":717},[15,1872,1873],{},"\"The model answered.\" That is the easiest sentence to fail an acceptance test with. The API returns 200, the prose reads well, and the tool call may never have left the building.",[15,1875,1876],{},"In July 2026, Vaibhav Jain pooled a few laptops to serve a model too large for any one machine. Chat worked; a coding agent did not. Every restart pushed a 32B checkpoint across Wi-Fi — 8.7 GB, roughly five minutes. He went single-node and noticed the real problem: \"the model is running\" is a very weak success criterion.",[15,1878,1879],{},"A checkpoint loads and answers. Then you ask it to inspect a file, call a tool, hold a long session, or preserve a repository contract. It may fail all of that, and your API still returns 200.",[15,1881,1882],{},"\"Compatible\" hides a lot. Behind one nominally compatible endpoint, tool-call formatting, turn control, context handling and cache reuse can all differ. Change one, and a workflow you verified on model A breaks on model B — usually without an error message.",[15,1884,1885,1890],{},[626,1886,1889],{"href":1887,"rel":1888},"https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.34241",[662],"arXiv 2610.34241"," covers July to October 2026 across vLLM, SGLang, llama.cpp and MLX. Its move is simple: split \"it runs\" into four levels, then run 45 trials with hidden checks to see what level four costs.",[19,1892,1894],{"id":1893},"four-levels-not-one-switch","Four levels, not one switch",[1896,1897,1898,1904,1910,1916],"ol",{},[577,1899,1900,1903],{},[180,1901,1902],{},"Served"," — the checkpoint loads and returns something.",[577,1905,1906,1909],{},[180,1907,1908],{},"Coherent"," — the text reads like language and answers the question.",[577,1911,1912,1915],{},[180,1913,1914],{},"Parsed and executed tool calls"," — it says \"read this file\", the client receives it, and the read actually happens.",[577,1917,1918,1921],{},[180,1919,1920],{},"Verified task completion"," — the code passes an independent behavioural check.",[15,1923,1924],{},"The paper's framing sentence is the spine of everything below:",[12,1926,1927],{},[15,1928,1929],{},"An endpoint can satisfy the first three and fail the fourth completely. And if it cannot satisfy the third, you cannot measure the fourth at all.",[15,1931,1932],{},"(My translation. The emphasis is the author's.)",[98,1934,1935,1948],{},[101,1936,1937],{},[104,1938,1939,1942,1945],{},[107,1940,1941],{},"Level",[107,1943,1944],{},"The question you ask",[107,1946,1947],{},"What passing it still doesn't tell you",[120,1949,1950,1961,1972,1983],{},[104,1951,1952,1955,1958],{},[125,1953,1954],{},"1 Served",[125,1956,1957],{},"Does it load and answer?",[125,1959,1960],{},"Whether the answer is language",[104,1962,1963,1966,1969],{},[125,1964,1965],{},"2 Coherent",[125,1967,1968],{},"Does it read and reason?",[125,1970,1971],{},"Whether it will act",[104,1973,1974,1977,1980],{},[125,1975,1976],{},"3 Tools",[125,1978,1979],{},"Are tool calls parsed and executed?",[125,1981,1982],{},"Whether the result is right",[104,1984,1985,1988,1991],{},[125,1986,1987],{},"4 Verified",[125,1989,1990],{},"Does the output pass an independent check?",[125,1992,1993],{},"This is \"done\"",[15,1995,1996],{},"These are not degrees of the same thing. They are four different gates. Level one passing says nothing about level two; level two passing says nothing about whether the model will pick up a tool. Most people stop at level two and think they are close, because the model sounds convincing.",[15,1998,1999],{},"The three tiers on an acceptance sheet map roughly like this: \"it responds\" is levels 1 and 2; \"the code runs\" is level 3; \"the tests pass\" is level 4. I keep 1 and 2 separate because their failures look similar and have different causes — one service never came up, the other came up and only talks.",[15,2001,2002],{},"And \"the tests pass\" depends on whose tests. If the vendor wrote them, you may still be at level three. Level four needs a check written by someone else and frozen before acceptance.",[15,2004,2005],{},"(Figure 1)",[71,2007,75,2009,75,2014],{"className":2008},[74],[77,2010],{"src":2011,"poster":2012,"autoPlay":81,"loop":81,"muted":81,"playsInline":81,"ariaLabel":2013},"\u002Fimages\u002Fblog\u002Fen\u002Frunning-is-not-done\u002Flevels.mp4","\u002Fimages\u002Fblog\u002Fen\u002Frunning-is-not-done\u002Flevels-poster.jpg","Four levels of progress: served, coherent, tools, verified, with a break before verified",[84,2015,2016],{},"Figure 1: four different gates, not four degrees. The first three can pass and the fourth still fail.",[19,2018,2020],{"id":2019},"same-api-different-protocol","Same API, different protocol",[15,2022,2023],{},"He pointed Claude Code at a gateway he wrote, with different models behind it. The expectation was \"swap an endpoint\". The experience was closer to handing the work to a person with different habits.",[15,2025,2026],{},"Some models follow the tool contract. Some narrate what they intend to do instead of calling the tool. Some rewrite an interface they find awkward, including a function signature that was supposed to stay stable. None of this is stupidity; it is a different understanding of what a tool is.",[15,2028,2029],{},"The logs make the gap concrete. Of 492 Gemma 4 requests that carried tools, 294 produced parsed tool_calls. Another 144 ended with stop and no tool call; 54 had no saved parsed response.",[15,2031,2032],{},"Those 144 are not all failures. A normal finish is also a stop, and no tool call may simply mean the task was done. That is the first trap here: a missing tool call is not an error signal by itself. You have to know whether one was expected.",[15,2034,2035,2036,2039,2040,2043,2044,2047],{},"Some failures are blunter. Eight saved Gemma 4 outputs arrived with a malformed ",[638,2037,2038],{},"\u003C|tool_call>"," prefix and never parsed. On GPT-OSS-120B over SGLang, ",[638,2041,2042],{},"tool_choice=required"," fought the model's own Harmony tool format and had to be downgraded to ",[638,2045,2046],{},"auto","; tool rounds also needed a higher output allowance, or reasoning and tool JSON were truncated together.",[15,2049,2050],{},"Turn control is the other half: a missing session identifier, a tool loop, a thinking channel that does not line up. Each is enough to make a conversation look alive while it is lost.",[15,2052,2053],{},"The episode worth memorising: every API schema check passed, and the real Claude Code client still broke. In one case the working directory never reached the model-facing request, so the model guessed paths. In another, streamed tool-argument fragments were treated as replacements instead of appends, cutting the arguments mid-call.",[15,2055,2056],{},"Neither shows up in a unit test. Both require the actual client, end to end.",[15,2058,2059],{},"His diagnostic question is the useful part: when a tool fails, did the model fail, or did something in the middle drop the call? If it is the latter, fix the source — template, parser rules, serve flags, request translation — rather than patching the gateway forever. He reduced the gateway to a thin translator doing protocol and security checks, and made upstream failures surface as errors instead of a stream that ends normally with nothing in it.",[15,2061,2062],{},"\"Configured\" is not \"served and evaluated\". When a vendor says \"supported\", it often means \"it can be configured\".",[19,2064,2066],{"id":2065},"the-conversation-looks-continuous-it-isnt","The conversation looks continuous. It isn't.",[15,2068,2069],{},"To you it is one continuous thread. To the runtime it is three things: a token prefix, a KV cache, and an execution slot. Your \"continuous\" is not necessarily its continuous.",[15,2071,2072],{},"He kept 4,165 gateway turns and 623 debug jobs. Eighty-five cold-prefix jobs — no reuse at all, 30,000 to 36,000 input tokens — had a median time to first token of 69.4 seconds. Two hundred and eighty high-reuse jobs — at least 90% reused, under 2,000 fresh tokens — had a median of 2.8 seconds.",[15,2074,2075],{},"About 25x.",[15,2077,2078],{},"The author labels this observational: the two groups are composed differently, not randomised, not matched. You cannot write \"caching makes it 25x faster\". What you can say is that prefix reuse is associated with a difference worth engineering.",[15,2080,2081],{},"The mechanism is mundane. An inference backend can reuse KV state for an identical token prefix. Change the prefix — a timestamp, a system banner, tool definitions in a different order, a different JSON serialisation — and the whole context is prefilled again.",[15,2083,2084],{},"You changed a banner. You paid for the whole context.",[15,2086,2087],{},"Auxiliary requests are the pettier version. A coding agent also generates titles and permission prompts; with one reusable execution slot, an auxiliary request can evict the main conversation's resident state. In his records, 65 requests were classified as auxiliary and 44 showed no cache reuse. Among high-reuse records, auxiliary requests had a median TTFT of about 12.2 seconds (13 observations) against 2.7 seconds for main requests (267 observations).",[15,2089,2090],{},"Compaction has the same smell. He lowered the threshold deliberately: in 12 tests, sub-agent compaction fired six times, none of those six tasks completed, and one was still labelled PASS.",[15,2092,2093],{},"\"Compaction happened\" is not \"work finished\". In a log, the two look alike.",[15,2095,2096],{},"So he treats session continuity as an application responsibility, not a backend setting:",[574,2098,2099,2102,2105,2108,2111],{},[577,2100,2101],{},"Keep the message sequence append-only unless earlier content genuinely must change.",[577,2103,2104],{},"Do not re-serialise tool messages for cosmetics.",[577,2106,2107],{},"Make streamed and final tool JSON agree.",[577,2109,2110],{},"Keep per-turn metadata out of the frozen system prefix.",[577,2112,2113],{},"Preserve session affinity. A stable prompt is not enough if the next turn lands on a slot without the cache.",[15,2115,2116],{},"Caching is not a backend flag. It is application discipline.",[15,2118,2119],{},"(Figure 2)",[71,2121,75,2123,75,2128],{"className":2122},[74],[77,2124],{"src":2125,"poster":2126,"autoPlay":81,"loop":81,"muted":81,"playsInline":81,"ariaLabel":2127},"\u002Fimages\u002Fblog\u002Fen\u002Frunning-is-not-done\u002Fcache.mp4","\u002Fimages\u002Fblog\u002Fen\u002Frunning-is-not-done\u002Fcache-poster.jpg","Two paths for one conversation: a stable prefix reuses the cache, a changed prefix refills it",[84,2129,2130],{},"Figure 2: one conversation. The prefix holds and the cache is reused; the prefix changes and the whole context is prefilled again.",[19,2132,2134],{"id":2133},"level-four-is-where-the-work-is-judged","Level four is where the work is judged",[15,2136,2137],{},"Deployment, protocol and sessions are now good enough to run bounded tasks. Can an open-weight model produce code that passes?",[15,2139,2140],{},"Five TypeScript task categories (quote, prorate, customer, split, parts), three conditions, three runs each — 45 sessions. The parent is fixed: Claude Code with Claude Opus 5.5. The worker is a quantized Qwen3-Coder 30B-A3B. Every session is judged by task-specific hidden behavioural checks, not by the agent's own claim of success.",[15,2142,2143],{},"The conditions: direct; optimised delegation, where the parent specifies, reviews and applies but does not implement; and batch-labelled delegation that may split work into sequential packages — though only a minority actually split.",[15,2145,2146],{},"Results: direct 15\u002F15; optimised delegation 12\u002F15; batch-labelled 14\u002F15.",[15,2148,2149],{},"These do not rank. Each cell has three runs, the Wilson intervals overlap, and the author says the data cannot establish general success-rate differences. It does show workers produced usable code: of 30 delegated trials, 26 verified and all 26 retained worker-produced implementations. Not every time.",[15,2151,2152],{},"And the 14\u002F15 is not evidence that batching helps. Only 5 of the 15 batch-labelled trials used a batch call; the rest used single delegation. The paper calls it an adaptive policy, not a batch-versus-single comparison.",[15,2154,2155],{},"The second suite is stricter: 15 tasks, three runs each, 45 sessions per arm, artifacts clearing a fixed interface contract before parent review.",[15,2157,2158],{},"Direct 45\u002F45. Nemotron Cascade 2 worker: 12\u002F45, of which 11 were genuine worker substitutions and one verified only after the parent rewrote the source. Qwen3-Coder worker: 0\u002F45.",[98,2160,2161,2174],{},[101,2162,2163],{},[104,2164,2165,2168,2171],{},[107,2166,2167],{},"Study",[107,2169,2170],{},"Direct",[107,2172,2173],{},"Delegated",[120,2175,2176,2187,2198],{},[104,2177,2178,2181,2184],{},[125,2179,2180],{},"45-trial suite",[125,2182,2183],{},"15\u002F15",[125,2185,2186],{},"Qwen3-Coder 12\u002F15 (optimised), 14\u002F15 (batch-labelled)",[104,2188,2189,2192,2195],{},[125,2190,2191],{},"Second suite (15 tasks x 3)",[125,2193,2194],{},"45\u002F45",[125,2196,2197],{},"Nemotron 12\u002F45 (11 worker substitutions); Qwen 0\u002F45",[104,2199,2200,2203,2206],{},[125,2201,2202],{},"Same, counting interruptions as failures",[125,2204,2205],{},"37\u002F45",[125,2207,2208],{},"Nemotron 8\u002F45",[15,2210,2211],{},"That second suite also has a disclosure: 16 sessions were stopped by a subscription limit before any model call and were replaced. Eight were in the direct arm and eight in the Nemotron arm. Counting the original 16 as failures gives 37\u002F45 and 8\u002F45. Those are scheduling interruptions, not inference failures — but if you cite the headline numbers, cite both.",[15,2213,2214],{},"The 45\u002F45 deserves a caveat too. The tasks were built for this study, five categories with three runs each; there is no difficulty stratification.",[15,2216,2217],{},"One more honest detail: the task definitions and hidden tests were produced with AI assistance. The combined test file was created about 39 seconds before the first evaluated session and first committed after the last of the 45. What he can claim is narrower: task instructions and visible checks were identical across the 45 saved runs, and recorded score labels agree with the scorer as later saved. He cannot claim an independently snapshotted benchmark.",[15,2219,2220],{},"The first study's harness had limits: the parent's separate constraints field was not forwarded to the worker; the gateway dropped a duplicate system message and truncated messages over 1,800 characters; two of 41 reconstructed worker prompts lost about 220 characters, both in failed trials.",[15,2222,2223],{},"So \"the worker ignored instructions\" needs a discount. Some failures were capability; some were a worker that never received the full instruction.",[15,2225,2226],{},"What about 0\u002F45? It does not mean the model can do nothing. That arm had no direct control, parent acceptance and application were not replayed, and the parent's prompt-cache state was not reset between studies. The author scopes it to these tasks, this contract, this worker configuration.",[15,2228,2229],{},"Verification is level four. It is not a by-product of level three.",[19,2231,2233],{"id":2232},"the-bill-runs-the-other-way","The bill runs the other way",[15,2235,2236],{},"Frontier output tokens are expensive; a local worker is cheap. Delegate and save money — except delegation is not free. The parent writes the spec, reads what came back, decides whether to accept it, and cleans up failures. All of that is frontier tokens.",[15,2238,2239],{},"The cost model has five terms: parent context, specification, review, recovery, worker. The study measured the frontier terms. Electricity, hardware and engineering time were not measured — and unmeasured is not zero.",[15,2241,2242],{},"First suite, frontier cost per verified success: $0.136 direct; $0.214 optimised delegation; $0.199 batch-labelled. About 1.6x.",[15,2244,2245],{},"Second suite: $0.153 direct; $1.313 for Nemotron delegation. 8.6x.",[15,2247,2248],{},"Qwen's per-success cost is undefined, not zero — it had no successes.",[15,2250,2251],{},"One small case is worth keeping: a fee task where the worker's implementation passed and was applied, yet the frontier cost was $0.1856 versus $0.1293 direct, and wall time 126.4 seconds versus 27.4.",[15,2253,2254],{},"A successful substitution is not a saving.",[15,2256,2257,2258,2263],{},"This does not generalise to \"delegation never pays\". The author cites ",[626,2259,2262],{"href":2260,"rel":2261},"https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.04804",[662],"Scrouting",": a small model scouts the repository, frontier models fix, and it solved 159 of 266 tasks against 158 for the strongest single model — one task apart — at roughly one fifth the cost. The direction is reversed: scout small, fix big. He limits his finding to his own contract, frontier-specifies \u002F local-implements.",[15,2265,2266],{},"The cost definition is narrow too: API list prices, no hardware, electricity or engineering time. \"8.6x\" is the frontier side of the bill, not your total.",[15,2268,2269],{},"Tokens are cheap. Coordination is not.",[19,2271,2273],{"id":2272},"the-acceptance-script-will-reject-good-code","The acceptance script will reject good code",[15,2275,2276],{},"The most practical part of the report is not about models. It is about acceptance.",[15,2278,2279,2280,2283,2284,2287],{},"Worker artifacts had to clear a static contract: imports, exports, and exported function parameter lists had to match. The stubs used underscored parameters for unused arguments (",[638,2281,2282],{},"_entries","); workers naturally dropped the underscore (",[638,2285,2286],{},"entries",").",[15,2289,2290],{},"For ordinary positional callers this changes nothing. But the contract compared parameter-name strings, so the artifact was rejected before parent review, before the hidden checks ran.",[15,2292,2293],{},"The post-hoc diagnostic: 36 valid-JSON artifacts rejected solely for dropping the underscore were applied unchanged to fresh task copies, with that rule bypassed, and re-run against the original sealed checks.",[15,2295,2296],{},"19 passed. 17 still failed.",[15,2298,2299],{},"So the rule was not simply wrong. It screened out both good and bad output. Rejecting bad output is its job; rejecting good output is its side effect — and if that gate is your only gate, the side effect is the final result.",[98,2301,2302,2318],{},[101,2303,2304],{},[104,2305,2306,2309,2312,2315],{},[107,2307,2308],{},"Worker",[107,2310,2311],{},"Rejected for underscores",[107,2313,2314],{},"Passed sealed checks unchanged",[107,2316,2317],{},"Still failed",[120,2319,2320,2334,2348],{},[104,2321,2322,2325,2328,2331],{},[125,2323,2324],{},"Qwen3-Coder",[125,2326,2327],{},"23",[125,2329,2330],{},"12",[125,2332,2333],{},"11",[104,2335,2336,2339,2342,2345],{},[125,2337,2338],{},"Nemotron",[125,2340,2341],{},"13",[125,2343,2344],{},"7",[125,2346,2347],{},"6",[104,2349,2350,2353,2356,2359],{},[125,2351,2352],{},"Total",[125,2354,2355],{},"36",[125,2357,2358],{},"19",[125,2360,2361],{},"17",[15,2363,2364],{},"He did not count the 19 as successes. They never went through parent review and application; the frozen scores stay Qwen 0\u002F45 and Nemotron 12\u002F45. He also did not conclude the contract should go. He proposed the next experiment: give the worker the specific validation error and allow one bounded repair.",[15,2366,2367],{},"That experiment has not been run.",[15,2369,2370],{},"On this contract, acceptance design failed more easily than model capability did.",[19,2372,2374],{"id":2373},"what-this-cannot-prove","What this cannot prove",[574,2376,2377,2380,2383,2386],{},[577,2378,2379],{},"A single-author retrospective, not a controlled experiment. The workload changed as the engineering changed.",[577,2381,2382],{},"Small sample: five task categories, three runs per cell, one fixed parent, one worker configuration.",[577,2384,2385],{},"Cost covers frontier tokens at list prices only.",[577,2387,2388],{},"The raw records are private. You cannot recompute the numbers.",[15,2390,2391],{},"The studies are not pooled and do not rank models or runtimes.",[15,2393,2394],{},"So this gives you an acceptance framework, not a leaderboard. The most valuable part, in my reading, is not the numbers — it is that \"done\" becomes something you can check level by level.",[19,2396,2398],{"id":2397},"a-checklist-you-can-use-tomorrow","A checklist you can use tomorrow",[15,2400,2401],{},"The author's five rules, rewritten for a PoC document:",[1896,2403,2404,2410,2416,2422,2428],{},[577,2405,2406,2409],{},[180,2407,2408],{},"Define success at the task boundary."," Accept on \"the output passed a check it does not own\", not on \"it responded\". Exercise the real client, not a harness you wrote.",[577,2411,2412,2415],{},[180,2413,2414],{},"Treat deployment as a recipe, not a model name."," Checkpoint revision, quantisation, runtime, template, context settings, gateway, client, resident memory. Change one, re-verify.",[577,2417,2418,2421],{},[180,2419,2420],{},"Make session continuity an application responsibility."," Stable prefix, no cosmetic re-serialisation, auxiliary requests must not evict the main conversation.",[577,2423,2424,2427],{},[180,2425,2426],{},"Price delegation per verified success."," Not per token, not per call.",[577,2429,2430,2433],{},[180,2431,2432],{},"Verify by behaviour where you can."," Static gates are cheap and worth keeping, but know what they reject.",[15,2435,2436],{},"If you are the buyer, three questions are enough:",[574,2438,2439,2442,2445],{},[577,2440,2441],{},"Which level does this demo stop at? \"It answers\" is level two.",[577,2443,2444],{},"How do you verify level four, and who wrote the checks, and when were they frozen?",[577,2446,2447],{},"Is the cost per call or per verified success?",[15,2449,2450,2451,2455],{},"Our own ",[626,2452,2454],{"href":2453},"\u002Fwork\u002Fvideo-transcript-pipeline","video transcription pipeline"," ran into the same class of problem: the model ran, the pipeline still stalled, and every failure looked like \"the model is a bit dumb\". We have not run measurements at this scale, so this essay borrows the paper's numbers.",[15,2457,2458,2459,2462],{},"The previous essay was about ",[626,2460,2461],{"href":644},"files that look right and cannot be used","; this one is about models that look right and cannot finish the job.",[15,2464,2465,2466,2468],{},"If you want help filling in the table, our ",[626,2467,1260],{"href":628}," does exactly that: what can be handed to AI now, and what should wait.",[19,2470,2472],{"id":2471},"conclusion-which-level-counts-as-done","Conclusion: which level counts as done",[15,2474,2475],{},"The first three levels can be demonstrated in an afternoon. Load a model, send a few prompts, connect a client.",[15,2477,2478],{},"Level four takes weeks: design tasks, write hidden checks, run enough trials, classify failures, price the work. What you get is rarely \"this model is good\" — it is \"on these tasks, under this contract, it is good\".",[15,2480,2481],{},"The missing column on your acceptance sheet is where the price difference lives.",[15,2483,2484],{},"Tomorrow: write \"passes the tests\" into your PoC definition, and ask who wrote the tests and when they froze them.",{"title":674,"searchDepth":675,"depth":675,"links":2486},[2487,2488,2489,2490,2491,2492,2493,2494,2495,2496],{"id":717,"depth":675,"text":718},{"id":1893,"depth":675,"text":1894},{"id":2019,"depth":675,"text":2020},{"id":2065,"depth":675,"text":2066},{"id":2133,"depth":675,"text":2134},{"id":2232,"depth":675,"text":2233},{"id":2272,"depth":675,"text":2273},{"id":2373,"depth":675,"text":2374},{"id":2397,"depth":675,"text":2398},{"id":2471,"depth":675,"text":2472},"\u002Fimages\u002Fblog\u002Fen\u002Frunning-is-not-done.svg","Cover for an essay on the four levels between a served model and finished work","2026-11-03","Same endpoint, different behaviour: tool calls, turn control, context, cache. A 4,165-turn solo report splits 'it runs' into four levels, finding local delegation costs 8.6x more per verified success.",{},"\u002Fimages\u002Fblog\u002Fen\u002Frunning-is-not-done.jpg","\u002Fblog\u002Fen\u002Frunning-is-not-done",{"title":1861,"description":2500},"running-is-not-done","blog\u002Fen\u002Frunning-is-not-done",[1856,2508,2509,2510],"agents","serving","tool-calling","0DdevvjhzLCzG5RvWM1GlSfrg6_pwOm2oIXJq0yz_YU",1793954705106]