[{"data":1,"prerenderedAt":2912},["ShallowReactive",2],{"post-macaq-one-layer-at-a-time":3,"site-info":1710,"related-en-macaq-one-layer-at-a-time":1806},{"zh":4,"en":916},{"id":5,"title":6,"author":7,"body":8,"cover":897,"coverAlt":898,"date":899,"description":900,"draft":901,"extension":902,"featured":901,"lang":903,"meta":904,"navigation":307,"ogImage":905,"path":906,"seo":907,"slug":908,"stem":909,"tags":910,"translationKey":908,"updated":914,"__hash__":915},"blog\u002Fblog\u002Fzh-TW\u002Fmacaq-one-layer-at-a-time.md","量化 122B，一次只裝一層","Jecision",{"type":9,"value":10,"toc":887},"minimark",[11,34,38,41,44,51,54,57,60,63,67,70,73,76,79,82,85,88,91,94,97,187,190,193,196,199,202,205,208,211,214,217,223,293,296,313,321,324,327,330,334,337,340,343,346,349,356,403,406,409,413,416,419,422,425,428,431,434,537,540,543,546,549,561,564,568,571,574,577,584,622,625,628,706,709,728,735,738,762,769,777,780,787,790,793,841,844,847,872,880,883],[12,13,14,18],"blockquote",{},[15,16,17],"p",{},"數字來自 arXiv 2611.10649（v1 11-07、v2 11-11）與官方程式庫 README。作者是華為 Cloud & AI Lab；程式庫是 Apache 2.0。我們沒有獨立重跑，「我認為」的地方會標出來。MACaQ-R 的 R 是 refine。",[15,19,20,21,28,29],{},"論文：",[22,23,27],"a",{"href":24,"rel":25},"https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.10649",[26],"nofollow","arXiv 2611.10649","｜程式庫：",[22,30,33],{"href":31,"rel":32},"https:\u002F\u002Fgithub.com\u002Fhuawei-csl\u002FMACaQ",[26],"huawei-csl\u002FMACaQ",[35,36,37],"h2",{"id":37},"先講一個畫面",[15,39,40],{},"量化一個 122B 模型，64 分鐘、32.5 GB 顯示記憶體，一台單卡機就夠。就算你手上沒有 122B 也沒關係。同一套做法讓 MACaQ-R 能修好你已經下載的那個 GGUF。",[15,42,43],{},"這件事以前得先有「放得下 244 GB」的機器。",[15,45,46,50],{},[47,48,49],"code",{},"Qwen3.5-122B-A10B"," 的 FP16 權重是 244 GB。它是 96 GB 卡的 2.5 倍，兩張放不下，三張才勉強。",[15,52,53],{},"量化聽起來是「把模型變小」。但量化這個動作本身，得先有模型。旋轉、校準、算 Hessian，每一樣都吃記憶體。Hessian 用來估計「這個權重改一點，輸出會被影響多少」。",[15,55,56],{},"所以大模型量化卡住的地方，常常不是算力，是「你得先裝得下它」。",[15,58,59],{},"MACaQ 換了一個順序：一次只需要一層在 GPU 上。其餘的權重留在主機記憶體，用完就換下一層。",[15,61,62],{},"這篇講三件事。舊方法為什麼被記憶體卡住。它把哪個成本搬到推論時、代價多少。還有你手上那個 GGUF（llama.cpp 的模型檔格式）怎麼變好。",[35,64,66],{"id":65},"量化-122b要先裝得下-122b","量化 122B，要先裝得下 122B",[15,68,69],{},"先講為什麼量化大模型要旋轉。",[15,71,72],{},"權重和激活裡總有幾個特別大的值。壓到 4-bit 時，一個離群值會把整組的尺度撐大。其他值就被壓成同一個格子。",[15,74,75],{},"（這裡的 4-bit 是 MXFP4，一種 4-bit 浮點。每 32 個值共用一個 2 的冪次尺度。）",[15,77,78],{},"做法是先把資料旋轉一下。旋轉不改變矩陣乘法的結果，但會把離群值攤開。每個座標看起來比較像鐘形分布，格點才夠用。",[15,80,81],{},"問題在「什麼時候旋轉」。",[15,83,84],{},"QuaRot、SpinQuant、DuQuant++ 都是這一類。它們把旋轉矩陣融進相鄰層的權重。推論時幾乎不用多做，旋轉已經變成權重的一部分。這是它們最漂亮的地方。",[15,86,87],{},"代價在量化時。旋轉要跨層傳遞，你得知道這個模型的層怎麼排。attention 的輸出投影在哪、residual 從哪裡加回來。架構不同，融法就不同。",[15,89,90],{},"而且層跟層之間有了依賴，量化時就沒辦法「只看一層」。",[15,92,93],{},"把成本放在量化時還是推論時，是這個領域主要的分岔。沒有誰比較笨。",[15,95,96],{},"表 1：四種做法的旋轉放在哪",[98,99,100,122],"table",{},[101,102,103],"thead",{},[104,105,106,110,113,116,119],"tr",{},[107,108,109],"th",{},"方法",[107,111,112],{},"旋轉放哪",[107,114,115],{},"量化時跨層",[107,117,118],{},"只裝一層",[107,120,121],{},"推論成本",[123,124,125,143,157,170],"tbody",{},[104,126,127,131,134,137,140],{},[128,129,130],"td",{},"QuaRot",[128,132,133],{},"融進相鄰層權重",[128,135,136],{},"要",[128,138,139],{},"不行",[128,141,142],{},"幾乎不另做",[104,144,145,148,151,153,155],{},[128,146,147],{},"SpinQuant",[128,149,150],{},"融進相鄰層權重（旋轉可學習）",[128,152,136],{},[128,154,139],{},[128,156,142],{},[104,158,159,162,164,166,168],{},[128,160,161],{},"DuQuant++",[128,163,133],{},[128,165,136],{},[128,167,139],{},[128,169,142],{},[104,171,172,175,178,181,184],{},[128,173,174],{},"MACaQ",[128,176,177],{},"推論時線上做",[128,179,180],{},"不用",[128,182,183],{},"可以",[128,185,186],{},"多 2.3%",[15,188,189],{},"表 1 是概念對照，不是本文的實測名單。這篇只拿 DuQuant++ 與 MR-GPTQ 對照；論文另外也比了 QuaRot 與 FlatQuant。",[15,191,192],{},"它要講的只有一件事。MACaQ 不是更聰明的旋轉，是把旋轉換到另一個時間點。",[35,194,195],{"id":195},"它把旋轉留在推論時做",[15,197,198],{},"MACaQ 選了另一邊：旋轉不融進權重，留在推論時算。",[15,200,201],{},"每一層的 activation 進來，先乘上一個 per-channel 尺度。接著做 block-Hadamard 轉換。Hadamard 是快速、可逆的線性變換，負責把離群值攤平。",[15,203,204],{},"權重那邊，預先把反轉換吸收掉。所以結果等價，只是把同一份乘法從量化時搬到推論時。",[15,206,207],{},"關鍵不是省不省，是誰依賴誰。",[15,209,210],{},"旋轉留線上之後，每一層只需要自己的權重、自己的 activation 統計。層與層之間沒有依賴了。",[15,212,213],{},"於是量化可以串流。搬一層進 GPU、算完、寫出去、釋放，再搬下一層。同一時間，GPU 上只有一層。",[15,215,216],{},"論文的做法是 lazy loading，其餘權重留在主機記憶體。原文的說法很直白：只有正在處理的那一層需要待在 GPU 上。",[15,218,219,220,222],{},"表 2：",[47,221,49],{}," 這次的數字（論文 §Large-Model Quantization）",[98,224,225,235],{},[101,226,227],{},[104,228,229,232],{},[107,230,231],{},"項目",[107,233,234],{},"數字",[123,236,237,245,253,261,269,277,285],{},[104,238,239,242],{},[128,240,241],{},"總參數",[128,243,244],{},"122B（MoE 混合專家，A10B 是每次啟用的參數）",[104,246,247,250],{},[128,248,249],{},"FP16 權重",[128,251,252],{},"約 244 GB",[104,254,255,258],{},[128,256,257],{},"機器",[128,259,260],{},"單張 96 GB GPU（只給 504 TFLOPS FP16、1.79 TB\u002Fs，不寫型號）",[104,262,263,266],{},[128,264,265],{},"量化時 GPU 峰值",[128,267,268],{},"32.5 GB",[104,270,271,274],{},[128,272,273],{},"量化時間",[128,275,276],{},"64 分鐘（32 flips、512K 校準 token）",[104,278,279,282],{},[128,280,281],{},"WikiText-2 困惑度",[128,283,284],{},"5.14（FP16 是 5.06）",[104,286,287,290],{},[128,288,289],{},"量化後推論需求",[128,291,292],{},"74 GB，同一張卡跑得起來",[15,294,295],{},"如圖 1，左邊是「旋轉跟著層走」，量化時兩層互相依賴。右邊是「旋轉留在線上」，同一時間只有一層在 GPU 裡。",[297,298,301,302,301,309],"figure",{"className":299},[300],"article-figure","\n  ",[303,304],"video",{"src":305,"poster":306,"autoPlay":307,"loop":307,"muted":307,"playsInline":307,"ariaLabel":308},"\u002Fimages\u002Fblog\u002Fmacaq-one-layer-at-a-time\u002Fstreaming.mp4","\u002Fimages\u002Fblog\u002Fmacaq-one-layer-at-a-time\u002Fstreaming-poster.jpg",true,"左邊是旋轉跨層傳播、量化時兩層依賴；右邊是每層獨立、一次只有一層進 GPU",[310,311,312],"figcaption",{},"圖 1：左邊的轉換跟著層走，量化時要跨層處理；右邊把轉換留到推論，一層進、一層出。",[15,314,315,316,320],{},"讀者最容易誤讀的一格是 32.5 GB。那是",[317,318,319],"strong",{},"顯示記憶體","的高峰，不是全部。那 244 GB 的 FP16 權重還在主機記憶體裡，只是不用同時進 GPU。",[15,322,323],{},"所以真正的門檻是「一張夠大的卡，加一台夠大的主機」。不是「32.5 GB 就夠」。這點很容易漏。",[15,325,326],{},"還有一件事：它不改架構。同一套流程拿去跑影像模型也行。程式庫列了三個範例。FLUX.2 Klein 4B 是影像，Wan2.2 是影片，stable-audio 是音訊。論文自己只示範了一個影像模型，說法是「不需要針對架構做修改」。",[15,328,329],{},"我認為對手上什麼模型都有的人，這個「模型無關」比 64 分鐘更有價值。",[35,331,333],{"id":332},"代價多-23","代價：多 2.3%",[15,335,336],{},"線上做轉換不是免費的。相對純 MXFP4，Qwen3-32B 在 vLLM 上端到端多 2.3%。122B 那組，論文沒有量推論開銷。",[15,338,339],{},"論文另外用一張圖拆給你看。這個開銷在 decode 端量：batch 1、context 512。模型越大，占比越小，因為矩陣乘法越久，旋轉那幾個乘法相對越輕。",[15,341,342],{},"我認為這 2.3% 買到的是「不用理解架構」。這是整篇的樞紐。",[15,344,345],{},"另一組常被搞混的數字在表 3。模型是 Qwen3.8-27B，後端是 vLLM。相對 BF16，prefill 是 2.33×、decode 是 2.58×。",[15,347,348],{},"這是「相對 BF16 的吞吐量」。數字是 2.33×／2.58×，不是「相對 MXFP4 快 2.33 倍」。兩個基準差很遠，寫混了整段就報廢。",[15,350,351,352,355],{},"表 3：",[47,353,354],{},"Qwen3.8-27B"," 在 vLLM 上的品質與速度（官方 README，基準 BF16 = 1.00×）",[98,357,358,374],{},[101,359,360],{},[104,361,362,365,368,371],{},[107,363,364],{},"精度",[107,366,367],{},"困惑度 ↓（README 未寫資料集）",[107,369,370],{},"prefill",[107,372,373],{},"decode",[123,375,376,389],{},[104,377,378,381,384,387],{},[128,379,380],{},"BF16",[128,382,383],{},"2.8463",[128,385,386],{},"1.00×",[128,388,386],{},[104,390,391,394,397,400],{},[128,392,393],{},"MACaQ W4A4",[128,395,396],{},"2.8994",[128,398,399],{},"2.33×",[128,401,402],{},"2.58×",[15,404,405],{},"W4A4 的意思是權重 4-bit、激活也 4-bit。困惑度從 2.8463 到 2.8994，是 +1.9%。這不是「幾乎無損」，是「有損但很便宜」。",[15,407,408],{},"表 3 的數字來自官方 README，跟表 2 不是同一組實驗。別跨表比。",[35,410,412],{"id":411},"同樣的品質時間少一截","同樣的品質，時間少一截",[15,414,415],{},"4-bit 的格點有限。四捨五入（round-to-nearest，RTN）選最近的格子。它只看自己，不看別人。",[15,417,418],{},"MACaQ 多做一步：翻一個碼。",[15,420,421],{},"做法是把單一 FP4 碼挪到相鄰的可表示值。不是重新量化，只是換一格。",[15,423,424],{},"要不要翻，由整層的損失決定。用 Hessian 近似：這一格改了，輸出會被影響多少。然後貪心地翻，一次一個。",[15,426,427],{},"為什麼便宜？候選只有左邊那格和右邊那格。搜尋空間小，不用做連續最佳化。",[15,429,430],{},"有兩個旋鈕：翻幾輪（flips），和用多少校準 token。要快就少翻，要好就多翻。表 4 是論文附錄逐格往上爬的原始數字（Llama-3-8B）。",[15,432,433],{},"表 4：Llama-3-8B 的取捨（MXFP4 W4A4，WikiText-2 困惑度）",[98,435,436,448],{},[101,437,438],{},[104,439,440,443,445],{},[107,441,442],{},"設定",[107,444,273],{},[107,446,447],{},"困惑度 ↓",[123,449,450,461,472,483,494,505,516,526],{},[104,451,452,455,458],{},[128,453,454],{},"FP16（不量化，參考）",[128,456,457],{},"—",[128,459,460],{},"6.14",[104,462,463,466,469],{},[128,464,465],{},"RTN：只四捨五入",[128,467,468],{},"2.2 秒",[128,470,471],{},"8.29",[104,473,474,477,480],{},[128,475,476],{},"MACaQ：0 flips（旋轉＋指數篩選）",[128,478,479],{},"5.5 秒",[128,481,482],{},"7.64",[104,484,485,488,491],{},[128,486,487],{},"MACaQ：1 塊校準（2048 token）",[128,489,490],{},"14.8 秒",[128,492,493],{},"7.11",[104,495,496,499,502],{},[128,497,498],{},"MACaQ：128 flips、31 塊",[128,500,501],{},"88.1 秒",[128,503,504],{},"6.91",[104,506,507,510,513],{},[128,508,509],{},"MACaQ：512 flips、977 塊",[128,511,512],{},"636.9 秒",[128,514,515],{},"6.85",[104,517,518,521,524],{},[128,519,520],{},"MR-GPTQ：128 塊",[128,522,523],{},"319.1 秒",[128,525,493],{},[104,527,528,531,534],{},[128,529,530],{},"DuQuant++（GPTQ）：128 塊",[128,532,533],{},"624.4 秒",[128,535,536],{},"7.16",[15,538,539],{},"MACaQ 那幾列是同一條曲線上的不同取捨，校準量不一樣。MR-GPTQ 與 DuQuant++ 那兩列是各自的 128 塊設定。別逐列比速度，那是不同預算下的單點。論文自己的公平對照是 128 flips、約 20 萬 token，Llama-3-8B 上約 150 秒，落在 6.90。",[15,541,542],{},"這張表比一個「快 11×」的標題有用。你可以直接挑自己的那一列。",[15,544,545],{},"摘要的說法是：在 Llama-3-8B 上打平對手最好的困惑度，時間只要 1\u002F11。我們不換算成一個比值放進文章。表 4 是逐格往上爬的單點，摘要那條是曲線。要選型，看表。",[15,547,548],{},"如圖 2，翻一個碼的意思：四捨五入選最近的格子，翻碼是換到隔壁一格。但那次換要不要做，看的是整層的損失。",[297,550,301,552,301,558],{"className":551},[300],[553,554],"img",{"src":555,"alt":556,"loading":557},"\u002Fimages\u002Fblog\u002Fmacaq-one-layer-at-a-time\u002Fflip-grid.svg","FP4 格點示意：四捨五入到最近的點，以及翻到相鄰點的候選，取捨由整層損失決定","lazy",[310,559,560],{},"圖 2：格點之間只有鄰居可選，所以「翻一個碼」很便宜；貴的是判斷要不要翻——那要算整層的損失。示意圖，位置不是論文實測值。",[15,562,563],{},"還有一件事要講清楚：這不是「4-bit 無損」。最好的設定也是 6.14 變 6.85，困惑度多了大約 12%。RTN 是 8.29。兩者都是「有損」，差別在損失多少、以及你願不願意付時間。",[35,565,567],{"id":566},"你手上那個-gguf可以修","你手上那個 GGUF，可以修",[15,569,570],{},"上面講的是「重新量化一個模型」。多數人手上沒有那個情境，只有一堆下載好的 GGUF。",[15,572,573],{},"MACaQ-R 是給這個情境的。拿一個現成的 GGUF，配它的 FP16 原檔當 teacher。跑一次 refine。",[15,575,576],{},"碼重排，格式與 block scale 都不動。所以修完的檔案，用標準 llama.cpp 就能跑。",[15,578,579,580,583],{},"表 5：",[47,581,582],{},"Qwen3.6-35B-A3B"," 的 MACaQ-R 前後（官方 README 自測）",[98,585,586,599],{},[101,587,588],{},[104,589,590,593,596],{},[107,591,592],{},"版本",[107,594,595],{},"GGUF 大小",[107,597,598],{},"對 FP16 的 KL 散度 ↓",[123,600,601,612],{},[104,602,603,606,609],{},[128,604,605],{},"Unsloth UD-Q4_K_XL",[128,607,608],{},"22.36 GB",[128,610,611],{},"0.015731",[104,613,614,617,619],{},[128,615,616],{},"MACaQ-R Q4_K_XL",[128,618,608],{},[128,620,621],{},"0.013367",[15,623,624],{},"KL 散度是「兩個分布差多少」，越低越好。同一個大小，從 0.015731 降到 0.013367，低 15.0%。檔案沒變大，執行方式也沒變。換的只是每一格碼選哪個值。",[15,626,627],{},"指令是 README 原文，四行：",[629,630,635],"pre",{"className":631,"code":632,"language":633,"meta":634,"style":634},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight",".venv-llamacpp\u002Fbin\u002Fpython llamacpp\u002Fmacaq.py refine \\\n  --model models\u002Fmodel-q4.gguf --hf models\u002Foriginal \\\n  --teacher models\u002Foriginal-f16.gguf --calibration data\u002Fcalibration.txt \\\n  --output models\u002Fmodel-macaq-r.gguf --work-dir work\u002Frefinement\n","bash","",[47,636,637,657,674,691],{"__ignoreMap":634},[638,639,642,646,650,653],"span",{"class":640,"line":641},"line",1,[638,643,645],{"class":644},"sBMFI",".venv-llamacpp\u002Fbin\u002Fpython",[638,647,649],{"class":648},"sfazB"," llamacpp\u002Fmacaq.py",[638,651,652],{"class":648}," refine",[638,654,656],{"class":655},"sTEyZ"," \\\n",[638,658,660,663,666,669,672],{"class":640,"line":659},2,[638,661,662],{"class":648},"  --model",[638,664,665],{"class":648}," models\u002Fmodel-q4.gguf",[638,667,668],{"class":648}," --hf",[638,670,671],{"class":648}," models\u002Foriginal",[638,673,656],{"class":655},[638,675,677,680,683,686,689],{"class":640,"line":676},3,[638,678,679],{"class":648},"  --teacher",[638,681,682],{"class":648}," models\u002Foriginal-f16.gguf",[638,684,685],{"class":648}," --calibration",[638,687,688],{"class":648}," data\u002Fcalibration.txt",[638,690,656],{"class":655},[638,692,694,697,700,703],{"class":640,"line":693},4,[638,695,696],{"class":648},"  --output",[638,698,699],{"class":648}," models\u002Fmodel-macaq-r.gguf",[638,701,702],{"class":648}," --work-dir",[638,704,705],{"class":648}," work\u002Frefinement\n",[15,707,708],{},"修完直接跑，後端不用換：",[629,710,712],{"className":631,"code":711,"language":633,"meta":634,"style":634},"llama cli -m models\u002Fmodel-macaq-r.gguf\n",[47,713,714],{"__ignoreMap":634},[638,715,716,719,722,725],{"class":640,"line":641},[638,717,718],{"class":644},"llama",[638,720,721],{"class":648}," cli",[638,723,724],{"class":648}," -m",[638,726,727],{"class":648}," models\u002Fmodel-macaq-r.gguf\n",[15,729,730,731,734],{},"llama.cpp 不同版本的執行檔名稱不一樣。舊版是 ",[47,732,733],{},"llama-cli","，照你裝的那個用。",[15,736,737],{},"看起來簡單，前置條件不簡單。README 有寫的三條：",[739,740,741,753,759],"ul",{},[742,743,744,745,748,749,752],"li",{},"要 ",[317,746,747],{},"Python 3.12","，而且每個 backend 一個獨立環境。不是 ",[47,750,751],{},"pip install"," 就結束",[742,754,744,755,758],{},[317,756,757],{},"FP16 原檔","當 teacher。只有 GGUF、沒有原檔的人，這一步就卡住",[742,760,761],{},"要一份校準文字檔，以及對應的原始權重目錄",[15,763,764,765,768],{},"還有一個限制。表 5 只有一組對照，單一模型、單一 baseline。",[317,766,767],{},"不能推廣成「所有 GGUF 都能改善 15%」。"," 你的模型可能進步更多，也可能沒差。要判斷，只能自己量 KL。",[15,770,771,772,776],{},"修完之後跑多快，不是這篇的主題。那取決於位元寬與 kernel 怎麼寫。系列裡寫過：",[22,773,775],{"href":774},"\u002Fblog\u002Fquantization-kernel-dispatch","量化會不會加速，取決於你怎麼寫 kernel","。",[35,778,779],{"id":779},"別過度解讀",[15,781,782,783,786],{},"論文自己把代價寫在 Limitations 第一句。layer-local 的設計，意味著",[317,784,785],{},"跨層錯誤補償做不到","。要補，就得再加一趟跨層統計的 pass。那大概會退回其他方法的時間。",[15,788,789],{},"這是設計上的取捨，不是疏忽。",[15,791,792],{},"其他該知道的：",[739,794,795,801,807,813,819,825,835],{},[742,796,797,800],{},[317,798,799],{},"沒有第三方複現。"," 11-07 首發，對照的 DuQuant++、MR-GPTQ 都是作者自己跑的",[742,802,803,806],{},[317,804,805],{},"W4A4 的公開數字集中在少數模型。"," 別把 Qwen3.8-27B 的 +1.9% 當通則",[742,808,809,812],{},[317,810,811],{},"MoE 只量化 expert 層。"," 另外 embeddings 與 lm-head 在所有模型上都不量化",[742,814,815,818],{},[317,816,817],{},"122B 那組的預算跟其他實驗不同","：32 flips、512K 校準 token；公平對照時用的是 128 flips、約 20 萬 token",[742,820,821,824],{},[317,822,823],{},"Qwen3.5／3.6／3.8 是不同世代","，數字不要互相對照",[742,826,827,830,831,834],{},[317,828,829],{},"Ascend 不是開箱即用。"," 要 CANN 與 ",[47,832,833],{},"torch_npu","。官方只給 kernel PR，整模型還要接推理 runtime",[742,836,837,840],{},[317,838,839],{},"32.5 GB 是顯示記憶體，不是全部。"," 你的主機要有地方放那 244 GB",[15,842,843],{},"還有那個 +2.3%：它是對「純 MXFP4」算的。如果你的比較基準是 BF16，MACaQ 還是快兩倍多（表 3）。同一個方法，換一個分母，結論就換一個方向。",[35,845,846],{"id":846},"明天可以做的三件事",[848,849,850,860,866],"ol",{},[742,851,852,855,856,776],{},[317,853,854],{},"先量基準","。如果你手上已經有一個 GGUF，配 FP16 原檔算一次 KL，就有自己的一條線。README 沒有附量 KL 的腳本，得自己找。沒有基準，任何「低 15%」都是別人的數字。也先確認你手上那個不是壞的，見",[22,857,859],{"href":858},"\u002Fblog\u002Fbroken-on-arrival-quantized-models","你下載的量化模型可能是壞的",[742,861,862,865],{},[317,863,864],{},"跑一次 refine","。有原檔就跑，跑完用你平常那支 llama.cpp 跑同一題，比輸出。沒變好，你只損失一次 refine 的時間。",[742,867,868,871],{},[317,869,870],{},"要自己量化 100B 級模型的人，現在一台單卡機就夠。"," 前提是主機記憶體放得下 FP16 權重。",[15,873,874,875,879],{},"如果只是想在自己電腦上跑大模型，先看",[22,876,878],{"href":877},"\u002Fblog\u002Fstrata-125b-on-12gb","125B 模型怎麼跑在 12GB 顯卡上","。那是另一條路：不改權重，改誰在什麼時候被載入。",[15,881,882],{},"量化的門檻換了位置：從一張卡，搬到一台主機。",[884,885,886],"style",{},"html pre.shiki code .sBMFI, html code.shiki .sBMFI{--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B}html pre.shiki code .sfazB, html code.shiki .sfazB{--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D}html pre.shiki code .sTEyZ, html code.shiki .sTEyZ{--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":634,"searchDepth":659,"depth":659,"links":888},[889,890,891,892,893,894,895,896],{"id":37,"depth":659,"text":37},{"id":65,"depth":659,"text":66},{"id":195,"depth":659,"text":195},{"id":332,"depth":659,"text":333},{"id":411,"depth":659,"text":412},{"id":566,"depth":659,"text":567},{"id":779,"depth":659,"text":779},{"id":846,"depth":659,"text":846},"\u002Fimages\u002Fblog\u002Fmacaq-one-layer-at-a-time.svg","標題「量化 122B，一次只裝一層」的技術筆記封面","2026-11-19","量化一個 122B 模型只要 64 分鐘、32.5 GB 顯示記憶體——因為它不把旋轉矩陣跨層傳播，每層獨立，一次只裝一層。權重留在主機記憶體；在 32B 上量到的代價是推論多 2.3%。MACaQ-R 在一個 35B GGUF 上把 KL 修低 15%，前提是有 FP16 原檔。",false,"md","zh-TW",{},"\u002Fimages\u002Fblog\u002Fmacaq-one-layer-at-a-time.jpg","\u002Fblog\u002Fzh-tw\u002Fmacaq-one-layer-at-a-time",{"title":6,"description":900},"macaq-one-layer-at-a-time","blog\u002Fzh-TW\u002Fmacaq-one-layer-at-a-time",[911,912,913],"quantization","local-llm","gguf",null,"im6BNKmEd2jNobz5OpyhsjwIzUViuMdNqM7ywhcggCI",{"id":917,"title":918,"author":7,"body":919,"cover":1699,"coverAlt":1700,"date":899,"description":1701,"draft":901,"extension":902,"featured":901,"lang":1702,"meta":1703,"navigation":307,"ogImage":1704,"path":1705,"seo":1706,"slug":908,"stem":1707,"tags":1708,"translationKey":908,"updated":914,"__hash__":1709},"blog\u002Fblog\u002Fen\u002Fmacaq-one-layer-at-a-time.md","Quantize a 122B Model, One Layer at a Time",{"type":9,"value":920,"toc":1689},[921,936,940,943,946,951,954,957,960,963,967,970,973,976,979,982,985,988,991,994,997,1075,1078,1081,1085,1088,1091,1094,1097,1100,1106,1175,1178,1189,1196,1199,1202,1205,1209,1212,1215,1218,1221,1228,1234,1272,1275,1278,1282,1285,1288,1291,1294,1297,1300,1303,1399,1402,1405,1408,1411,1421,1424,1428,1431,1434,1437,1443,1476,1479,1482,1530,1533,1547,1553,1556,1579,1586,1593,1597,1604,1607,1654,1657,1661,1684,1687],[12,922,923],{},[15,924,925,926,930,931,935],{},"Sources. The ",[22,927,929],{"href":24,"rel":928},[26],"paper"," (arXiv 2611.10649, v1 Nov 7 2026, v2 Nov 11 2026; Huawei Cloud & AI Lab) supplies the 122B run, the Llama-3-8B ladder in its appendix and the limitations. The ",[22,932,934],{"href":31,"rel":933},[26],"repository README"," (Apache 2.0) supplies the vLLM numbers, the MACaQ-R comparison and the command. We did not rerun anything; where a claim is ours, it says so.",[35,937,939],{"id":938},"the-model-does-not-fit","The model does not fit",[15,941,942],{},"A 122B model, quantized in 64 minutes, peaking at 32.5 GB of GPU memory, on a single-GPU machine. That matters even if you never touch a 122B model: the same layout is what lets MACaQ-R improve a GGUF you already downloaded.",[15,944,945],{},"Before this, you first needed a machine that could hold 244 GB.",[15,947,948,950],{},[47,949,49],{}," weighs 244 GB in FP16. That is 2.5× a 96 GB card. Two cards hold 192 GB, so it does not fit. Three hold 288 GB, with 44 GB to spare.",[15,952,953],{},"Quantization sounds like making a model smaller. But quantization is an operation that needs the model first. Rotations, calibration, Hessians: each one costs memory. Hessians here mean \"how far does the output move if this weight nudges.\"",[15,955,956],{},"So the wall for large-model quantization is rarely arithmetic throughput. It is the simple fact that you have to hold the thing.",[15,958,959],{},"MACaQ reverses the order. Only one layer has to be on the GPU at a time. Everything else waits in host memory.",[15,961,962],{},"Four things below: why the old layout hits the memory wall, what the rotation costs once it moves, how the flips buy quality back, and how to improve the GGUF you already have.",[35,964,966],{"id":965},"quantizing-122b-means-holding-122b","Quantizing 122B means holding 122B",[15,968,969],{},"Start with why quantization needs a rotation at all.",[15,971,972],{},"Every weight matrix and activation has a few large values. Squeezing to 4 bits, one outlier stretches the shared block scale. Everything else collapses onto the same grid point.",[15,974,975],{},"(The 4 bits here are MXFP4: a 4-bit float where 32 values share one power-of-two scale.)",[15,977,978],{},"The fix is to rotate the data first. A rotation does not change what the matrix product means, but it spreads the outliers out. Every coordinate looks closer to a bell curve, and the grid starts to cover it.",[15,980,981],{},"The question is when the rotation happens.",[15,983,984],{},"QuaRot, SpinQuant and DuQuant++ are one family. They fuse the rotation into the neighbouring layer's weights. At inference there is almost nothing extra to do, because the rotation is now part of the weight. That is the elegant part.",[15,986,987],{},"The bill arrives at quantization time. The rotation travels across layers, so you need to know how the model is wired: where the attention output projection sits, where the residual comes back in. Different architecture, different fusion.",[15,989,990],{},"And once layers depend on each other, you cannot quantize one at a time.",[15,992,993],{},"Whether you pay at quantization time or at inference is the real fork in this field.",[15,995,996],{},"Table 1: where each method keeps the rotation",[98,998,999,1018],{},[101,1000,1001],{},[104,1002,1003,1006,1009,1012,1015],{},[107,1004,1005],{},"Method",[107,1007,1008],{},"Rotation lives",[107,1010,1011],{},"Cross-layer at quant time",[107,1013,1014],{},"One layer at a time",[107,1016,1017],{},"Inference cost",[123,1019,1020,1036,1049,1061],{},[104,1021,1022,1024,1027,1030,1033],{},[128,1023,130],{},[128,1025,1026],{},"fused into the next layer's weights",[128,1028,1029],{},"yes",[128,1031,1032],{},"no",[128,1034,1035],{},"almost nothing extra",[104,1037,1038,1040,1043,1045,1047],{},[128,1039,147],{},[128,1041,1042],{},"fused, with learned rotations",[128,1044,1029],{},[128,1046,1032],{},[128,1048,1035],{},[104,1050,1051,1053,1055,1057,1059],{},[128,1052,161],{},[128,1054,1026],{},[128,1056,1029],{},[128,1058,1032],{},[128,1060,1035],{},[104,1062,1063,1065,1068,1070,1072],{},[128,1064,174],{},[128,1066,1067],{},"online, at inference",[128,1069,1032],{},[128,1071,1029],{},[128,1073,1074],{},"+2.3%",[15,1076,1077],{},"Table 1 is a conceptual map, not the paper's benchmark list. This essay compares against DuQuant++ and MR-GPTQ only; the paper also benchmarks QuaRot and FlatQuant.",[15,1079,1080],{},"The map says one thing: the rotation itself is not smarter, it just runs at a different time. The quality comes later, from the flips.",[35,1082,1084],{"id":1083},"moving-the-rotation-to-inference-time","Moving the rotation to inference time",[15,1086,1087],{},"MACaQ takes the other branch. The rotation stays out of the weights and runs at inference.",[15,1089,1090],{},"Each layer's incoming activations get a per-channel scale, then a block-Hadamard transform. Hadamard is a fast, invertible linear map whose job is to flatten outliers.",[15,1092,1093],{},"The inverse is absorbed into the weights at quantization time. The arithmetic is equivalent. The same multiply just moves from the quantizer to the runtime.",[15,1095,1096],{},"The interesting consequence is not the saving, it is who depends on whom. With the rotation online, a layer needs only its own weights and its own activation statistics. Nothing points backwards.",[15,1098,1099],{},"So quantization can stream: load a layer onto the GPU, quantize it, write it out, free it, load the next. The paper calls this streaming with lazy loading, and puts it in one sentence; the remaining weights stay in host memory, and only the layer being processed has to sit on the GPU.",[15,1101,1102,1103,1105],{},"Table 2: the ",[47,1104,49],{}," run",[98,1107,1108,1118],{},[101,1109,1110],{},[104,1111,1112,1115],{},[107,1113,1114],{},"Item",[107,1116,1117],{},"Value",[123,1119,1120,1128,1136,1144,1151,1159,1167],{},[104,1121,1122,1125],{},[128,1123,1124],{},"Parameters",[128,1126,1127],{},"122B (mixture-of-experts; about 10B active per token)",[104,1129,1130,1133],{},[128,1131,1132],{},"FP16 weights",[128,1134,1135],{},"about 244 GB",[104,1137,1138,1141],{},[128,1139,1140],{},"Machine",[128,1142,1143],{},"one 96 GB GPU (the paper lists 504 TFLOPS FP16 and 1.79 TB\u002Fs, not a model name)",[104,1145,1146,1149],{},[128,1147,1148],{},"Peak GPU memory while quantizing",[128,1150,268],{},[104,1152,1153,1156],{},[128,1154,1155],{},"Quantization time",[128,1157,1158],{},"64 minutes (32 flips, 512K calibration tokens)",[104,1160,1161,1164],{},[128,1162,1163],{},"WikiText-2 perplexity",[128,1165,1166],{},"5.14 (FP16 is 5.06)",[104,1168,1169,1172],{},[128,1170,1171],{},"Footprint of the result",[128,1173,1174],{},"74 GB; runs on the same card",[15,1176,1177],{},"Figure 1, left: the rotation travels with the layers, so quantizing two of them couples them. Right: the rotation stays online, so one layer is on the GPU at a time.",[297,1179,301,1181,301,1186],{"className":1180},[300],[303,1182],{"src":1183,"poster":1184,"autoPlay":307,"loop":307,"muted":307,"playsInline":307,"ariaLabel":1185},"\u002Fimages\u002Fblog\u002Fen\u002Fmacaq-one-layer-at-a-time\u002Fstreaming.mp4","\u002Fimages\u002Fblog\u002Fen\u002Fmacaq-one-layer-at-a-time\u002Fstreaming-poster.jpg","Left: the rotation propagates across layers and couples them during quantization. Right: each layer stands alone, one layer on the GPU at a time.",[310,1187,1188],{},"Figure 1: on the left the transform travels with the layers, so quantization has to look across them; on the right it stays at inference and each layer goes in and out on its own. The 32.5 GB badge is the 122B number from Table 2, not a general bound.",[15,1190,1191,1192,1195],{},"The number people misread is 32.5 GB. That is ",[317,1193,1194],{},"device memory"," at its peak, not the whole footprint. The 244 GB of FP16 weights is still in host memory; it is just never resident on the GPU at once.",[15,1197,1198],{},"So the real requirement is a big enough card plus a big enough host. Not \"32.5 GB is enough.\" Almost every repost drops that half.",[15,1200,1201],{},"No architecture changes, either. The repo lists FLUX.2 Klein 4B for images, Wan2.2 for video and stable-audio for audio. The paper itself demonstrates one image model and describes the method as applying \"without architecture-specific modifications.\"",[15,1203,1204],{},"If you keep a zoo of models, that portability matters more than the 64 minutes.",[35,1206,1208],{"id":1207},"the-bill-23","The bill: 2.3%",[15,1210,1211],{},"The online transform is not free. Against pure MXFP4, MACaQ adds 2.3% end-to-end on Qwen3-32B in vLLM. The paper never measures the 122B run; that is a 32B number.",[15,1213,1214],{},"The paper's Figure 3 breaks it down: the cost is measured on decode, batch 1, context 512. It shrinks as models grow, because bigger matrices take longer and a few extra multiplies weigh less.",[15,1216,1217],{},"I think what that 2.3% buys is \"you do not have to understand the architecture.\" That is the hinge of the design.",[15,1219,1220],{},"The other number that gets mangled is in Table 3: W4A4 (4-bit weights, 4-bit activations) on Qwen3.8-27B in vLLM, a serving engine. MACaQ runs at 2.33× prefill and 2.58× decode.",[15,1222,1223,1224,1227],{},"Those are throughput ratios ",[317,1225,1226],{},"against BF16",", not against pure MXFP4. The two denominators answer different questions; mixing them wrecks the paragraph.",[15,1229,1230,1231,1233],{},"Table 3: ",[47,1232,354],{}," in vLLM (README; BF16 = 1.00×)",[98,1235,1236,1250],{},[101,1237,1238],{},[104,1239,1240,1243,1246,1248],{},[107,1241,1242],{},"Precision",[107,1244,1245],{},"Perplexity ↓ (README; dataset unstated)",[107,1247,370],{},[107,1249,373],{},[123,1251,1252,1262],{},[104,1253,1254,1256,1258,1260],{},[128,1255,380],{},[128,1257,383],{},[128,1259,386],{},[128,1261,386],{},[104,1263,1264,1266,1268,1270],{},[128,1265,393],{},[128,1267,396],{},[128,1269,399],{},[128,1271,402],{},[15,1273,1274],{},"Perplexity goes from 2.8463 to 2.8994, a 1.9% increase. That is not lossless; it is lossy and cheap, which is a different claim.",[15,1276,1277],{},"Table 3 comes from the README and is a different experiment from Table 2. Do not compare across tables.",[35,1279,1281],{"id":1280},"the-rounding-is-the-easy-part","The rounding is the easy part",[15,1283,1284],{},"Four bits give you a small grid. Round-to-nearest (RTN) picks the closest point. It looks only at the weight itself.",[15,1286,1287],{},"MACaQ does one more thing: it flips a code.",[15,1289,1290],{},"A flip moves a single FP4 code to an adjacent representable value. Not a re-quantization, just a different point in the same grid.",[15,1292,1293],{},"Whether to flip is decided by the whole layer's loss. A Hessian approximation asks how far the output moves if this code changes. Then it flips greedily, one code at a time.",[15,1295,1296],{},"Why is that cheap? The only candidates are the neighbour on the left and the neighbour on the right. The search space is tiny. No continuous optimization.",[15,1298,1299],{},"Two knobs control it: how many flips, and how many calibration tokens. Fast means few flips, good means many. Table 4 is the appendix ladder on Llama-3-8B.",[15,1301,1302],{},"Table 4: the trade on Llama-3-8B (MXFP4 W4A4, WikiText-2 perplexity)",[98,1304,1305,1318],{},[101,1306,1307],{},[104,1308,1309,1312,1315],{},[107,1310,1311],{},"Setting",[107,1313,1314],{},"Time",[107,1316,1317],{},"Perplexity ↓",[123,1319,1320,1329,1339,1349,1359,1369,1379,1389],{},[104,1321,1322,1325,1327],{},[128,1323,1324],{},"FP16 (reference)",[128,1326,457],{},[128,1328,460],{},[104,1330,1331,1334,1337],{},[128,1332,1333],{},"RTN only",[128,1335,1336],{},"2.2 s",[128,1338,471],{},[104,1340,1341,1344,1347],{},[128,1342,1343],{},"MACaQ, 0 flips (rotation + exponent screen)",[128,1345,1346],{},"5.5 s",[128,1348,482],{},[104,1350,1351,1354,1357],{},[128,1352,1353],{},"MACaQ, 1 calibration block (2048 tokens)",[128,1355,1356],{},"14.8 s",[128,1358,493],{},[104,1360,1361,1364,1367],{},[128,1362,1363],{},"MACaQ, 128 flips, 31 blocks",[128,1365,1366],{},"88.1 s",[128,1368,504],{},[104,1370,1371,1374,1377],{},[128,1372,1373],{},"MACaQ, 512 flips, 977 blocks",[128,1375,1376],{},"636.9 s",[128,1378,515],{},[104,1380,1381,1384,1387],{},[128,1382,1383],{},"MR-GPTQ, 128 blocks",[128,1385,1386],{},"319.1 s",[128,1388,493],{},[104,1390,1391,1394,1397],{},[128,1392,1393],{},"DuQuant++ (GPTQ), 128 blocks",[128,1395,1396],{},"624.4 s",[128,1398,536],{},[15,1400,1401],{},"The MACaQ rows are points on one curve with different calibration budgets. The MR-GPTQ and DuQuant++ rows use 128 blocks each. Do not read down the time column and call it a race; those are different budgets. The paper's own fair comparison is 128 flips and about 200K tokens, roughly 150 s on Llama-3-8B, landing at 6.90.",[15,1403,1404],{},"This table is more useful than a headline ratio: you can pick your own row.",[15,1406,1407],{},"The abstract claims MACaQ \"already matches their best perplexity on Llama-3-8B in 11× less quantization time.\" We did not collapse that into a single ratio: Table 4 is a set of single points, the abstract is describing a curve. For a decision, read the table.",[15,1409,1410],{},"Figure 2: rounding picks the nearest point. A flip moves one code to the neighbour, and the layer's loss decides whether that happens.",[297,1412,301,1414,301,1418],{"className":1413},[300],[553,1415],{"src":1416,"alt":1417,"loading":557},"\u002Fimages\u002Fblog\u002Fen\u002Fmacaq-one-layer-at-a-time\u002Fflip-grid.svg","Two choices on an FP4 grid: round to nearest picks 1.0, a single flip moves to 1.5, and the layer loss decides",[310,1419,1420],{},"Figure 2: the only candidates are the neighbours, which makes a flip cheap; the expensive part is deciding, because that needs the layer's loss. Schematic, not measured values.",[15,1422,1423],{},"One thing to say plainly: this is not lossless 4-bit. Even the best setting goes from 6.14 to 6.85, about 12% worse perplexity. RTN is 8.29. Both are lossy. The difference is how much, and whether you will pay the time.",[35,1425,1427],{"id":1426},"fixing-the-gguf-you-already-have","Fixing the GGUF you already have",[15,1429,1430],{},"Everything above is \"quantize a model from scratch.\" Most people are not doing that. They have a folder of downloaded GGUFs.",[15,1432,1433],{},"MACaQ-R is for that case. Take an existing GGUF and its original FP16 file as a teacher, run one refine pass.",[15,1435,1436],{},"The weight codes get rewritten. The native format and block scales stay. So the result still runs on a stock llama.cpp build.",[15,1438,1439,1440,1442],{},"Table 5: ",[47,1441,582],{}," before and after MACaQ-R (README, single run)",[98,1444,1445,1458],{},[101,1446,1447],{},[104,1448,1449,1452,1455],{},[107,1450,1451],{},"Version",[107,1453,1454],{},"GGUF size",[107,1456,1457],{},"KL to FP16 ↓",[123,1459,1460,1468],{},[104,1461,1462,1464,1466],{},[128,1463,605],{},[128,1465,608],{},[128,1467,611],{},[104,1469,1470,1472,1474],{},[128,1471,616],{},[128,1473,608],{},[128,1475,621],{},[15,1477,1478],{},"KL divergence measures how far two distributions are apart; lower is better. Same size, 0.015731 down to 0.013367, a 15.0% drop. Same file size, same runtime, different code in each slot.",[15,1480,1481],{},"The command is straight from the README:",[629,1483,1484],{"className":631,"code":632,"language":633,"meta":634,"style":634},[47,1485,1486,1496,1508,1520],{"__ignoreMap":634},[638,1487,1488,1490,1492,1494],{"class":640,"line":641},[638,1489,645],{"class":644},[638,1491,649],{"class":648},[638,1493,652],{"class":648},[638,1495,656],{"class":655},[638,1497,1498,1500,1502,1504,1506],{"class":640,"line":659},[638,1499,662],{"class":648},[638,1501,665],{"class":648},[638,1503,668],{"class":648},[638,1505,671],{"class":648},[638,1507,656],{"class":655},[638,1509,1510,1512,1514,1516,1518],{"class":640,"line":676},[638,1511,679],{"class":648},[638,1513,682],{"class":648},[638,1515,685],{"class":648},[638,1517,688],{"class":648},[638,1519,656],{"class":655},[638,1521,1522,1524,1526,1528],{"class":640,"line":693},[638,1523,696],{"class":648},[638,1525,699],{"class":648},[638,1527,702],{"class":648},[638,1529,705],{"class":648},[15,1531,1532],{},"Then it runs, with nothing to swap in the backend:",[629,1534,1535],{"className":631,"code":711,"language":633,"meta":634,"style":634},[47,1536,1537],{"__ignoreMap":634},[638,1538,1539,1541,1543,1545],{"class":640,"line":641},[638,1540,718],{"class":644},[638,1542,721],{"class":648},[638,1544,724],{"class":648},[638,1546,727],{"class":648},[15,1548,1549,1550,1552],{},"llama.cpp binaries changed names across versions. Older builds use ",[47,1551,733],{},"; use whichever you installed.",[15,1554,1555],{},"Simple command, not-simple prerequisites. Three of them, from the README:",[739,1557,1558,1565,1572],{},[742,1559,1560,1562,1563],{},[317,1561,747],{},", and a separate environment per backend. Not just ",[47,1564,751],{},[742,1566,1567,1568,1571],{},"The ",[317,1569,1570],{},"original FP16 file"," as teacher. If all you have is the GGUF, you stop here",[742,1573,1574,1575,1578],{},"A ",[317,1576,1577],{},"calibration text file"," and the original weights directory",[15,1580,1581,1582,1585],{},"One more limit. Table 5 is a single comparison: one model, one baseline. ",[317,1583,1584],{},"Do not generalize it to \"any GGUF improves by 15%.\""," Yours may improve more, or not at all. Measure KL yourself.",[15,1587,1588,1589,1592],{},"How fast it runs afterwards is a separate topic, decided by bit width and kernel. We covered that in ",[22,1590,1591],{"href":774},"Quantization Speed Is Often a Kernel Problem",".",[35,1594,1596],{"id":1595},"what-this-does-not-prove","What this does not prove",[15,1598,1599,1600,1603],{},"The paper puts the cost in the first line of Limitations. A layer-local design means ",[317,1601,1602],{},"cross-layer error compensation is impossible."," A second pass over cross-layer statistics would likely bring back the slow quantization of the other methods. That is a design trade, not an oversight.",[15,1605,1606],{},"Also worth knowing:",[739,1608,1609,1615,1621,1627,1633,1639,1648],{},[742,1610,1611,1614],{},[317,1612,1613],{},"No third-party replication yet."," First posted Nov 7; the DuQuant++ and MR-GPTQ comparisons were run by the authors",[742,1616,1617,1620],{},[317,1618,1619],{},"W4A4 numbers are concentrated on a few models."," Do not read the +1.9% on Qwen3.8-27B as a general result",[742,1622,1623,1626],{},[317,1624,1625],{},"MoE: expert layers only."," Embeddings and lm-head are skipped in every model, not just MoE",[742,1628,1629,1632],{},[317,1630,1631],{},"The 122B budget differs from the rest."," 32 flips and 512K calibration tokens; the fair comparison used 128 flips and about 200K tokens",[742,1634,1635,1638],{},[317,1636,1637],{},"Qwen3.5 \u002F 3.6 \u002F 3.8 are different generations."," Do not cross-compare",[742,1640,1641,1644,1645,1647],{},[317,1642,1643],{},"Ascend is not plug and play."," Needs CANN and ",[47,1646,833],{},"; upstream has kernel PRs only, and full-model execution still needs a serving runtime",[742,1649,1650,1653],{},[317,1651,1652],{},"32.5 GB is device memory, not the whole footprint."," Your host still has to hold 244 GB",[15,1655,1656],{},"And the +2.3% again: it is measured against pure MXFP4. Against BF16 the same method is over 2× faster (Table 3). Same method, different denominator, opposite headline.",[35,1658,1660],{"id":1659},"three-things-to-do-tomorrow","Three things to do tomorrow",[848,1662,1663,1672,1678],{},[742,1664,1665,1668,1669,1592],{},[317,1666,1667],{},"Measure a baseline first."," If you own a GGUF, compute KL against its FP16 original. The README does not give you a KL script; you will have to find one. Without a baseline, \"15% lower\" is somebody else's number. Check the file is not broken first, too: ",[22,1670,1671],{"href":858},"Your quantized model might be broken",[742,1673,1674,1677],{},[317,1675,1676],{},"Run one refine pass."," If you have the original, run it, then run the same prompt on your usual llama.cpp build and compare outputs. Worst case you lose one refine.",[742,1679,1680,1683],{},[317,1681,1682],{},"Quantizing something 100B-class is a single-GPU job now."," Provided the host can hold the FP16 weights.",[15,1685,1686],{},"The gate moved. It used to be a card. Now it is a machine.",[884,1688,886],{},{"title":634,"searchDepth":659,"depth":659,"links":1690},[1691,1692,1693,1694,1695,1696,1697,1698],{"id":938,"depth":659,"text":939},{"id":965,"depth":659,"text":966},{"id":1083,"depth":659,"text":1084},{"id":1207,"depth":659,"text":1208},{"id":1280,"depth":659,"text":1281},{"id":1426,"depth":659,"text":1427},{"id":1595,"depth":659,"text":1596},{"id":1659,"depth":659,"text":1660},"\u002Fimages\u002Fblog\u002Fen\u002Fmacaq-one-layer-at-a-time.svg","Cover for the English essay, Quantize a 122B Model One Layer at a Time","MACaQ quantizes a 122B model in 64 minutes on 32.5 GB GPU memory, one layer at a time, rest in host RAM. At 32B the inference cost is 2.3%. MACaQ-R cut KL 15% on a 35B GGUF, given the FP16 original.","en",{},"\u002Fimages\u002Fblog\u002Fen\u002Fmacaq-one-layer-at-a-time.jpg","\u002Fblog\u002Fen\u002Fmacaq-one-layer-at-a-time",{"title":918,"description":1701},"blog\u002Fen\u002Fmacaq-one-layer-at-a-time",[911,912,913],"h1P4p-K3DukBf765f6-nLM5UhLte72EnDfcR2_c1VJ0",{"id":1711,"company":1712,"extension":1723,"meta":1724,"services":1725,"social":1803,"stem":1804,"__hash__":1805},"site\u002Fsite.yml",{"name":1713,"nameEn":1714,"email":1715,"address":1716,"addressEn":1717,"lineUrl":1718,"lineId":1719,"duns":1720,"youtube":1721,"youtubeId":1722},"銡策數位有限公司","Jecision Digital Co., Ltd.","contact@jecision.com","231 新北市新店區北新路二段 262 號 3 樓之 3","3F.-3, No. 262, Sec. 2, Beixin Rd., Xindian Dist., New Taipei City 231, Taiwan","https:\u002F\u002Fline.me\u002FR\u002Fti\u002Fp\u002F@374qnumi","@374qnumi","963905705","https:\u002F\u002Fwww.youtube.com\u002Fshorts\u002F6UE0WTmyjsg","6UE0WTmyjsg","yml",{},[1726,1750,1776],{"key":1727,"zh":1728,"en":1739},"audit",{"title":1729,"tagline":1730,"summary":1731,"points":1732,"deliverable":1736,"detail":1737,"duration":1738},"企業 AI 導入健檢","先看清楚，再花錢","60 分鐘線上訪談，盤點最耗時、最依賴人力的營運環節；我們會告訴你哪些工作現在就能交給 AI、哪些還不成熟。",[1733,1734,1735],"現況盤點：梳理日常營運中最耗時、最依賴人力的環節","痛點診斷：判斷哪些工作現在就能交給 AI，哪些建議先保留人力","可行性評估：依你的業務特性與預算，篩選適合的模型與工具","《企業 AI 導入建議報告書》（PDF）","含 1–2 個「開發成本最低、成效最明顯」的優先項目、技術棧建議、預期節省的人力時間與下一步","訪談 60 分鐘，報告書 3 個工作天內交付",{"title":1740,"tagline":1741,"summary":1742,"points":1743,"deliverable":1747,"detail":1748,"duration":1749},"AI Readiness Audit","Look before you spend","A 60-minute remote audit of the workflows that eat the most time and people-power — and an honest answer on what AI can take over today, and what it cannot.",[1744,1745,1746],"Audit the daily workflows that consume the most time and headcount","Triage which tasks AI can own today, and which should stay human","Shortlist the models and tools that fit your budget and constraints","AI Adoption Recommendation Report (PDF)","One or two highest-ROI entry points, a recommended tech stack, expected hours saved, and the concrete next step","60-minute call, report delivered within 3 business days",{"key":1751,"zh":1752,"en":1763},"prompt",{"title":1753,"tagline":1754,"summary":1755,"points":1756,"deliverable":1760,"detail":1761,"duration":1762},"提示詞工程","讓 AI 穩定產出，而不是碰運氣","針對你的實際任務設計提示詞與多步驟工作流，並做極限測試，確保換了資料、換了人操作也不會走鐘。",[1757,1758,1759],"客製提示詞：ChatGPT、Claude、Gemini、Midjourney 等主流工具的專屬模板","除錯與優化：修掉幻覺、格式跑掉、時好時壞的問題","工作流建置：把複雜任務拆成多步驟提示詞與可重複的 SOP","提示詞文件或設定檔","附使用說明與範例，並提供 1 對 1 教學，確保你與團隊能自己維護","依任務複雜度報價，通常 3–7 個工作天",{"title":1764,"tagline":1765,"summary":1766,"points":1767,"deliverable":1773,"detail":1774,"duration":1775},"Prompt Engineering","Make AI produce reliably, not by luck","Prompt sets and multi-step workflows designed for your actual tasks, stress-tested so they keep working when the data or the operator changes.",[1768,1769,1771],"Custom prompts for ChatGPT, Claude, Gemini, Midjourney and friends",{"Debugging and tuning":1770},"kill hallucinations, broken formats and flaky output",{"Workflow design":1772},"break complex work into repeatable multi-step SOPs","Prompt library or exportable config","Ships with usage notes and examples, plus a 1:1 walkthrough so your team can maintain it","Quoted by complexity, typically 3–7 business days",{"key":1777,"zh":1778,"en":1789},"poc",{"title":1779,"tagline":1780,"summary":1781,"points":1782,"deliverable":1786,"detail":1787,"duration":1788},"極速 PoC","五天，看到能跑的東西","不談架構藍圖，直接做一個能動的雛形。五個工作天內交付線上可操作的智能體，用你真實的資料驗證。",[1783,1784,1785],"Day 1：15 分鐘對焦，鎖定單一痛點與資料盤點","Day 2–4：模型串接、邏輯設計與雛形開發","Day 5：Demo 展示與 AI 擴展建議書","可線上操作的智能體雛形","交付範圍為概念驗證與展示，不含後續維運或大規模伺服器部署；要進階成完整系統會另行報價","5 個工作天",{"title":1790,"tagline":1791,"summary":1792,"points":1793,"deliverable":1800,"detail":1801,"duration":1802},"Rapid PoC","Five days to something that runs","No architecture decks — a working prototype built in five business days, validated against your real data.",[1794,1796,1798],{"Day 1":1795},"15-minute scoping call, one pain point, data inventory",{"Day 2–4":1797},"model integration, logic design, prototype build",{"Day 5":1799},"live demo plus an AI expansion recommendation","A runnable online agent prototype","Proof of concept and demo scope only; ongoing operations and large-scale deployment are quoted separately","5 business days",[],"site","XTwARfVAOcrXNiKehsM6V6cdkd2HXy4d5yZVrSWrI3Q",[1807,2266],{"id":1808,"title":1671,"author":7,"body":1809,"cover":2254,"coverAlt":2255,"date":2256,"description":2257,"draft":901,"extension":902,"featured":901,"lang":1702,"meta":2258,"navigation":307,"ogImage":2259,"path":2260,"seo":2261,"slug":2262,"stem":2263,"tags":2264,"translationKey":2262,"updated":914,"__hash__":2265},"blog\u002Fblog\u002Fen\u002Fbroken-on-arrival-quantized-models.md",{"type":9,"value":1810,"toc":2243},[1811,1820,1824,1831,1834,1837,1843,1847,1850,1857,1863,1874,1878,1881,1888,1898,1904,1908,1911,1969,1972,1978,1982,1985,1992,1998,2001,2005,2008,2113,2119,2130,2134,2147,2153,2167,2171,2189,2193,2198,2204,2211,2214],[12,1812,1813],{},[15,1814,1815,1816,1819],{},"Numbers come from arXiv 2609.05881v3 and the author's ",[47,1817,1818],{},"quantcheck"," audit data. Whether the bad files are still being served is our own check; the method is at the end.",[35,1821,1823],{"id":1822},"a-scene","A scene",[15,1825,1826,1827,1830],{},"You are on a laptop. You run ",[47,1828,1829],{},"ollama pull qwen2.5-coder:3b",", choosing a 2-bit tag because memory is tight.",[15,1832,1833],{},"The model starts. It answers, and it is fast. Then you ask for a sorting function and the code does not compile. You try another task; same thing.",[15,1835,1836],{},"Your first instinct is to blame yourself: the prompt? the temperature? maybe this model is just weak?",[15,1838,1839,1840],{},"A paper turned that doubt into a census: ",[317,1841,1842],{},"of 305 code-capable tags in the official Ollama library, seven solved at most six of 164 HumanEval+ tasks. Six solved none.",[35,1844,1846],{"id":1845},"the-counter-intuitive-part-the-files-are-fine","The counter-intuitive part: the files are fine",[15,1848,1849],{},"When a model misbehaves, you suspect a corrupt download.",[15,1851,1852,1853,1856],{},"The author rebuilt five of the seven from the original weights with llama.cpp. Comparing tensor by tensor, ",[317,1854,1855],{},"all five matched exactly"," — 434 tensors for the Qwen2.5-Coder-3B tags.",[15,1858,1859,1860,1592],{},"So these are not corrupt files. They are ",[317,1861,1862],{},"correctly reproduced broken artifacts",[15,1864,1865,1866,1869,1870,1873],{},"Checksums pass. File sizes match. ",[47,1867,1868],{},"ollama show"," looks normal. Because the file really is exactly what it claims to be. What is broken is the ",[317,1871,1872],{},"quantization recipe",": same weights, same toolchain, same tensor mix, but without calibration the model collapsed into a shape that cannot write code.",[35,1875,1877],{"id":1876},"why-this-happens","Why this happens",[15,1879,1880],{},"Quantization squeezes weights from 16 bits down to 2–4 bits. Every step has to decide where the rounding error lands.",[15,1882,1883,1884,1887],{},"The default approach spreads it evenly. The better approach uses ",[317,1885,1886],{},"calibration data"," — llama.cpp's importance matrix (imatrix) — to push error away from the weights that matter, based on activation statistics.",[15,1889,1890,1891,1894,1895,1592],{},"Without imatrix, small models at low bit are the ones that fall apart. All seven suspects sit in ",[317,1892,1893],{},"q2_K through q3_K_L, in models of at most 4B parameters",". That subgroup contains 48 tags, so the failure rate inside it is ",[317,1896,1897],{},"14.6%",[15,1899,1900,1901,1592],{},"But the recipe alone does not predict failure: the same uncalibrated recipe left Qwen2.5-Coder-1.5B functional (13–15 of 15). The problem is not \"low bit always breaks\". It is that ",[317,1902,1903],{},"when it breaks, you cannot tell",[35,1905,1907],{"id":1906},"calibration-rescues-most-of-them","Calibration rescues most of them",[15,1909,1910],{},"Holding weights, toolchain and tensor mix fixed, adding only an imatrix:",[98,1912,1913,1926],{},[101,1914,1915],{},[104,1916,1917,1920,1923],{},[107,1918,1919],{},"Tag",[107,1921,1922],{},"Before",[107,1924,1925],{},"After imatrix (of 164)",[123,1927,1928,1939,1949,1959],{},[104,1929,1930,1933,1936],{},[128,1931,1932],{},"Qwen2.5-Coder-3B q2_K",[128,1934,1935],{},"0",[128,1937,1938],{},"106–113",[104,1940,1941,1944,1946],{},[128,1942,1943],{},"Qwen2.5-Coder-3B q3_K_M",[128,1945,1935],{},[128,1947,1948],{},"76–89",[104,1950,1951,1954,1956],{},[128,1952,1953],{},"Qwen2.5-Coder-3B q3_K_L",[128,1955,1935],{},[128,1957,1958],{},"80–86",[104,1960,1961,1964,1966],{},[128,1962,1963],{},"Qwen2.5-Coder-3B q3_K_S",[128,1965,1935],{},[128,1967,1968],{},"13–30 (still weak)",[15,1970,1971],{},"(The paper's abstract says \"0 → 76–113\"; the per-level numbers above come from the v3.4 audit data, which we treat as authoritative.)",[15,1973,1974,1975],{},"Same model, same bit width, one calibration table: ",[317,1976,1977],{},"from zero tasks to three digits.",[35,1979,1981],{"id":1980},"another-failure-mode-a-missing-chat-template","Another failure mode: a missing chat template",[15,1983,1984],{},"Beyond the seven, the author also tested 22 popular Hugging Face community files. Two of them (TheBloke's deepseek-coder-6.7B Q3_K_M and Q4_K_M) scored 5 and 6 of 15 in the census.",[15,1986,1987,1988,1991],{},"The cause was not quantization but a ",[317,1989,1990],{},"missing chat template",". With no template, Ollama falls back to a default ChatML format that does not match the model's instruction format: 28 of 30 answers ran on into invented chat turns.",[15,1993,1994,1995,1592],{},"With the right template: ",[317,1996,1997],{},"98 and 101 of 164",[15,1999,2000],{},"So \"it runs\" and \"it answers properly\" are separated by a template.",[35,2002,2004],{"id":2003},"we-checked-the-bad-tags-are-still-being-served","We checked: the bad tags are still being served",[15,2006,2007],{},"A September incident is only interesting if it is still downloadable. We compared the blob digests in the audit data against the current Ollama registry manifests:",[98,2009,2010,2025],{},[101,2011,2012],{},[104,2013,2014,2016,2019,2022],{},[107,2015,1919],{},[107,2017,2018],{},"Audited digest",[107,2020,2021],{},"Today",[107,2023,2024],{},"Status",[123,2026,2027,2041,2053,2065,2077,2089,2101],{},[104,2028,2029,2032,2035,2038],{},[128,2030,2031],{},"qwen2.5-coder:3b-instruct-q2_K",[128,2033,2034],{},"14c4247b8ce4…",[128,2036,2037],{},"same",[128,2039,2040],{},"still served",[104,2042,2043,2046,2049,2051],{},[128,2044,2045],{},"qwen2.5-coder:3b-instruct-q3_K_S",[128,2047,2048],{},"81d167af7fa3…",[128,2050,2037],{},[128,2052,2040],{},[104,2054,2055,2058,2061,2063],{},[128,2056,2057],{},"qwen2.5-coder:3b-instruct-q3_K_M",[128,2059,2060],{},"9bb5d28b842a…",[128,2062,2037],{},[128,2064,2040],{},[104,2066,2067,2070,2073,2075],{},[128,2068,2069],{},"qwen2.5-coder:3b-instruct-q3_K_L",[128,2071,2072],{},"7a88dc11faae…",[128,2074,2037],{},[128,2076,2040],{},[104,2078,2079,2082,2085,2087],{},[128,2080,2081],{},"phi3.5:3.8b-mini-instruct-q2_K",[128,2083,2084],{},"54d47caa8bf3…",[128,2086,2037],{},[128,2088,2040],{},[104,2090,2091,2094,2097,2099],{},[128,2092,2093],{},"llama3.2:1b-instruct-q2_K",[128,2095,2096],{},"d1ede02a2111…",[128,2098,2037],{},[128,2100,2040],{},[104,2102,2103,2106,2109,2111],{},[128,2104,2105],{},"llama3.2:1b-instruct-q3_K_S",[128,2107,2108],{},"81881dcc12ec…",[128,2110,2037],{},[128,2112,2040],{},[15,2114,2115,2118],{},[317,2116,2117],{},"Seven out of seven are still there, with identical digests."," This is not history; it is something you can pull today.",[15,2120,2121,2122,2125,2126,2129],{},"(Method: fetch ",[47,2123,2124],{},"https:\u002F\u002Fregistry.ollama.ai\u002Fv2\u002Flibrary\u002F\u003Cmodel>\u002Fmanifests\u002F\u003Ctag>"," and compare the ",[47,2127,2128],{},"application\u002Fvnd.ollama.image.model"," layer digest. Script at the end.)",[35,2131,2133],{"id":2132},"three-things-you-can-do-now","Three things you can do now",[15,2135,2136,2139,2140,2142,2143,2146],{},[317,2137,2138],{},"Screen."," The author released ",[47,2141,1818],{},", a 15-task screen per tag. It costs about ",[317,2144,2145],{},"30 seconds of generation"," (on a 4090, excluding download). None of 66 sampled tags that passed the screen turned out to be non-functional, so it works as a pre-filter.",[15,2148,2149,2152],{},[317,2150,2151],{},"If you publish quantizations:"," add an imatrix, and run a functional test before publishing. Registries will not do it for you.",[15,2154,2155,2158,2159,2162,2163,2166],{},[317,2156,2157],{},"Check the chat template."," ",[47,2160,2161],{},"ollama show \u003Cmodel> --template","; if it prints ",[47,2164,2165],{},"{{ .Prompt }}",", there is no template and the model may be answering in the wrong format.",[35,2168,2170],{"id":2169},"do-not-over-read-this","Do not over-read this",[739,2172,2173,2176,2179,2186],{},[742,2174,2175],{},"2.3% is the share of official, code-capable, ≤8 GB tags — not of all quantized models",[742,2177,2178],{},"Only coding ability was tested (HumanEval+); perplexity is a pre-check, not proof of function",[742,2180,2181,2182,2185],{},"Seven is a ",[317,2183,2184],{},"lower bound",": all 12 deepseek-coder-1.3B tags, including fp16, score 1–6 of 15, so a group baseline cannot judge them and the author does not count them",[742,2187,2188],{},"The author is an independent researcher; data and tools are public, but this is one registry and one runtime version",[35,2190,2192],{"id":2191},"conclusion","Conclusion",[15,2194,2195],{},[317,2196,2197],{},"Exact reproducibility is not function.",[15,2199,2200,2201,2203],{},"Identical tensors, a passing checksum, a normal ",[47,2202,1868],{}," — all of that proves the file is the file. None of it proves the model can do the job.",[15,2205,2206,2207,2210],{},"So the question is not whether quantized models are usable. It is ",[317,2208,2209],{},"whether anyone tested whether this one is",". Right now the registries have not, which leaves it to you.",[2212,2213],"hr",{},[15,2215,2216],{},[2217,2218,2219,2220,2225,2226,2225,2230,2235,2236,2239,2240,1592],"em",{},"Sources: ",[22,2221,2224],{"href":2222,"rel":2223},"https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.05881",[26],"arXiv 2609.05881v3",", ",[22,2227,1818],{"href":2228,"rel":2229},"https:\u002F\u002Fgithub.com\u002Faditi-p31\u002Fquantcheck",[26],[22,2231,2234],{"href":2232,"rel":2233},"https:\u002F\u002Fgithub.com\u002Follama\u002Follama\u002Fissues\u002F18252",[26],"Ollama issue #18252",". Our digest check: ",[47,2237,2238],{},"scripts\u002Fcheck-ollama-tags.ts","; run ",[47,2241,2242],{},"npm run check:ollama -- --list",{"title":634,"searchDepth":659,"depth":659,"links":2244},[2245,2246,2247,2248,2249,2250,2251,2252,2253],{"id":1822,"depth":659,"text":1823},{"id":1845,"depth":659,"text":1846},{"id":1876,"depth":659,"text":1877},{"id":1906,"depth":659,"text":1907},{"id":1980,"depth":659,"text":1981},{"id":2003,"depth":659,"text":2004},{"id":2132,"depth":659,"text":2133},{"id":2169,"depth":659,"text":2170},{"id":2191,"depth":659,"text":2192},"\u002Fimages\u002Fblog\u002Fen\u002Fbroken-on-arrival-quantized-models.svg","A quantized model file that passes its checksum but fails every coding task","2026-11-02","Someone tested every code-capable tag in Ollama's library. Seven low-bit quantizations failed nearly all tasks, yet the files were exact rebuilds. Checksums pass; the recipe is broken.",{},"\u002Fimages\u002Fblog\u002Fen\u002Fbroken-on-arrival-quantized-models.jpg","\u002Fblog\u002Fen\u002Fbroken-on-arrival-quantized-models",{"title":1671,"description":2257},"broken-on-arrival-quantized-models","blog\u002Fen\u002Fbroken-on-arrival-quantized-models",[911,912,913],"tOVVVr1HmeG95b9UhxSxh_uy4dZQvyUwEripBKmKjww",{"id":2267,"title":1591,"author":7,"body":2268,"cover":2897,"coverAlt":2898,"date":2899,"description":2900,"draft":901,"extension":902,"featured":901,"lang":1702,"meta":2901,"navigation":307,"ogImage":2902,"path":2903,"seo":2904,"slug":2905,"stem":2906,"tags":2907,"translationKey":2905,"updated":914,"__hash__":2911},"blog\u002Fblog\u002Fen\u002Fquantization-kernel-dispatch.md",{"type":9,"value":2269,"toc":2887},[2270,2275,2277,2280,2283,2286,2289,2293,2296,2299,2302,2305,2308,2311,2314,2319,2322,2333,2337,2340,2343,2405,2408,2411,2414,2417,2423,2429,2435,2438,2534,2537,2540,2543,2547,2550,2553,2556,2626,2629,2632,2635,2638,2641,2644,2648,2651,2654,2702,2705,2708,2711,2714,2718,2721,2727,2733,2739,2745,2806,2808,2828,2832,2835,2838,2852,2855,2863,2871,2873],[12,2271,2272],{},[15,2273,2274],{},"Every number here comes from arXiv 2605.05699v1 (May 7, 2026). The author is Mohamed Amine Bergach at Illumina; this is a single-author preprint, not peer-reviewed. We did not rerun it on an Apple M1. Where I add a judgement, I say so.",[35,2276,1823],{"id":1822},[15,2278,2279],{},"You are running a 1B model on a laptop; push the context from 256 to 4K and decoding slows as the KV cache (the K\u002FV state kept during decoding) grows with every token. This essay is about two ledgers hidden in the usual advice — \"quantize, but it slows you down\" — and when to keep an fp16 fallback.",[15,2281,2282],{},"The implementation tax: the same algorithm written as four eager dispatches or two fused kernels. The deployment bill: an int4 cache against fp16, end to end, including where it breaks. By the end you should know which number is being compared to what, and four questions to ask of any benchmark that claims quantization is faster.",[15,2284,2285],{},"The paper runs the same int4 KV-cache method both ways. The eager version is 17% slower on a 360M model, measured against SRHT (a sign-randomized Hadamard rotation used as the control). Fuse the four steps into two kernels and SmolLM comes out 1% faster; on a 1.7B model it is 5% faster. Qwen is still 1% slower. End to end against fp16, Gemma-3 1B at a 256-token prefix has 8.2% lower latency.",[15,2287,2288],{},"Then the warning. On Qwen2.5-1.5B at a 2K prefix, the int4 cache emits two tokens and stops. fp16 emits 64.",[35,2290,2292],{"id":2291},"first-the-17","First, the 17%",[15,2294,2295],{},"The method is rotate-then-quantize. Each K and V vector in the KV cache is rotated, compressed to int4 (a 4-bit integer), then dequantized and rotated back when attention reads it.",[15,2297,2298],{},"The rotation here is SRFT (sign-randomized FFT): multiply by a fixed random ±1 pattern, then take an FFT. The point is to spread out outliers. In a K\u002FV vector, a few large values stretch the quantization scale and squash the rest toward zero. Rotate first and the coordinates look close to Gaussian; uniform quantization no longer has to survive a handful of spikes.",[15,2300,2301],{},"The problem is how the four steps are written. The eager version calls rotate, quantize, dequantize and inverse-rotate as four separate PyTorch\u002FMPS operations. Every layer, every decoding step, pays four kernel dispatches.",[15,2303,2304],{},"Dispatch is the queueing and data movement around a GPU kernel launch. The arithmetic may take microseconds; four launches and four round trips through memory can cost more than the arithmetic itself.",[15,2306,2307],{},"The paper's measured tax: 17% higher latency on SmolLM2-360M, 16% on 1.7B, 12% on Qwen2.5-1.5B. The baseline is SRHT (sign-randomized Hadamard transform), another rotation, not fp16.",[15,2309,2310],{},"Why SRHT? Because SRFT and SRHT produce quantization quality within noise of each other; the difference is whether the rotation is an FFT or a Hadamard transform. Using SRHT as the control isolates the cost of the implementation from the cost of the rotation.",[15,2312,2313],{},"The paper puts the mechanism in one line, which I paraphrase:",[12,2315,2316],{},[15,2317,2318],{},"The cost is dispatch, not compute. The underlying rfft and irfft calls take single-digit microseconds, but the four-step pipeline multiplies that overhead.",[15,2320,2321],{},"That is why the same algorithm turns from +17% into −1% — not because quantization improved, but because four launches became two (Figure 1).",[297,2323,301,2325,301,2330],{"className":2324},[300],[303,2326],{"src":2327,"poster":2328,"autoPlay":307,"loop":307,"muted":307,"playsInline":307,"ariaLabel":2329},"\u002Fimages\u002Fblog\u002Fen\u002Fquantization-kernel-dispatch\u002Fkernel-fusion.mp4","\u002Fimages\u002Fblog\u002Fen\u002Fquantization-kernel-dispatch\u002Fkernel-fusion-poster.jpg","Left: four eager kernel dispatches. Right: two fused kernels, forward and inverse.",[310,2331,2332],{},"Figure 1: The same quantization, eager with four dispatches per layer per step; fused into a forward and an inverse kernel.",[35,2334,2336],{"id":2335},"fusing-four-steps-into-two-kernels-is-not-enough","Fusing four steps into two kernels is not enough",[15,2338,2339],{},"The fusion puts the first two steps and the last two into one Metal kernel each: forward does rotate plus quantize; inverse does dequantize plus inverse-rotate. Data stays in fast on-chip memory instead of round-tripping to device memory.",[15,2341,2342],{},"The same comparison flips for SmolLM, but Qwen stays 1% slower:",[98,2344,2345,2361],{},[101,2346,2347],{},[104,2348,2349,2352,2355,2358],{},[107,2350,2351],{},"Model",[107,2353,2354],{},"SRHT baseline",[107,2356,2357],{},"Eager Δ",[107,2359,2360],{},"Fused SRFT (Δ)",[123,2362,2363,2377,2391],{},[104,2364,2365,2368,2371,2374],{},[128,2366,2367],{},"SmolLM2-360M",[128,2369,2370],{},"37.4 ms\u002Ftok",[128,2372,2373],{},"+17%",[128,2375,2376],{},"37.0 ms\u002Ftok (−1%)",[104,2378,2379,2382,2385,2388],{},[128,2380,2381],{},"SmolLM2-1.7B",[128,2383,2384],{},"222.2 ms\u002Ftok",[128,2386,2387],{},"+16%",[128,2389,2390],{},"211.9 ms\u002Ftok (−5%)",[104,2392,2393,2396,2399,2402],{},[128,2394,2395],{},"Qwen2.5-1.5B",[128,2397,2398],{},"84.9 ms\u002Ftok",[128,2400,2401],{},"+12%",[128,2403,2404],{},"85.7 ms\u002Ftok (+1%)",[15,2406,2407],{},"(Δ against SRHT; percentages rounded; 256-token prompt plus 128 generated tokens; Apple M1. Not against fp16.)",[15,2409,2410],{},"Why is the gap larger on 1.7B? The eager penalty is roughly the same (+17% vs +16%). The difference appears after fusion: 1.7B has a 2048-dim KV per layer, 360M only 320, so the fused kernel's fixed launch cost is spread across more work. That is why it moves from −1% to −5%.",[15,2412,2413],{},"At this point the story looks clean: four steps become two kernels, and 17% slower becomes 1% faster (on 360M). But if you stop here and conclude \"quantization is always faster\", you will miss the next section.",[15,2415,2416],{},"The end-to-end −8.2% against fp16 is not fusion alone. It needs three conditions.",[15,2418,2419,2422],{},[317,2420,2421],{},"First, the quantizer."," With a single per-token scale (one scale per token), Qwen's 4-bit ΔPPL explodes to +7975. The author switches to per-channel λ plus per-group abs-max (one scale per channel, one abs-max per 32 values) and pulls the same number down to +638.6. That recipe runs inside the fused kernel.",[15,2424,2425,2428],{},[317,2426,2427],{},"Second, a dequant-prefix cache."," The dequantized prefix is cached between steps instead of recomputed; the last 16 tokens stay in fp16 and the cache is invalidated when that window fills. Without it, Qwen at a 1024-token prefix is 2.4% slower than fp16; with it, it is 1.6% faster. The paper's text says −0.7%; its own ms\u002Ftok, 137.1 → 134.9, implies −1.6%. We use the table's numbers and note the discrepancy.",[15,2430,2431,2434],{},[317,2432,2433],{},"Third, the model architecture."," Gemma-3 1B has sliding-window layers: most layers see a fixed window and stay in fp16; only a few full-attention layers (which see the whole prefix) pay the dequant cost. That is why its advantage survives to 4096 tokens. Qwen has 28 full-attention layers, so the cost grows linearly with prefix length.",[15,2436,2437],{},"Put those three back into the end-to-end table:",[98,2439,2440,2455],{},[101,2441,2442],{},[104,2443,2444,2446,2449,2452],{},[107,2445,2351],{},[107,2447,2448],{},"Prefix",[107,2450,2451],{},"fp16 → int4 (ms\u002Ftok)",[107,2453,2454],{},"Latency Δ",[123,2456,2457,2471,2484,2497,2510,2522],{},[104,2458,2459,2462,2465,2468],{},[128,2460,2461],{},"Gemma-3 1B",[128,2463,2464],{},"256",[128,2466,2467],{},"71.2 → 65.4",[128,2469,2470],{},"−8.2%",[104,2472,2473,2475,2478,2481],{},[128,2474,2461],{},[128,2476,2477],{},"1024",[128,2479,2480],{},"83.3 → 82.5",[128,2482,2483],{},"−0.9%",[104,2485,2486,2488,2491,2494],{},[128,2487,2461],{},[128,2489,2490],{},"2048",[128,2492,2493],{},"129.5 → 121.3",[128,2495,2496],{},"−6.3%",[104,2498,2499,2501,2504,2507],{},[128,2500,2461],{},[128,2502,2503],{},"4096",[128,2505,2506],{},"188.5 → 182.4",[128,2508,2509],{},"−3.2%",[104,2511,2512,2514,2516,2519],{},[128,2513,2395],{},[128,2515,2464],{},[128,2517,2518],{},"90.7 → 88.3",[128,2520,2521],{},"−2.6%",[104,2523,2524,2526,2528,2531],{},[128,2525,2395],{},[128,2527,2477],{},[128,2529,2530],{},"137.1 → 134.9",[128,2532,2533],{},"−1.6%",[15,2535,2536],{},"(Δ against fp16; Apple M1; 256-token prompt plus 128 generated tokens. Qwen at 2048 is not in the table; see the next section. Qwen 1024's −1.6% is the arithmetic from 137.1 → 134.9; the paper's text says −0.7%. We use the table's numbers.)",[15,2538,2539],{},"On memory: Qwen is 3.0–3.1×; Gemma's 19.5× is not a like-for-like comparison (an apples-to-apples comparison inside the full-attention layers is about 3.2×), so do not compare it directly with Qwen.",[15,2541,2542],{},"The −8.2% is the best case, at a 256-token prefix. The same model at 4096 is −3.2%; Qwen at 1024 is −1.6%. \"Quantization is 8% faster\" is not a constant; it is a curve that moves with prefix length.",[35,2544,2546],{"id":2545},"before-you-celebrate-check-the-quality-bill","Before you celebrate, check the quality bill",[15,2548,2549],{},"Speed is one ledger. Quality is the other.",[15,2551,2552],{},"Quantization quality is usually measured as perplexity (PPL). ΔPPL is the change from the fp16 baseline; lower is better.",[15,2554,2555],{},"The 4-bit per-token ΔPPL on Cosmopedia (8192 held-out tokens):",[98,2557,2558,2576],{},[101,2559,2560],{},[104,2561,2562,2564,2567,2570,2573],{},[107,2563,2351],{},[107,2565,2566],{},"FP16 PPL",[107,2568,2569],{},"No rotation (identity)",[107,2571,2572],{},"SRHT",[107,2574,2575],{},"SRFT",[123,2577,2578,2595,2611],{},[104,2579,2580,2583,2586,2589,2592],{},[128,2581,2582],{},"SmolLM2-135M",[128,2584,2585],{},"6.61",[128,2587,2588],{},"+1.96 ± 0.15",[128,2590,2591],{},"+0.34 ± 0.01",[128,2593,2594],{},"+0.38 ± 0.06",[104,2596,2597,2599,2602,2605,2608],{},[128,2598,2367],{},[128,2600,2601],{},"4.87",[128,2603,2604],{},"+0.97 ± 0.08",[128,2606,2607],{},"+0.20 ± 0.03",[128,2609,2610],{},"+0.21 ± 0.02",[104,2612,2613,2615,2618,2621,2624],{},[128,2614,2381],{},[128,2616,2617],{},"3.77",[128,2619,2620],{},"+1.04",[128,2622,2623],{},"+0.16",[128,2625,2623],{},[15,2627,2628],{},"(The first two rows are the mean ± 1σ over three seeds; 1.7B ran one seed because of compute cost. 6-bit and 8-bit are lossless in this test.)",[15,2630,2631],{},"Rotation is not optional. At 135M, no rotation is +1.96; with rotation it is +0.38, roughly a 5× reduction. That is also why \"quantization breaks\" and \"quantization is just slower\" are often two sides of the same problem: if the rotation is wrong, the 4-bit error lands directly in PPL.",[15,2633,2634],{},"6-bit and 8-bit are lossless; 4-bit is the trade-off. 3-bit is worse in absolute terms; the paper does not sell 3-bit, and I would not start there.",[15,2636,2637],{},"Qwen's 4-bit is a different story. The Python reference's per-token ΔPPL is +7975.3; the Metal per-token version is +7727.9. With per-channel λ plus per-group g32, the fused version lands at +638.6 — a 12.5× improvement, in the same range as the Python reference's +654.6.",[15,2639,2640],{},"But +638.6 is still +638.6. A 12.5× improvement is not the same as deployable. The paper calls this \"closing the catastrophe\"; I think it is closer to moving a fire from the whole house into one room. If you enable 4-bit on Qwen, validate with task metrics, not just the relative PPL gain.",[15,2642,2643],{},"Gemma looks better: per-token +18.7, fused +3.6. The Python reference is −5.2; the +8.8 gap comes from the kernel's fp32 reduction and rounding order, not the algorithm. The author rules that out with two ablations.",[35,2645,2647],{"id":2646},"it-really-breaks-qwen-at-2k-emits-two-tokens","It really breaks: Qwen at 2K emits two tokens",[15,2649,2650],{},"Back to the warning.",[15,2652,2653],{},"Qwen2.5-1.5B at a 2048-token prefix, int4, emits two tokens and hits EOS (end-of-sequence); fp16 emits 64.",[98,2655,2656,2668],{},[101,2657,2658],{},[104,2659,2660,2662,2665],{},[107,2661,1114],{},[107,2663,2664],{},"fp16",[107,2666,2667],{},"int4",[123,2669,2670,2681,2692],{},[104,2671,2672,2675,2678],{},[128,2673,2674],{},"Generated tokens (2048 prefix)",[128,2676,2677],{},"64",[128,2679,2680],{},"2 (EOS)",[104,2682,2683,2686,2689],{},[128,2684,2685],{},"Peak memory",[128,2687,2688],{},"4447 MB",[128,2690,2691],{},"5527 MB",[104,2693,2694,2697,2699],{},[128,2695,2696],{},"Author's fallback",[128,2698,457],{},[128,2700,2701],{},"fp16 above about 1.5K",[15,2703,2704],{},"There is more than one cause. The dequant-prefix cache plus int4 storage pushes peak memory from 4447 to 5527 MB (+24%); combined with quantization noise across 28 full-attention layers, the logits tip into an early EOS.",[15,2706,2707],{},"I think the design that makes 1024 faster is also part of what makes 2048 break. The prefix cache turns dequantization from \"slower as the prefix grows\" into a fixed cost, but it also occupies memory; on a full-attention stack like Qwen, both happen at once.",[15,2709,2710],{},"Why is Gemma fine? Its sliding-window layers stay in fp16; only a few full-attention layers pay the dequant cost. Longer prefixes grow the cost more slowly. Same cache, different architecture, different curve.",[15,2712,2713],{},"If you want to enable this cache on your own model, test 2K first. Not \"does it generate\", but \"is the output still correct, and does it stop early\". The author's fallback is explicit: above about 1.5K, go back to fp16.",[35,2715,2717],{"id":2716},"four-questions-for-any-quantization-benchmark","Four questions for any quantization benchmark",[15,2719,2720],{},"The most useful part of the paper is not \"Apple Silicon is fast\". It is a way to split a benchmark into questions.",[15,2722,2723,2726],{},[317,2724,2725],{},"First, the platform."," Unified memory or separate HBM? The paper says the conventional trade-off returns on GPUs with separate HBM and high dispatch overhead. The same kernel does not necessarily stay faster when you move it. Ask where the number was measured.",[15,2728,2729,2732],{},[317,2730,2731],{},"Second, the kernel."," Fused, or four separate steps? If the benchmark does not say, the number may be dominated by dispatch tax. Ask: how many dispatches per layer per step?",[15,2734,2735,2738],{},[317,2736,2737],{},"Third, the prefix and the architecture."," The −8% at 256 tokens is not the −3.2% at 4K; full-attention and sliding-window stacks curve differently. Ask for 256, 1K and 2K, not just the best chart.",[15,2740,2741,2744],{},[317,2742,2743],{},"Fourth, the quality."," What is the ΔPPL baseline, and which quantizer? +638.6 is better than +7975, but it is still +638.6. Per-token and per-channel+group are different things; do not compare them as if they were the same.",[98,2746,2747,2760],{},[101,2748,2749],{},[104,2750,2751,2754,2757],{},[107,2752,2753],{},"Question",[107,2755,2756],{},"Why it matters",[107,2758,2759],{},"How to check",[123,2761,2762,2773,2784,2795],{},[104,2763,2764,2767,2770],{},[128,2765,2766],{},"Unified memory or separate HBM?",[128,2768,2769],{},"Where the trade-off flips",[128,2771,2772],{},"Hardware spec; ask for per-platform numbers",[104,2774,2775,2778,2781],{},[128,2776,2777],{},"Fused kernel or four dispatches?",[128,2779,2780],{},"Dispatch tax can be 12–17% vs SRHT",[128,2782,2783],{},"Does the benchmark say \"4 dispatches\" or \"fused\"?",[104,2785,2786,2789,2792],{},[128,2787,2788],{},"Prefix length and attention pattern?",[128,2790,2791],{},"Long context may break",[128,2793,2794],{},"Ask for 256 \u002F 1K \u002F 2K; check full-attention vs sliding-window",[104,2796,2797,2800,2803],{},[128,2798,2799],{},"ΔPPL baseline and quantizer?",[128,2801,2802],{},"+638.6 is better than +7975 but still +638.6",[128,2804,2805],{},"Check the baseline; per-token or per-channel+group",[35,2807,1596],{"id":1595},[739,2809,2810,2813,2816,2819,2822,2825],{},[742,2811,2812],{},"One platform: Apple M1 only; no discrete-HBM GPU.",[742,2814,2815],{},"Small models: 135M–1.7B; nothing above 7B.",[742,2817,2818],{},"A single-author preprint, not peer-reviewed; we did not reproduce it.",[742,2820,2821],{},"Gemma's 19.5× is not like-for-like; an apples-to-apples comparison inside the full-attention layers is about 3.2×.",[742,2823,2824],{},"Qwen at 2K breaks; the author's own fallback is 1.5K.",[742,2826,2827],{},"This is not the only path. It only addresses the cost of writing the four steps separately.",[35,2829,2831],{"id":2830},"conclusion-the-question-is-not-is-quantization-fast","Conclusion: the question is not \"is quantization fast?\"",[15,2833,2834],{},"Quantization is not a switch. It is a stack: platform, kernel, quantizer, prefix length, architecture. Change one and the answer changes.",[15,2836,2837],{},"The two ledgers:",[739,2839,2840,2846],{},[742,2841,2842,2845],{},[317,2843,2844],{},"Implementation tax."," The same method, eager four steps vs two fused kernels: 11–21 percentage points.",[742,2847,2848,2851],{},[317,2849,2850],{},"Deployment bill."," int4 cache vs fp16: up to −8.2% at a short prefix, single digits at long prefixes (Qwen 1024 is −1.6%, Gemma 4096 is −3.2%), and it can stop early.",[15,2853,2854],{},"One thing to do tomorrow: run your model at 256, 1K and 2K, record ms\u002Ftok and output quality. If it is a full-attention stack, keep an fp16 fallback above 1.5K; 2K is the test point.",[15,2856,2857,2858,2862],{},"If you want help running those three prefixes on your own model, our ",[22,2859,2861],{"href":2860},"\u002Fen\u002Fservices","AI readiness audit"," is exactly that: what can be handed to AI now, and what should wait.",[15,2864,2865,2866,2870],{},"The series so far: ",[22,2867,2869],{"href":2868},"\u002Fen\u002Fblog\u002Fbroken-on-arrival-quantized-models","broken-on-arrival"," on the recipe, and this essay on the implementation. The earlier memory-trade-off essay is available in Chinese. The question moves from \"should we quantize?\" to \"how is that kernel written?\"",[2212,2872],{},[15,2874,2875],{},[2217,2876,2877,2878,2883,2884,1592],{},"Paper: ",[22,2879,2882],{"href":2880,"rel":2881},"https:\u002F\u002Farxiv.org\u002Fabs\u002F2605.05699",[26],"arXiv 2605.05699v1"," (Mohamed Amine Bergach, Illumina). Previous essay: ",[22,2885,2886],{"href":2868},"the quantized model may be broken",{"title":634,"searchDepth":659,"depth":659,"links":2888},[2889,2890,2891,2892,2893,2894,2895,2896],{"id":1822,"depth":659,"text":1823},{"id":2291,"depth":659,"text":2292},{"id":2335,"depth":659,"text":2336},{"id":2545,"depth":659,"text":2546},{"id":2646,"depth":659,"text":2647},{"id":2716,"depth":659,"text":2717},{"id":1595,"depth":659,"text":1596},{"id":2830,"depth":659,"text":2831},"\u002Fimages\u002Fblog\u002Fen\u002Fquantization-kernel-dispatch.svg","Cover for the English essay, Quantization Speed Is Often a Kernel Problem","2026-11-05","Against SRHT, eager int4 KV-cache quantization ran 12–17% slower; fusing the four steps into Metal kernels cut most of the tax. The 8.2% win at 256 tokens has conditions—and Qwen breaks at 2K.",{},"\u002Fimages\u002Fblog\u002Fen\u002Fquantization-kernel-dispatch.jpg","\u002Fblog\u002Fen\u002Fquantization-kernel-dispatch",{"title":1591,"description":2900},"quantization-kernel-dispatch","blog\u002Fen\u002Fquantization-kernel-dispatch",[911,2908,2909,2910],"kv-cache","apple-silicon","inference","VrMGy2tQ__ko07oo7IbI-0WNfg5nBCIyvoC2Snzani0",1795088286153]