A running model is not a finished job

Every number here comes from arXiv 2610.34241v1, a single-author engineering report. It is not a controlled experiment. Where I add a judgement, I say so.

A scene

"The model answered." That is the easiest sentence to fail an acceptance test with. The API returns 200, the prose reads well, and the tool call may never have left the building.

In July 2026, Vaibhav Jain pooled a few laptops to serve a model too large for any one machine. Chat worked; a coding agent did not. Every restart pushed a 32B checkpoint across Wi-Fi — 8.7 GB, roughly five minutes. He went single-node and noticed the real problem: "the model is running" is a very weak success criterion.

A checkpoint loads and answers. Then you ask it to inspect a file, call a tool, hold a long session, or preserve a repository contract. It may fail all of that, and your API still returns 200.

"Compatible" hides a lot. Behind one nominally compatible endpoint, tool-call formatting, turn control, context handling and cache reuse can all differ. Change one, and a workflow you verified on model A breaks on model B — usually without an error message.

arXiv 2610.34241 covers July to October 2026 across vLLM, SGLang, llama.cpp and MLX. Its move is simple: split "it runs" into four levels, then run 45 trials with hidden checks to see what level four costs.

Four levels, not one switch

1. Served — the checkpoint loads and returns something.
2. Coherent — the text reads like language and answers the question.
3. Parsed and executed tool calls — it says "read this file", the client receives it, and the read actually happens.
4. Verified task completion — the code passes an independent behavioural check.

The paper's framing sentence is the spine of everything below:

An endpoint can satisfy the first three and fail the fourth completely. And if it cannot satisfy the third, you cannot measure the fourth at all.

(My translation. The emphasis is the author's.)

These are not degrees of the same thing. They are four different gates. Level one passing says nothing about level two; level two passing says nothing about whether the model will pick up a tool. Most people stop at level two and think they are close, because the model sounds convincing.

The three tiers on an acceptance sheet map roughly like this: "it responds" is levels 1 and 2; "the code runs" is level 3; "the tests pass" is level 4. I keep 1 and 2 separate because their failures look similar and have different causes — one service never came up, the other came up and only talks.

And "the tests pass" depends on whose tests. If the vendor wrote them, you may still be at level three. Level four needs a check written by someone else and frozen before acceptance.

(Figure 1)

Same API, different protocol

He pointed Claude Code at a gateway he wrote, with different models behind it. The expectation was "swap an endpoint". The experience was closer to handing the work to a person with different habits.

Some models follow the tool contract. Some narrate what they intend to do instead of calling the tool. Some rewrite an interface they find awkward, including a function signature that was supposed to stay stable. None of this is stupidity; it is a different understanding of what a tool is.

The logs make the gap concrete. Of 492 Gemma 4 requests that carried tools, 294 produced parsed tool_calls. Another 144 ended with stop and no tool call; 54 had no saved parsed response.

Those 144 are not all failures. A normal finish is also a stop, and no tool call may simply mean the task was done. That is the first trap here: a missing tool call is not an error signal by itself. You have to know whether one was expected.

Some failures are blunter. Eight saved Gemma 4 outputs arrived with a malformed <|tool_call> prefix and never parsed. On GPT-OSS-120B over SGLang, tool_choice=required fought the model's own Harmony tool format and had to be downgraded to auto; tool rounds also needed a higher output allowance, or reasoning and tool JSON were truncated together.

Turn control is the other half: a missing session identifier, a tool loop, a thinking channel that does not line up. Each is enough to make a conversation look alive while it is lost.

The episode worth memorising: every API schema check passed, and the real Claude Code client still broke. In one case the working directory never reached the model-facing request, so the model guessed paths. In another, streamed tool-argument fragments were treated as replacements instead of appends, cutting the arguments mid-call.

Neither shows up in a unit test. Both require the actual client, end to end.

His diagnostic question is the useful part: when a tool fails, did the model fail, or did something in the middle drop the call? If it is the latter, fix the source — template, parser rules, serve flags, request translation — rather than patching the gateway forever. He reduced the gateway to a thin translator doing protocol and security checks, and made upstream failures surface as errors instead of a stream that ends normally with nothing in it.

"Configured" is not "served and evaluated". When a vendor says "supported", it often means "it can be configured".

The conversation looks continuous. It isn't.

To you it is one continuous thread. To the runtime it is three things: a token prefix, a KV cache, and an execution slot. Your "continuous" is not necessarily its continuous.

He kept 4,165 gateway turns and 623 debug jobs. Eighty-five cold-prefix jobs — no reuse at all, 30,000 to 36,000 input tokens — had a median time to first token of 69.4 seconds. Two hundred and eighty high-reuse jobs — at least 90% reused, under 2,000 fresh tokens — had a median of 2.8 seconds.

About 25x.

The author labels this observational: the two groups are composed differently, not randomised, not matched. You cannot write "caching makes it 25x faster". What you can say is that prefix reuse is associated with a difference worth engineering.

The mechanism is mundane. An inference backend can reuse KV state for an identical token prefix. Change the prefix — a timestamp, a system banner, tool definitions in a different order, a different JSON serialisation — and the whole context is prefilled again.

You changed a banner. You paid for the whole context.

Auxiliary requests are the pettier version. A coding agent also generates titles and permission prompts; with one reusable execution slot, an auxiliary request can evict the main conversation's resident state. In his records, 65 requests were classified as auxiliary and 44 showed no cache reuse. Among high-reuse records, auxiliary requests had a median TTFT of about 12.2 seconds (13 observations) against 2.7 seconds for main requests (267 observations).

Compaction has the same smell. He lowered the threshold deliberately: in 12 tests, sub-agent compaction fired six times, none of those six tasks completed, and one was still labelled PASS.

"Compaction happened" is not "work finished". In a log, the two look alike.

So he treats session continuity as an application responsibility, not a backend setting:

• Keep the message sequence append-only unless earlier content genuinely must change.
• Do not re-serialise tool messages for cosmetics.
• Make streamed and final tool JSON agree.
• Keep per-turn metadata out of the frozen system prefix.
• Preserve session affinity. A stable prompt is not enough if the next turn lands on a slot without the cache.

Caching is not a backend flag. It is application discipline.

(Figure 2)

Level four is where the work is judged

Deployment, protocol and sessions are now good enough to run bounded tasks. Can an open-weight model produce code that passes?

Five TypeScript task categories (quote, prorate, customer, split, parts), three conditions, three runs each — 45 sessions. The parent is fixed: Claude Code with Claude Opus 5.5. The worker is a quantized Qwen3-Coder 30B-A3B. Every session is judged by task-specific hidden behavioural checks, not by the agent's own claim of success.

The conditions: direct; optimised delegation, where the parent specifies, reviews and applies but does not implement; and batch-labelled delegation that may split work into sequential packages — though only a minority actually split.

Results: direct 15/15; optimised delegation 12/15; batch-labelled 14/15.

These do not rank. Each cell has three runs, the Wilson intervals overlap, and the author says the data cannot establish general success-rate differences. It does show workers produced usable code: of 30 delegated trials, 26 verified and all 26 retained worker-produced implementations. Not every time.

And the 14/15 is not evidence that batching helps. Only 5 of the 15 batch-labelled trials used a batch call; the rest used single delegation. The paper calls it an adaptive policy, not a batch-versus-single comparison.

The second suite is stricter: 15 tasks, three runs each, 45 sessions per arm, artifacts clearing a fixed interface contract before parent review.

Direct 45/45. Nemotron Cascade 2 worker: 12/45, of which 11 were genuine worker substitutions and one verified only after the parent rewrote the source. Qwen3-Coder worker: 0/45.

That second suite also has a disclosure: 16 sessions were stopped by a subscription limit before any model call and were replaced. Eight were in the direct arm and eight in the Nemotron arm. Counting the original 16 as failures gives 37/45 and 8/45. Those are scheduling interruptions, not inference failures — but if you cite the headline numbers, cite both.

The 45/45 deserves a caveat too. The tasks were built for this study, five categories with three runs each; there is no difficulty stratification.

One more honest detail: the task definitions and hidden tests were produced with AI assistance. The combined test file was created about 39 seconds before the first evaluated session and first committed after the last of the 45. What he can claim is narrower: task instructions and visible checks were identical across the 45 saved runs, and recorded score labels agree with the scorer as later saved. He cannot claim an independently snapshotted benchmark.

The first study's harness had limits: the parent's separate constraints field was not forwarded to the worker; the gateway dropped a duplicate system message and truncated messages over 1,800 characters; two of 41 reconstructed worker prompts lost about 220 characters, both in failed trials.

So "the worker ignored instructions" needs a discount. Some failures were capability; some were a worker that never received the full instruction.

What about 0/45? It does not mean the model can do nothing. That arm had no direct control, parent acceptance and application were not replayed, and the parent's prompt-cache state was not reset between studies. The author scopes it to these tasks, this contract, this worker configuration.

Verification is level four. It is not a by-product of level three.

The bill runs the other way

Frontier output tokens are expensive; a local worker is cheap. Delegate and save money — except delegation is not free. The parent writes the spec, reads what came back, decides whether to accept it, and cleans up failures. All of that is frontier tokens.

The cost model has five terms: parent context, specification, review, recovery, worker. The study measured the frontier terms. Electricity, hardware and engineering time were not measured — and unmeasured is not zero.

First suite, frontier cost per verified success: $0.136 direct; $0.214 optimised delegation; $0.199 batch-labelled. About 1.6x.

Second suite: $0.153 direct; $1.313 for Nemotron delegation. 8.6x.

Qwen's per-success cost is undefined, not zero — it had no successes.

One small case is worth keeping: a fee task where the worker's implementation passed and was applied, yet the frontier cost was $0.1856 versus $0.1293 direct, and wall time 126.4 seconds versus 27.4.

A successful substitution is not a saving.

This does not generalise to "delegation never pays". The author cites Scrouting: a small model scouts the repository, frontier models fix, and it solved 159 of 266 tasks against 158 for the strongest single model — one task apart — at roughly one fifth the cost. The direction is reversed: scout small, fix big. He limits his finding to his own contract, frontier-specifies / local-implements.

The cost definition is narrow too: API list prices, no hardware, electricity or engineering time. "8.6x" is the frontier side of the bill, not your total.

Tokens are cheap. Coordination is not.

The acceptance script will reject good code

The most practical part of the report is not about models. It is about acceptance.

Worker artifacts had to clear a static contract: imports, exports, and exported function parameter lists had to match. The stubs used underscored parameters for unused arguments (_entries); workers naturally dropped the underscore (entries).

For ordinary positional callers this changes nothing. But the contract compared parameter-name strings, so the artifact was rejected before parent review, before the hidden checks ran.

The post-hoc diagnostic: 36 valid-JSON artifacts rejected solely for dropping the underscore were applied unchanged to fresh task copies, with that rule bypassed, and re-run against the original sealed checks.

19 passed. 17 still failed.

So the rule was not simply wrong. It screened out both good and bad output. Rejecting bad output is its job; rejecting good output is its side effect — and if that gate is your only gate, the side effect is the final result.

He did not count the 19 as successes. They never went through parent review and application; the frozen scores stay Qwen 0/45 and Nemotron 12/45. He also did not conclude the contract should go. He proposed the next experiment: give the worker the specific validation error and allow one bounded repair.

That experiment has not been run.

On this contract, acceptance design failed more easily than model capability did.

What this cannot prove

• A single-author retrospective, not a controlled experiment. The workload changed as the engineering changed.
• Small sample: five task categories, three runs per cell, one fixed parent, one worker configuration.
• Cost covers frontier tokens at list prices only.
• The raw records are private. You cannot recompute the numbers.

The studies are not pooled and do not rank models or runtimes.

So this gives you an acceptance framework, not a leaderboard. The most valuable part, in my reading, is not the numbers — it is that "done" becomes something you can check level by level.

A checklist you can use tomorrow

The author's five rules, rewritten for a PoC document:

1. Define success at the task boundary. Accept on "the output passed a check it does not own", not on "it responded". Exercise the real client, not a harness you wrote.
2. Treat deployment as a recipe, not a model name. Checkpoint revision, quantisation, runtime, template, context settings, gateway, client, resident memory. Change one, re-verify.
3. Make session continuity an application responsibility. Stable prefix, no cosmetic re-serialisation, auxiliary requests must not evict the main conversation.
4. Price delegation per verified success. Not per token, not per call.
5. Verify by behaviour where you can. Static gates are cheap and worth keeping, but know what they reject.

If you are the buyer, three questions are enough:

• Which level does this demo stop at? "It answers" is level two.
• How do you verify level four, and who wrote the checks, and when were they frozen?
• Is the cost per call or per verified success?

Our own video transcription pipeline ran into the same class of problem: the model ran, the pipeline still stalled, and every failure looked like "the model is a bit dumb". We have not run measurements at this scale, so this essay borrows the paper's numbers.

The previous essay was about files that look right and cannot be used; this one is about models that look right and cannot finish the job.

If you want help filling in the table, our AI readiness audit does exactly that: what can be handed to AI now, and what should wait.

Conclusion: which level counts as done

The first three levels can be demonstrated in an afternoon. Load a model, send a few prompts, connect a client.

Level four takes weeks: design tasks, write hidden checks, run enough trials, classify failures, price the work. What you get is rarely "this model is good" — it is "on these tasks, under this contract, it is good".

The missing column on your acceptance sheet is where the price difference lives.

Tomorrow: write "passes the tests" into your PoC definition, and ask who wrote the tests and when they froze them.