LLM Inference Systems
Inference research on real interactive workloads: instrument, isolate the mechanism, check correctness, then make a production decision or fix upstream.
Solo project, including the upstream PRs.
Inference optimizations often look good under an isolated benchmark and fail under a real interactive workload. This program investigates those failures.
The scope is runtime, serving, reusable state, speculative execution, correctness, and heterogeneous compute, all of it on one Apple silicon machine.
How the work runs
- Production observation Behavior that shows up in real traffic and not in a benchmark.
- Runtime instrumentation Measure the mechanism, not only the wall clock.
- Controlled experiment Same prompt, one mechanism switched, everything else held.
- Mechanism The quantity that predicts the outcome, not the one everybody reports.
- Correctness Faster does not count unless the output survives it.
- Production / upstream A changed default, a pull request, or a stated refusal to change anything.
Every claim in the repository carries an evidence level: observed, measured, derived, inferred, hypothesized, not established. Three subjects are filed as open threads, not experiments, because each names the evidence it still lacks.
From baseline to optimized serving
This study did not start from a prefix-cache problem. The first goal was plain: take a 27B-class dense model at 4-bit on one Apple silicon machine from “it runs” to an inference service you can actually interact with.
At 16K context, dense prefill ran at about 302 tok/s; with sparse prefill, 1,046 tok/s. At 32K, 277 tok/s became 1,112 tok/s. Time to first token comes from the same two rows: 16K from 57.84 s to 19.24 s, 32K from 122.7 s to 33.5 s. Stacking the heterogeneous prefill path on top of that is a separate measurement, in throughput only: roughly 300 to 1,328 tok/s at 16K, from a single isolated smoke run with one resident engine, and the most fragile number on this page.
The optimization was not confined to model compute. Background dense prefix recovery contended with foreground generation on the same executor, and foreground decode fell to 13.5 tok/s. Two defects were behind that: a request was invisible to the scheduler until its admission ran, and a slice starved the loop that accepts requests. With both fixed, foreground decode under the same background work came back to about 47 tok/s, in the one run where it was measured. That work was an experimental branch; the rebuilt version is EXP-003 below.
Prefill throughput
tok/s · higher is better
- 16K 302 → 1,0463.5×
- 32K 277 → 1,1124.0×
- Stacked, one smoke run ~300 → 1,3284.4×
Time to first token
seconds · lower is better
- 16K 57.84 s → 19.24 s
- 32K 122.7 s → 33.5 s
Scheduler isolation
tok/s · foreground decode under background recovery
- Foreground decode 13.5 → 47
By this point both the single-request numbers and the background work staying out of the foreground’s way looked good. Then the same configuration was put behind a coding agent that reads files, calls tools, and grows its context turn after turn, and the session came out slower.
That is where EXP-001 actually begins: if every request is faster, why is the whole interactive workload slower?
Reusable State Dynamics (EXP-001)
The question: does a faster request make the whole interactive session slower, by destroying the reusable state the next request needed?
Sparse prefill (here SpecPrefill, an attention-based mechanism) delivers on a cold long prompt: 16K time-to-first-token fell from 57.84 s to 19.24 s, and 32K from 122.7 s to 33.5 s. But an agent’s next request is mostly its previous request again, and a sparsified suffix does not advance the normal reusable dense prefix state.
The request-level trace is what made the mechanism visible. The reusable checkpoint climbed to 37,888 tokens, collapsed back to 28,672, and stayed pinned there for the following ten requests, while the uncached suffix each request had to recompute grew from 17,060 to 33,979 tokens. The collapse happened before any sparse admission, so sparse prefill did not cause it; but every observed suffix after the collapse was sparsified, the checkpoint never recovered, and the recomputation accumulating from there is the prefix-cache debt.
The correctness defect found along the way mattered more than any of the performance work. The server derived the boundary of the protected prefix by subtraction, and with tools in play it fell as little as 37 tokens short, which placed the end of the tool instructions and the start of the operator’s system prompt inside the region the optimization was free to drop. The fix measures the boundary instead of inferring it, and was sent upstream.
The session is the unit of measurement, not the request
Why: A cold benchmark reports what one request saved for itself. It cannot see how much reusable state that request left for the next one, and that is the bill an agent workload actually pays.
Consequence: Track the reusable checkpoint and the uncached suffix, not the per-request saving. The slower agent session that started this is an observation, not a measured effect size: the two sessions took different trajectories.
The full findings, data, and limitations are in EXP-001; the long-form write-up is 當 prefill 變快,agent 反而變慢 (Traditional Chinese).
Speculative Decoding Cost Model (EXP-002)
The question: under what conditions does speculative decoding actually save time? Is break-even decided by the acceptance rate, or by the cost of one verify cycle?
By the verify cycle. Acceptance is the number usually reported for this mechanism, but across thirty-six runs high acceptance did not guarantee a speedup. What decides whether speculation pays is the cost of one verify cycle measured in dense decode steps, and that cost is a property of the model rather than of the content.
- On a 35B-A3B mixture-of-experts, a four-position verify forward costs 2.43 dense steps. A fixed draft depth of 3 came out 10% slower than dense decoding on code and 44% slower on prose, at acceptance rates of 56% and 25%.
- On a dense 27B, same runtime and the same prompt, the identical forward costs 1.37 dense steps, and at 79% acceptance the mechanism decodes 1.81× faster on a matched 13.6K-token coding prompt (1.05× end to end on the same row).
- A cost model built from the runtime’s own timers predicts the measured speedup to within 5% in ten of the eleven matched pairs, across the 0.56×–1.81× range.
- The runtime’s existing adaptive depth controller pulled every measured losing region back to the break-even line: it turned all four losses into parity or a small deficit, and in the one cell where the fixed depth already won, the controller won by a further 12% on top, by drafting shallower and buying a cheaper cycle.
Because the code under test was already choosing correctly, the finding came with no upstream proposal attached. The production decision is to change nothing and keep the current adaptive MTP.
Evaluate the controller that exists before proposing one that does not
Why: Proposing a new policy before establishing whether the current one misdecides trades an unproven problem for an untested solution.
Consequence: The measurement ended in a decision to change nothing.
One correctness caveat, flagged here and not opened up: with speculation on, greedy generation stopped being reproducible. Two runs of one prompt gave two different completions, both different from the dense one, while the dense arm was character-for-character identical. The semantic impact was not measured, so neither degradation nor its absence is claimed. The open question is filed in the correctness thread.
EXP-002 states each result with its evidence level, alongside the raw measurements for all five workload cells. The long-form write-up is 推測解碼何時真的會加速? (Traditional Chinese).
Progressive Canonical State Recovery (EXP-003)
The question: if a sparse prefill leaves no reusable state behind, can that state be rebuilt later without the foreground paying for it?
Yes, but the gain was not the one it was built for. Across a seven-turn session the sparse arm’s reusable canonical prefix never left zero while the prompt grew to 43,065 tokens, so every turn recomputed everything. In the recovery arm that prefix was rebuilt during foreground-idle windows and published only at cache-block boundaries the ordinary serving path could independently restore from. That took cumulative session latency from 228.38 s to 79.06 s on that workload. The foreground was pinned to SpecPrefill in both arms.
Making it safe to serve was the larger half. A share-of-time budget bounds how often background work collides with a request, not how long that request then waits. The worst collision stayed in the same 12–15 s band across a twentyfold budget change. What bounds the wait is the execution slice, and it is independent of the publication grain: every slice setting that ran reached identical boundaries, and shrinking the slice from the block grain to 512 took the worst client-observed wait from 15.08 s to 1.30 s for about 2.4% of recovery throughput at the extreme. Shrinking further to 256 did not clearly help: its trace-derived bound is lower, but its observed maximum is higher. One more defect had to be fixed before it could be proposed upstream. The recovery budget was owned per engine while the accelerator is shared, so each loaded model multiplied the cap.
The fastest configuration never returned to dense prefill; it left the next turn less to compute.
Why: The feature was built to return the foreground to dense prefill once the prefix recovered. The fastest configuration measured was the one where that never happened, and the turns that did switch routes were the most expensive turns of their sessions.
Consequence: Compared with sparse prefill alone, recovery left the next turn less to compute; switching routes didn’t help. Which route a turn should take is filed separately.
No foreground latency target was set before those runs. The worst uninterruptible execution slice in the runtime trace at 512-token slices, 2.39 s, is a derived bound on how long a request could wait. Nobody observed it.
EXP-003 carries the datasets, figures and limitations; the feature went upstream as
omlx#3964, which the maintainer closed unmerged. His reasons: recovery redoes skipped tokens as dense prefill, so on most turns it at best matches SpecPrefill off, and it added about 7,500 lines. At a route-switch threshold of 8,192 recovery went dense from turn 1 and behaved like SpecPrefill off. The one off run had its two largest prompts paused by the memory throttle, so it is no clean stand-in for the 79.06 s run. Long-form write-up:
償還 reusable state 的債 (Traditional Chinese).
Systems themes
Neither of the two below is an experiment: one is a configuration history, the other is still missing its judgment. The third open thread, on cross-runtime observations, is not written up here: no controlled comparison exists, and none was run. All three sit in the repository as supporting evidence, not as conclusions.
Correctness — fast but wrong is a regression
Every inference optimization changes something between the prompt and the answer (the arithmetic, the input, or the order of operations), so each one has to answer the same question: does the difference reach the output? Four cases give four different answers.
- Restoring a cached prefix. Seven prompts, fourteen runs, byte-identical output every time. The longest case went from 56.3 s to 2.3 s.
- Changing the attention route. At 68K context, three numerically different builds differed in their logits by up to about 0.4 and still produced the same argmax, the same top-3 token set, and the same output hash. The route itself was chosen per call from live memory headroom, which is why two identical processes could differ;
omlx#3685pins it to the bounded path. - The protected-prefix boundary. The output changed, because this is the only case that altered the model’s input; it is also the one that sounded most like bookkeeping.
- Speculative decoding. The output changed and stopped being reproducible.
Which of the four reaches the output is not guessable from how aggressive an optimization sounds. Reusing a cached prefix sounds risky and is exact. Speculative decoding sounds like the riskiest of the four, and here its output changed and stopped being reproducible; the cause was not isolated.
What this thread lacks is a judgment: once the output does change, every comparison so far can say whether the bytes differ, and none of them says whether the answer got worse.
Heterogeneous compute — accelerator enabled ≠ accelerator executed
The clearest result here is a null one. The neural-engine prefill path compiles for a fixed tile length, while the serving layer divides prefill into cache blocks. In the deployment where the block was 512 tokens and the compiled tile 2048, no delivered chunk could ever fill a tile: the path initialized, compiled, reported itself as enabled, and never executed a single tile.
None of that is visible in a throughput number. The configuration said “neural engine on”, the server agreed it was on, and the contribution was exactly zero. The accelerator will not take a prefill width below 1024 tokens at all, and omlx#3746 (merged) makes that geometry report itself as impossible instead of recommending a shape it cannot accept. What made the path usable was matching the compiled tile to the cache-block grain; the computation itself did not change.
In general, on a heterogeneous device, the unit of work an accelerator compiles for and the unit of work the serving layer hands out are two separate decisions.
Engineering and upstream consequence
None of the findings above was available to someone who only ran benchmarks. Getting to them meant building:
- A heterogeneous prefill path splitting each layer’s work between the GPU and the neural engine, on a 1024-token tile matched to the 1024-token cache block of the deployed configuration.
- Composition measurement of sparse prefill on top of it, stacking at 95–97% of the product of their individual speedups in an isolated qualification.
- A measured protected-prefix boundary, taken from two throwaway template probes and the token prefix that both of them agree on, replacing a derivation by subtraction.
- A background dense-prefix recovery job and a cooperative scheduler, designed to fail closed, with background slices yielding to inbound requests. Foreground decode went from 13.5 back to 47 tok/s in the one run measured. This was an experimental branch; a later review found four gaps in it and none of that code is in the served build. EXP-003 above is the rebuilt version, with those gaps closed and the serving safety measured rather than assumed.
- Request-level instrumentation of checkpoint position and uncached suffix, which is what made the trace possible at all.
- A transport-level request policy, added only after the real workload showed no single configuration was right for every request.
- A reproducible harness and dataset: every figure is redrawn by three scripts that read nothing but
data/, and nothing in it is smoothed, interpolated, or back-generated.
These pull requests went upstream to oMLX. An open pull request counts as a proposal, not a result:
omlx#3756: the correctness fix for the protected-prefix boundary. It was sent before the performance work because it is the only finding that changed the model’s input rather than only its speed.omlx#3762: per-request SpecPrefill fields on the Anthropic messages endpoint, matching what the OpenAI-compatible endpoint already had. It changes no upstream default.omlx#3685: merged. Deterministic SDPA-256 routing. The route was chosen per call from live memory headroom, so two otherwise identical processes could take different floating-point reductions; the change pins qualifying prefill to the bounded route.omlx#3840: merged. On a hybrid model the first layer can be a recurrent cache with no offset, so a restored draft cache was silently read as empty. The prompt was then prefilled on top of the state the cache already held, which corrupted the importance scores the selector relies on. The fix derives the position from an attention layer instead.omlx#3842: merged. Keeps reusable recurrent state at draft-cache block boundaries. It depended on #3840 and merged after it.omlx#3792: a SpecPrefill RoPE cleanup fix on the prefill-OOM requeue path, found while building EXP-003 and unrelated to the recovery feature. Upstream review showed no request reaches that path with the patch installed, so it was cut to a comment fix, which the maintainer made on main before closing it unmerged.omlx#3811: on mRoPE VLMs, SpecPrefill wrote its selected tokens at compacted rather than original positions. Validating progressive canonical state recovery (PCSR) is what exposed it; PCSR did not cause it, and it is present with background recovery switched off.omlx#3964: closed unmerged. The EXP-003 recovery feature, superseding #3793.omlx#3746: merged. The ANE prefill scheduler recommended a sequence length the accelerator cannot accept, since it requires a multiple of 64 and at least 1024 tokens; the merged change reports that geometry as impossible instead. It changes neither scheduling nor execution; it only makes a silent misconfiguration visible.omlx#3664: merged, and the one item here that did not come from an experiment. Tool groups in the Responses API were dropped before reaching the chat template, because only entries typed as functions survived the conversion, so a client’s namespaced tools never reached the model. It was a bug hit while running the stack.
Hardening the recovery feature (first #3793, then #3964) surfaced six defects that the experiment’s own workloads could not reach, under conditions such as a second model in the process, multi-token prediction on, an eviction mid-job, and a prompt whose length is an exact multiple of the cache block. Three of them apply beyond this runtime: background work must yield ownership and not only execution; foreground arrival must be visible process-wide and before execution begins; and a cache watermark is bookkeeping rather than the cache’s actual state, so it has to be able to move backward. The invariants are in EXP-003’s HARDENING.md.
After the experiment closed, running the mechanism under a real agent workload exposed two independent defects. Both are in SpecPrefill’s draft-cache path, not in background recovery, and their upstream fixes are omlx#3840 and omlx#3842. Neither changes EXP-003’s result or counts as a seventh or eighth hardening finding.
Evidence and limits
One machine, one vendor, one runtime. EXP-001 is a single 27B dense model at 4-bit, one run per cell. EXP-002 measures the 35B-A3B MoE and the 27B side by side, which is the closest thing here to a second configuration, though still on the same machine. EXP-003 is that 27B dense model again with multi-token prediction off, a single run per arm: enough to establish a mechanism, not enough to state an effect size. All three state that cross-model and cross-hardware generalization is not established: the mechanism arguments are about this runtime’s cache and scheduler; the numbers are about this machine.
github.com/tc3oliver/llm-inference-systems: the source of truth for methodology, raw data, figures, and limitations.- RESEARCH.md and EVIDENCE.md: the research map, and the ladder every claim is graded against.
- ENGINEERING.md: the system as it evolved, stage by stage, and what broke each stage’s assumption.
- Data and figures: every number behind every figure, with provenance and row counts.
- 當 prefill 變快,agent 反而變慢: the long-form EXP-001 article, with charts.
- 推測解碼何時真的會加速?從 Acceptance Rate 到 Verify-Cycle Cost: the long-form EXP-002 article.
- 償還 reusable state 的債: the long-form EXP-003 article, with charts.
What I still cannot do is classify a request as it arrives. Until that changes, the caller declares the shape and the server honors it.