Design: cost-aware allocation across compact32 and wide64 verification #709

Design a cost-aware selected-prefix allocation for the new compact32/wide64 projection boundary. This extension is not implemented or timed. There is no selector speedup result. A draft design PR will accompany this issue; it will not change serving defaults or close the implementation/qualification work.

Motivation from retained experiments

The fixed four-row cap makes compact mixer execution possible but sacrifices late accepted proposals. The qualified mixed experiment used 25.3% more summed request cycles than main; the unqualified code-heavy experiment used 27.1% more. These are descriptive request-cycle counts, not GPU command counts or a causal latency decomposition.

Combined packing qualified on the mixed eight-request profile (+28.37% vs pinned main, +16.00% vs uncapped-eight, +12.42% vs FFN-only). Code-heavy transfer remains unqualified: retained thermal failure and +9.09% mean completion, despite descriptive +25.79% throughput. These results motivate a hypothesis, not a measured benefit for a new selector.

Reuse the selection already in Splash

runtime/metal/kernels/decode/sampling.metal::draft_verify_rows already ranks calibrated proposal-prefix survival probabilities globally and maximizes expected committed tokens over a supplied total-row cost table. VerifyBudget already fits acceptance and a nonnegative linear cost model per lane count/constrained state. Credit that existing algorithm; do not introduce a duplicate solver.

The new experiment changes the execution cost: a total of at most 32 selected rows can use compact projections, while larger totals need the original wide path. A single line in total rows does not represent this mode boundary. Dense Auto normally verifies all rows, and the eight-request experiment disables the old four-lane cost prior above four lanes; the new policy is not automatically active.

For lane i and r(i) verification rows, approximate expected committed tokens as 1 + sum(j < r(i)-1, survival(i,j)). Compare allocations using independently measured total mode cost, including selector/map/gather/scatter overhead and unchanged attention/GDN work. Retaining a longer prefix must pay for a wider tile. Keeping the original full-width path may be preferable when all lanes have high acceptance.

Proposed implementation boundary

1. Replace the mixer's fixed four-slots-per-lane layout with a global selected-row map. The FFN helper supports variable 1…8 rows/lane with total≤32; the mixer helper does not.
2. Add a GPU-selected original-wide fallback for every affected projection when total rows exceed 32. Preserve original attention/GDN/state storage and validate all aliases/extents/padding before dispatch.
3. Supply a mode-aware cost table from independent calibration. Preserve exploration, censor late acceptance correctly and avoid training on unobserved labels. A permanently capped policy cannot learn late survival reliably.
4. Establish model-wide plan support before changing retention. Dense-policy eligibility and state restoration need explicit plumbing and tests; no estimator or model-specific flag should become a serving default merely because this design exists.
5. Test actual selector choices, compact/fallback dispatches, full-vocabulary logits and canonical state on matched histories, unforced target maxima, Engine cache/cancel/suspend/resume and natural responses, in ordinary/shader modes.
6. Pre-register fresh mixed and high-acceptance/code-heavy cohorts against both uncapped-eight and frozen fixed-cap combined packing, with small-width controls, individual/aggregate latency, startup, memory, model/source pins, thermal/power observation, no exclusions and no tuning on final test data.

Keep models, draft weights and all quantization fixed. Every committed token must still be checked by the exact target. A mode estimate is an execution-policy hypothesis, not permission to approximate committed target selection.

Evidence and non-goals

Historical experiments, failures and source/model provenance include the original NEXT-UNTESTED-BOUNDARY.md; this proposal was recorded before reading the final body/code-heavy timings. The draft design will update references to current source interfaces and retain CONTRIBUTING's fixed-rule/feedback-first requirement.

This issue does not revive already-tested GDN prefix shortening, selector-only experiments, prompt lookup, quantization changes or the rejected matrix primitives. It does not claim implementation, quality qualification, tokens/sec gains or generic rollout readiness for the proposed selector.

Publication status

Draft PR #710. Related pairs: FFN #707 / #711, mixer #708 / #712, design #709 / #710. Review and merge the FFN foundation before the incremental mixer follow-up; the design PR is independent.

This PR contains a design document only. No cost-aware selector implementation or measured selector gain exists. It specifies fixed independently measured costs first, then an estimator only after paired A/B evidence.

Historical evidence and separate current-main validation assets. The self-contained PR tests use synthetic metadata. Historical component metadata banks and compiled binaries are excluded from the archive; recreating those probes requires the pinned model inputs. No serving default has been enabled.