Research: compact mixer verification projections after FFN packing #708

Extend selected-prefix packing from the FFN to the target's GDN/attention input and output projections. This is the strongest archived native candidate: 199.50 tokens/sec, qualified +28.37% versus the pinned four-request main reference, +16.00% versus uncapped eight-request decoding, and +12.42% versus the FFN-only reference.

This is a default-off experiment, with a dependency on the eight-request/FFN foundation. It remains a draft review proposal: its fixed-model guard/build flags need generalization to satisfy CONTRIBUTING, and historical timings do not qualify a current-main integration.

Code mechanism

For a row selector S and row-wise projection, S(XWᵀ) = (SX)Wᵀ in real arithmetic. Gather up to four committable prefix rows/request into 32 physical slots, run the existing Linear kernels with the original K partitions/walk, then scatter into the original eight-slot layout. Floating-point identity is separately tested; the equation alone is not its proof.

• Private addVerifyBodyPack validates geometry, AF4/mixed BF16/F32 segments, epilogue, storage plan and every buffer/scratch extent before dispatch. Unsupported plans retain the original projection path.
• QwenTarget applies it to 128 mixer projections across 64 layers; attention, GDN recurrence, norms, head, draft, KV layout and state addressing remain on their existing paths.
• The GDN packed input has stride 16640 but only 16480 valid columns. Preserve its 160 padding columns. Three compact BF16 buffers add 3,194,880 bytes (3.047 MiB), serially reused through DecodeArena; original wide scratch remains allocated.
• Both cap points must agree before encoding. A generic version must validate model-wide format support before applying the cap, so unsupported projections do not pay its acceptance cost without the compute benefit.

Historical native concurrent-decode experiment

Baseline 031a25a6125af2d7acf7096580afe56eb33d65bc; M5 Pro/20 GPU cores/48 GiB/macOS 26.6; fixed Qwen3.8-27B + DFlash2; unchanged native quantization/KV/GDN formats; GPU-only, unconstrained greedy nonthinking, warm model/cold requests. Eight mixed prompts (73–86 input tokens), 512 emitted tokens/request, plus groups four/one. This is the native Engine harness, not HTTP; throughput includes wall time through the final completion, with model setup separately reported.

Twenty-four fresh processes: ABCDDCBA / BCDAADCB / CDABBADC. Qualification uses the geometric mean of three balanced block ratios, two samples/arm/block; pooled TPS is descriptive. D/A≥1.15, D/B and D/C≥1.10, all blocks positive, gain>2× reference drift, every arm/group endpoint drift≤5%, small-width controls≥0.95. All 72 intervals passed fixed budgets/Length/cold/width/AC/nominal-temperature/readable-memory/no-swap gates; maximum endpoint drift was 1.03%. No exclusions, replacements or pooled interrupted data.

Costs and failed transfer

• Versus main: mean completion +0.9%, final completion −22.1%. The aggregate completion tolerance passed, but four individual prompt cases completed 34–64% later. Every case completed faster than both eight-request references; this is not a universal latency win.
• Sampled peak footprint: 20.34 → 21.97 GiB; mean model setup 12.88 → 17.94 s. One-/four-request throughput stayed within 0.5% of main.
• Independent code-heavy 12-process cohort: 156.24 → 196.52 tokens/sec, descriptive +25.79%, unqualified. Baseline interval 04A ended at thermal state Fair; mean completion +9.09% also exceeded the 5% tolerance. All samples remain included. No validated code-heavy transfer claim.
• Both earlier AC-interrupted combined cohorts remain incomplete with no aggregate. Earlier short component timing remains unqualified; the separately registered long-window mixer primitive measured 1.72401×, not serving TPS.

Correctness evidence and review

The historical combined stack passed three CPU suites, eleven native shader groups, 34 matched-history full-model processes comparing selected full-vocabulary F32 and canonical committed state, 192 natural responses/30,720 function checks, raw target-maximum certificates and Engine lifecycle/preflight checks. Fresh actual-Engine certificates cover 12,304 emitted tokens and 1,613 selected choices after resume; code-heavy certificates cover another 16,384 committed tokens.

Unforced shader suspend/resume traces retain two wording forks. Each build can produce either alternative across invocations; every observed emission has its own raw target-maximum witness. This does not establish universal literal generation identity or the cause of the forks. All differences remain available for review.

Evidence, raw data, checksums, probes and reproduction guide · historical combined source reconstruction. The reconstructed source contents match 616 frozen hashes; binary identities remain the original 031a dirty builds. Reports stay outside code commits.

The follow-up PR will show its incremental diff over the FFN foundation and target main per CONTRIBUTING. Require exact-head source/build/numeric/state tests and a new paired qualification before treating rebased code as measured. Keep default-off; no generic model/device, single-chat, HTTP, sampled/thinking or long-context rollout is proposed.

Publication status

Draft PR #712. Related pairs: FFN #707 / #711, mixer #708 / #712, design #709 / #710. Review and merge the FFN foundation before the incremental mixer follow-up; the design PR is independent.

The current combined head builds and its model-free production mixer helper passes 15,840 comparisons, 252 aliases and 12 host guards in both ordinary and shader-validation modes. This does not qualify current-head inference performance, and the enabled-width CPU integration blockers of #711 remain.

Historical evidence and separate current-main validation assets. The self-contained PR tests use synthetic metadata. Historical component metadata banks and compiled binaries are excluded from the archive; recreating those probes requires the pinned model inputs. No serving default has been enabled.