Experiment: reduce GPU queue overhead with coarser tasks and static fusion #760

Article coverage audit and weighted execution experiment — 13 November 2026

Coverage is not the same as a complete architecture port. The accumulated experiments exercise the transferable local ideas: fusion, SIMD roles, shared/cache prefetch, old-KV and predicted-expert prefetch, hot-first ordering, loop size, instruction-warming analogues, synchronization/sentinels, cycle submission, graph overhead, speculative budgets/tree verification, profiling and B1–B4 batching. Non-temporal loads and native asynchronous-copy capability probes were also attempted. Results include rejected and inconclusive cases; “tested” does not imply improvement.

Two architectures remain partially implemented, not complete: a single launch for an entire attention/GDN-plus-MLP layer, and a general full-model GPU VM. CUDA TP8/NVLink, TMA, clusters/PDL and direct code-address/cache-priority instructions have no exact implementation on the one installed Apple GPU. Local analogues must not be described as those exact CUDA operations. Reference: Muse Spark Open Ultrafast.

The new weighted block experiment is published in draft #762 at c6f9ec0e. Its full matrix, native and same-helper controls, bitwise/shader validation and reproduction commands are there. At 160 workers it is 73.8–117.4% slower than native; the register-input workaround is also slower. No production integration, model TTFT/tok/s gain or full-architecture completion is claimed. The complete layer and full-model VM remain open architecture work.

Production relevance update — 13 November 2026

Draft #762 now includes a further twelve-variant screen of production-shaped projection/expert work, alongside real 27B/35B profiles. No merge-ready production gain was found.

The synthetic three-dispatch fusion result remains valid at its stated scope. It does not identify an equivalent unfused section in the real model, which already fuses important projection/normalization work. Real B1 GPU cycles take 66.20 ms (27B) / 19.01 ms (35B); the dominant attributed gate/up paths account for 38.7% / 31.9%. Full methods and attribution limits are in #761/#762.

Eight dense variants cover tile grouping, prefetch distance, code/loop shape and staging. Four expert variants cover paired work, specialization and streamed dequantization. There are 56 normal comparisons: 18 non-regression passes and 38 failures; pass is not evidence of a speedup. All nineteen separate shader-validation processes pass for the tested AF4G64 cases. Even reducing the column kernel's static shared-memory reservation 8,192 → 128 bytes through specialization leaves the full MoE layer 2.67–11.89% slower. Numerical checks compare expert intermediates in route order, since grouping atomics can reorder physical rows.

Detailed results and limits are in #762; local source/raw artifacts remain under ignored build/muse-experiments/merge-candidate-20261113/, not a hosted artifact. No new model TTFT/tok/s benefit is claimed. The current CI run passes production/CPU/native/app, Python 3.12/3.13/3.14 and sanitizers; the self-hosted release gate was skipped. CI success does not supply the missing performance evidence.

The Muse article remains the reference. The PR stays draft and production runtime remains unchanged.

Result

Draft PR #762 tracks this investigation. Keep it in draft. This revision investigates why the original fine-grained GPU queue loses, improves that prototype through coarsening, and tests static fusion as an alternative for the known DAG.

Reference: Muse Spark Open Ultrafast: Pushing beyond 1,000 Tokens per second, dynamic ready-task / GPU-VM direction. This remains a synthetic component experiment, not a complete GPU VM or a model inference speedup.

• Coarsening 256 tiny tasks into 96 macro tasks cuts queue GPU time by roughly half, including confirmation on new input sizes. It still loses to static dispatch.
• A fixed three-dispatch static implementation gives 43.19% lower GPU time geometrically on the held-out confirmation; a repeat from the committed head gives 46.16% lower GPU time. All three aggregate comparisons pass in each run. These are repeated measurements of the same held-out set, not two independent holdouts.
• Host wall time is noisier and not consistently qualified across sizes. Model TTFT, model tokens/sec and energy remain unmeasured; no production serving improvement is claimed.

Root causes and controlled experiments

Per-task coordination dominates this small, regular DAG. At 20 workers, removing all arithmetic leaves the queue at about 82 µs, close to its real workload time. Queue pops, dependency updates, fences and completion publication remain. The no-arithmetic variant checks scheduling completion only and is explicitly marked non-equivalent work. It is a diagnostic, not an optimization or an additive decomposition of runtime; contention changes when arithmetic is removed.

Larger tasks amortize bookkeeping. Three macro stages cover affine/SiLU/RMS, affine/SiLU/RMS, then affine/SiLU. Each row recomputes the neighboring affine value needed by SiLU. The coarse queue therefore schedules 96 tasks, retaining every original FP32 intermediate. It is compared both with the old queue at the same worker count and with three static dispatches.

Noisy cases are retained, not counted as passes. The coarse queue still fails most comparisons against even the coherent three-dispatch static control; it is not selected as the fastest executor.

The topology is fixed, so static fusion removes unnecessary runtime coordination. Eight per-stage dispatches become three macro dispatches, with 32 independent row groups. Ordinary-buffer eight- and three-dispatch controls also remove unneeded device coherence within a dispatch; command boundaries already synchronize their buffers. This isolates fusion from the coherence choice. Static kernels always use 32 groups; the worker field applies to persistent queue/scan variants.

Full fusion through halo recomputation has a crossover. A one-dispatch version gives each group a row and recomputes its bounded neighbor halo. It passes exactness, wins at the smallest initial size, but loses at larger sizes versus three-dispatch static fusion. It remains an experimental alternative, with no per-size selector or production default.

Benchmark and numerical fixes

The initial naive macro kernel failed exact comparisons by one ULP in stage 0 (for example 0.0745234415 versus 0.074523434). Compiler contraction had changed the affine evaluation. Explicit fma(column, bias, scaled_input) restores the original GPU stage’s operation order. No error bound was relaxed. Diagnostics are retained locally in scheduler-fma-diagnostic.log.

Large-vector timings also showed a repeatable observer effect from full mapped CPU memcmp checks between GPU runs. For 5120 columns, the selected implementation’s middle ABBA phases were approximately 19.48 / 22.98 µs with readback, versus 19.29 / 19.38 µs without it. The fixed ABBA spread drops 16.49% → 0.43%. The readback-on run is retained and remains inconclusive.

Timing now checks full outputs before warmup and after the timed sequence, using an immutable GPU reference already checked against the independent CPU oracle. Dynamic completion and small guards remain checked on every run. Separate shader validation repeats each arm eight times and checks every execution exactly. --readback-each-run reproduces the observer effect. Numerical checks and the existing performance thresholds are unchanged.

Fixed implementation confirmation

One implementation is selected for every case: three static macro dispatches with ordinary buffers, compared with the same ordinary-buffer eight-dispatch control. No per-shape choice is made. Initial columns are 256/2048/5120, seed 0; held-out columns are 512/1024/4096, seed 20261113.

Four warmup ABBA blocks precede eight recorded blocks in the initial sweep and sixteen in confirmation. The unchanged rule declares spread above 5% inconclusive; otherwise the regression allowance is max(2%, twice spread). Pass means non-regression, not a demonstrated gain in every individual block.

Committed source: 44347115db6a95ec3682345638dbe6d825cce995; working tree was clean for that run.

GPU clocks are not fixed. Absolute microseconds vary across process warm-up and submission gaps; interpret the paired within-run comparisons. All old readback-affected, failed and inconclusive runs are retained separately.

Metrics have separate scopes: GPU time excludes CPU queue initialization; host wall time includes initialization, command preparation/submission and waiting, excluding validation and one-time graph construction. Submission elapsed time can overlap GPU work and must not be added to GPU duration. These are standalone DAG metrics, not model-serving measurements.

Correctness, limits and reproduction

The initial and held-out shader sweeps each contain 81 comparison cases: 69 equivalent-work cases and 12 coordination-only diagnostics. Each arm is repeated eight times with exact intermediate checks. Every equivalent-work variant matches all eight stages bit-for-bit against the GPU reference, which passes the independent CPU oracle at the unchanged 2e-5 + 2e-5 * abs(reference) bound. Output/scratch/state guards, task completion and error counters pass. Coordination-only cases validate scheduling, not output values.

Local -O3 -Wall -Wextra -Werror builds, architecture validation, Python lint/format and patch whitespace pass. The engine remains at the unchanged 30,105-line budget. Changes are confined to experimental benchmarks. This does not establish benefits for arbitrary/ragged DAGs, model weights, HTTP serving or power consumption, and it does not reject all GPU-VM designs. Production runtime defaults are unchanged.

M5 Pro (20 GPU cores, 48 GB), macOS 26.6, Metal 4.0 / clang 21, deployment target 26.4. Baseline production source: d8af1054aab0f97de21d043e01aa98af3754f10e.

python3 dev/benchmarks/experiments/gpu_scheduler/run.py --output build/experiments/scheduler/full
python3 dev/benchmarks/experiments/gpu_scheduler/run.py --shader-validation --holdout --seed 20261113 --output build/experiments/scheduler/heldout-shader
python3 dev/benchmarks/experiments/gpu_scheduler/run.py --holdout --seed 20261113 --blocks 16 --output build/experiments/scheduler/heldout-full
python3 dev/benchmarks/experiments/gpu_scheduler/run.py --confirm --blocks 16 --readback-each-run --output build/experiments/scheduler/readback-control
python3 dev/benchmarks/experiments/gpu_scheduler/run.py --confirm --holdout --seed 20261113 --blocks 16 --output build/experiments/scheduler/heldout-confirm

Raw JSON, stderr, parameters, source/binary hashes, host timing and every ABBA block stay under ignored build directories. The experiment README defines all controls and flags. Companion fusion investigation is draft #761 / issue #759; tree research remains draft #727 / issue #728.

GPU-only initial raw JSON SHA-256: 6c014c2cb706ce07280ef7e4aefd6c5aafec9bc05e460f1da83318f2fd7c422c.

Held-out full raw JSON SHA-256: 1d93956e8bff6e2c39181b5b7e5916e813fcd2c2b80ec0da2469702e3eea932f.

Committed confirmation raw JSON SHA-256: 26553046f062b867097521e963fc17a62dc21f4ae9f289c4458c55b7da54acad.

Current hosted CI

Actions on the revised head passed Metal-toolchain installation and is running production/CPU, Python compatibility and sanitizer checks at the latest audit. Local validation above passed; hosted checks are not yet all complete.