Feature request: GLM-5.3-Flash family (glm5-next) with SSD expert streaming #740

Activity

1. dmkjr commented on Nov 12, 2026

   

   dmkjr

   on Nov 12, 2026

   Contributor

   More actions

   Thanks for the careful header dump — it makes the question easy to answer precisely.

   Short answer: the expert streaming is not the hard part, the attention is.

   Splash's expert store (the machinery behind Qwen3.8-Flash-Next) already does exactly what you describe for GLM's MoE side: routed experts paged from disk, a resident pool sized to the machine, top-8 routing, one shared expert, dense leading blocks. If GLM-5.3-Flash were "Qwen layers with GLM chat tags," the port would be a layout change and your 152-of-288 pruning would slot right in — the engine reads expert_count and expert_used_count from the GGUF header, not from any assumption about 288. Pruned checkpoints would be fine.

   What is missing, and what a GLM family would actually have to build:

   • KDA (Kimi Delta Attention). 34 of the 45 layers are linear-attention with a fixed-size recurrent state, no KV cache. Splash's entire state layout — cache blocks, spill, replay — assumes a conventional cache. A KDA state is a different object with different lifetime rules.
   • DSA (KPool sparse attention). The other 11 layers keep a pooled index of every token and attend to a selected subset over a shared 512-wide K/V. That is a third cache shape on top of the existing two.
   • mHC hyper-connections. Four residual streams mixed by a doubly-stochastic matrix replace the single residual. Every layer's dataflow changes, not just one op.

   So this is a new attention backbone in the runtime, not a family patch. It is not impossible — the streaming infrastructure is proven, the GGUF metadata is clean, and your math (41.7 + 0.6 + 5.0 GiB against 48 GB) is exactly the working-set profile the expert store is built for — but it is engine work measured in months, not days.

   Where you can help, if you want to:

   1. Measure the ceiling. On your M5 Max, run the ds4 Metal SSD-streaming path you cited on this exact GGUF and report decode tok/s with a ~16 GiB expert cache. That single number tells us whether the payoff justifies the backbone work before anyone writes a line of Splash code.
   2. Isolate the streaming economics. If llama.cpp and ds4 both read the same file, a side-by-side of their expert hit rates and SSD read volume at equal cache sizes would be directly usable for sizing the resident pool.
   3. Tensor-name map. A dump of the tensor names/shapes from the GGUF header would let us see how far the existing loader can go before the attention divergence — worth doing regardless.

   We can't promise a date, but the path is: prove the throughput on ds4, then decide whether to build KDA/DSA/mHC in the runtime. If the numbers are good, that changes the priority of this family.
2. dmkjr commented on Nov 12, 2026

   

   dmkjr

   on Nov 12, 2026

   Last edited by dmkjr

   Contributor

   More actions

   To be concrete about what we'd need to build:

   Missing today

   • No GLM family in the runtime. The engine's model descriptor knows Qwen3.8, Qwen3.6-MoE, Qwen4-Exp and Flint only — glm5-next has no layout, so the GGUF won't even load.
   • The expert store itself is ready: routed experts paged from SSD, a resident pool sized to the machine, top-8 routing, one shared expert, dense leading blocks. Your 152/288 pruning reads straight out of the header and would be accepted.
   • The attention is what isn't: KDA (linear attention with a fixed recurrent state — no KV cache at all), KPool-DSA (pooled sparse attention over a shared 512-wide K/V), and mHC (four residual streams mixed by a doubly-stochastic matrix). All three need Metal kernels and a state layout that doesn't exist yet.

   The possibility
   It's tractable, not speculative. The streaming half is already proven at 54 GB models on 48 GB machines. What remains is kernels for three attention types plus a draft model to pair with the target. The realistic sequence is: get a decode number on the ds4 Metal path for this exact file, and if it beats what llama.cpp's 0.29 tok/s shows by a wide enough margin, the backbone work becomes worth scheduling. The tensor-name dump and the hit-rate comparison you offered map almost 1:1 onto the layout work, so testing on your M5 Max is the cheapest way to de-risk the whole thing before anyone writes runtime code.