model : add Kolibri-1 (Aleph-Alpha) support - #30673

… itself

llama.cpp added the kolibri1 architecture upstream (ggml-org/llama.cpp#30673), and the
fork now sits on a pin that contains it, so the carried community patch no longer
applies (src/models/kolibri1.cpp already exists) and the rocmfp4 build failed fast.

Docs-Not-Needed: backend-internal patch series, documented in patches/README.md

Assisted-by: Claude Code:claude-opus-5-5

Problem: The merge of master into unstable (1f6790b) resolved
conflicts in favor of losing parts of the Kolibri-1 (ggml-org#30673) and
Beam (ggml-org#30670) commits. This caused two categories of damage:

1. Missing enum value (llama-model.h): LLM_TYPE_78B_A3_5B was absent
   from the llm_type enum, causing kolibri1.cpp to fail compilation
   with 'use of undeclared identifier'.

2. Missing model dispatch and layer fields (Beam): llama_model_beam
   struct was absent from models.h, wqkv_gate_b was absent from
   llama_layer, pattern_attn_gate_bias was absent from its declaration
   in llama-model.cpp, and the granularity_head split check for
   attn_gate_bias was absorbed into the sinks branch.

All five missing pieces are restored to match upstream commit state.

…e off by one position

llama.cpp gained Kolibri-1 support in ggml-org/llama.cpp#30673 on 2026-10-17, which makes
it a third independent implementation. Ran it on CPU over the 8 golden prompts and compared.
The result reframes the whole logit-gap question.

1. The primary oracle's per-position top-100 rows are stored one position ahead of its own
   generated chain. generated[k] == rows[k+1].top1 holds 195/195; generated[k] == rows[k].top1
   holds 63/202. The primary is greedy and correct; the recording is off by one. That is why
   prompt 0 position 0 appeared to show the primary generating a token at logprob -7.4476
   while its own top-1 sat at -0.325. Every position-by-position comparison against those
   rows was reading the wrong position. Raised with Maki; it is the vllm.cpp harness's bug.

2. llama.cpp sides with the transcription. First token 8/8 against the transcription and 6/8
   against the primary; identical 32-token sequences 7/8 against the transcription and 3/8
   against the primary. The primary is the outlier.

   The GGUF is a valid oracle: Kolibri-1-bf16 is the dequantisation of the fp8 checkpoint, not
   an independent model. Verified on layer 0 expert 0 gate_proj - fp8 is float8_e4m3fn with a
   128x128 block scale (4x20 for a 512x2560 tensor), dequant is fp8 * scale_inv, and the bf16
   tensor is that value rounded to bf16, mean abs difference 5.1e-7 on values of magnitude
   0.011, i.e. 4.6e-5 relative.

3. With the offset corrected, the primary and llama.cpp agree on the argmax at 91/91 positions
   of the common prefix. Every chain divergence is a near-tie: prompt 0 step 0 has the same
   top-2 with margins of 0.271 and 0.347 nats on the two sides, and a 0.05-0.3 nat difference
   in absolute logprob flips the order.

So the hypotheses this file spent its length on were fitted to a misaligned comparison and are
superseded. The forward is correct - the residual, RNoPE and router-dtype fixes stand, and the
semantics now match llama.cpp exactly: four RMSNorms per block with the post-norms carrying no
residual, RoPE on sliding layers only, QK-RMSNorm, and routing as select-on-(logits+bias),
weight-by-sigmoid(unbiased logits), no renorm, f32 router.

What remains is ours: chain argmax 166/248 against the transcription. Not yet measured
against the llama.cpp chains specifically, because the model needs ~150 GB resident and the
box was occupied.

Artifacts in todo/artifacts/llamacpp-oracle/: the llama.cpp chains with per-position top-20
logprobs, the runner, and the three analysis scripts. Two traps are documented there - the
native /completion endpoint is required for token ids, and LD_LIBRARY_PATH must point at the
build because a stale llama.cpp in ~/.local/lib64 breaks every binary.

#30865)

* model : drop the redundant precision overrides in the Kolibri-1 router

Follow-up to the review in #30673: the router matmul works with the
default precision, so both ggml_prec_set_* calls can go away.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* cont : fix comment

Co-authored-by: Sigbjørn Skjæret <[email protected]>

---------

Co-authored-by: Sigbjørn Skjæret <[email protected]>

…e off by one position

llama.cpp gained Kolibri-1 support in ggml-org/llama.cpp#30673 on 2026-10-17, which makes
it a third independent implementation. Ran it on CPU over the 8 golden prompts and compared.
The result reframes the whole logit-gap question.

1. The primary oracle's per-position top-100 rows are stored one position ahead of its own
   generated chain. generated[k] == rows[k+1].top1 holds 195/195; generated[k] == rows[k].top1
   holds 63/202. The primary is greedy and correct; the recording is off by one. That is why
   prompt 0 position 0 appeared to show the primary generating a token at logprob -7.4476
   while its own top-1 sat at -0.325. Every position-by-position comparison against those
   rows was reading the wrong position. Raised with Maki; it is the vllm.cpp harness's bug.

2. llama.cpp sides with the transcription. First token 8/8 against the transcription and 6/8
   against the primary; identical 32-token sequences 7/8 against the transcription and 3/8
   against the primary. The primary is the outlier.

   The GGUF is a valid oracle: Kolibri-1-bf16 is the dequantisation of the fp8 checkpoint, not
   an independent model. Verified on layer 0 expert 0 gate_proj - fp8 is float8_e4m3fn with a
   128x128 block scale (4x20 for a 512x2560 tensor), dequant is fp8 * scale_inv, and the bf16
   tensor is that value rounded to bf16, mean abs difference 5.1e-7 on values of magnitude
   0.011, i.e. 4.6e-5 relative.

3. With the offset corrected, the primary and llama.cpp agree on the argmax at 91/91 positions
   of the common prefix. Every chain divergence is a near-tie: prompt 0 step 0 has the same
   top-2 with margins of 0.271 and 0.347 nats on the two sides, and a 0.05-0.3 nat difference
   in absolute logprob flips the order.

So the hypotheses this file spent its length on were fitted to a misaligned comparison and are
superseded. The forward is correct - the residual, RNoPE and router-dtype fixes stand, and the
semantics now match llama.cpp exactly: four RMSNorms per block with the post-norms carrying no
residual, RoPE on sliding layers only, QK-RMSNorm, and routing as select-on-(logits+bias),
weight-by-sigmoid(unbiased logits), no renorm, f32 router.

What remains is ours: chain argmax 166/248 against the transcription. Not yet measured
against the llama.cpp chains specifically, because the model needs ~150 GB resident and the
box was occupied.

Artifacts in todo/artifacts/llamacpp-oracle/: the llama.cpp chains with per-position top-20
logprobs, the runner, and the three analysis scripts. Two traps are documented there - the
native /completion endpoint is required for token ids, and LD_LIBRARY_PATH must point at the
build because a stale llama.cpp in ~/.local/lib64 breaks every binary.

* model : add Kolibri-1 (Aleph-Alpha) support

Kolibri-1 is a bilingual German/English MoE reasoning model with 78B
parameters and 3.46B active per token: 50 layers, 384 routed experts plus
one shared expert.

Attention interleaves four sliding layers at a 513 window with one full
layer. The sliding layers rotate with NeoX RoPE, the full layers are NoPE.
GQA runs 48 query heads against 4 KV heads at head_dim 128 with a
QK-RMSNorm, and every block carries four RMSNorms, so the attention and the
MLP output are normalized before they rejoin the residual stream. Routing
picks the experts from logits + expert_bias and weights them with the
sigmoid of the unbiased logits, without renorm.

The converter reuses AfmoeModel for the per-expert checkpoint layout and
the expert selection bias, and writes the shared expert and the per layer
sliding window pattern. The released FP8 checkpoint needs no special code,
it goes through the generic weight_scale_inv path. Three tensor names are
added to the mapping (post_attn_norm, post_ffn_norm,
moe.router.expert_bias) and the 128k vocabulary registers the qwen2
pre-tokenizer by hash. The arch is registered in test-llama-archs, with GQA
and the sliding window pattern.

Assisted-by: DeepSeek Harness (DeepSeek V4.1 Flash)

* Update src/models/kolibri1.cpp

Co-authored-by: Sigbjørn Skjæret <[email protected]>

* Add fusion records for kolibri1

* Update src/models/kolibri1.cpp

Co-authored-by: Sigbjørn Skjæret <[email protected]>

---------

Co-authored-by: Sigbjørn Skjæret <[email protected]>

ggml-org#30865)

* model : drop the redundant precision overrides in the Kolibri-1 router

Follow-up to the review in ggml-org#30673: the router matmul works with the
default precision, so both ggml_prec_set_* calls can go away.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* cont : fix comment

Co-authored-by: Sigbjørn Skjæret <[email protected]>

---------

Co-authored-by: Sigbjørn Skjæret <[email protected]>