Repository navigation
qwen3 : add sliding-window attention pattern support - #26286
littlewine wants to merge 5 commits into
Conversation
|
Hi @littlewine, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
|
hi, |
Just a guess, but the formatting on [contributing guidelines] has been lost; perhaps that's it? |
|
thanks for your reply @HermiG ! |
Two corrections to what I told the user, both from Jina's own published files rather than from my measurements. **The download was incomplete.** The entry fetched `jina-reranker-v3.5-Q4_K_M.gguf` and nothing else. Jina's GGUF README says the scoring MLP is "not baked into the GGUF" — it is `projector.safetensors`, and scoring is `cos(projector(doc_hidden), projector(query_hidden))`. Downloaded it to check: 1.5 MB, two BF16 matrices, `projector.0.weight` [512, 1024] and `projector.2.weight` [512, 512], Linear -> ReLU -> Linear, sha256 exactly as declared. `rerank.py` also needs `tokenizer.json` for its 125-doc block splitting. So the 397 MB on offer could never have scored anything, with any runtime — an incomplete artifact wearing the model's name. Now 410 MB, all three files, each pinned by digest. **The stated reason was one third of the truth.** I said the blocker was the sliding-window config being discarded. Jina's README says something stronger and checkable: the model needs a FORKED `llama-embedding` (github.com/littlewine/llama.cpp) with three patches upstream lacks — `--output-token-ids` for per-token hidden states, a non-causal encoder mode, and the SWA fix. `rerank.py:6` lists all three together. Open PR ggml-org/llama.cpp#26286 covers only the third, and has been untouched since 2026-07-31. The other two are the larger problem: even with the SWA fix, node-llama-cpp exposes no per-position hidden states (`ControlledEvaluateIndexOutput` is vocabulary space; `getEmbeddingFor` probes as mean-like pooling), so there would still be nothing to feed the projector. Also off by one: the base repo's `config.json` marks **16** of the 28 layers `sliding_attention`, not 17. Counted from the config, not remembered. The base repo is not an alternative route either: safetensors + `custom_code` (`JinaForRanking` in `modeling.py`), no `onnx/`, so transformers.js cannot load it and there is nothing to export from without Python at install time. Behaviour note: an install that already fetched only the .gguf now reads as `partial` and resumes with the 13 MB delta. That is correct — it was partial. Verified: 55 reranker tests + 8 renderer tests green, electron typecheck clean; `projector.safetensors` and `tokenizer.json` both fetched from the pinned revision 884f7c67 and the projector's sha256 confirmed byte-for-byte. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01APVi12T9vuKqiHqpSFv9eP
|
I really hope this PR will be at least reviewed soon, ideally merged, to support the badly needed jina reranker. Thank you! |
| res->t_logits = cur; | ||
|
|
||
| ggml_build_forward_expand(gf, cur); | ||
| ggml_build_forward_expand(gf, cur); |
There was a problem hiding this comment.
The ggml_build_forward_expand still needs to be called, even when the lm_head is skipped, in order to construct the graph.
|
@littlewine before i get my hopes too high: any chance that llama-server will get these updates, or are they just for llama-embeddings? i just built llama-server, it still chokes on --output-token-ids... Thank you. |
* read sliding_window_pattern as bool array * fall back to scalar period for uniform interleave * route layers via llm_graph_input_attn_kv_iswa Assisted-by: Claude Sonnet
Assisted-by: Claude Sonnet
- parse --output-token-ids <id,id,...> before common_params_parse - mark only matching token IDs as outputs; emit a compact array of just those embeddings in encounter order Avoids serialising every token's hidden state when only a few special tokens are needed (e.g. reranker embed/rerank markers), cutting JSON output by orders of magnitude at long context. Assisted-by: Claude Sonnet
- emit qwen3.attention.sliding_window (scalar) and sliding_window_pattern (per-layer bool array) from the HF config - guarded on use_sliding_window so it is a no-op for standard Qwen3 dense models that do not set it Pairs with the graph-side reader that routes SWA layers to the iSWA KV cache; without this a freshly converted SWA checkpoint would lack the keys and silently fall back to full attention. Assisted-by: Claude Sonnet
Don't rely on build_pooling() to expand the graph in pooling=none mode. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
@ggerganov fixed in 491219a (rebased too, conflict resolved). let me know if it looks good or needs another pass. @mpetruc Happy to see that JR-v3.5 is welcomed by the community, but I don't have much more bandwidth to put into this PR right now. if anything else comes up, contributions welcome. |
|
@littlewine i went a different route: https://github.com/mpetruc/tabbyAPI/tree/feat/jina-reranker-flash-attn. Would be grateful for feedback. |
…tern, for jina-reranker-v3.5)
Two corrections to what I told the user, both from Jina's own published files rather than from my measurements. **The download was incomplete.** The entry fetched `jina-reranker-v3.5-Q4_K_M.gguf` and nothing else. Jina's GGUF README says the scoring MLP is "not baked into the GGUF" — it is `projector.safetensors`, and scoring is `cos(projector(doc_hidden), projector(query_hidden))`. Downloaded it to check: 1.5 MB, two BF16 matrices, `projector.0.weight` [512, 1024] and `projector.2.weight` [512, 512], Linear -> ReLU -> Linear, sha256 exactly as declared. `rerank.py` also needs `tokenizer.json` for its 125-doc block splitting. So the 397 MB on offer could never have scored anything, with any runtime — an incomplete artifact wearing the model's name. Now 410 MB, all three files, each pinned by digest. **The stated reason was one third of the truth.** I said the blocker was the sliding-window config being discarded. Jina's README says something stronger and checkable: the model needs a FORKED `llama-embedding` (github.com/littlewine/llama.cpp) with three patches upstream lacks — `--output-token-ids` for per-token hidden states, a non-causal encoder mode, and the SWA fix. `rerank.py:6` lists all three together. Open PR ggml-org/llama.cpp#26286 covers only the third, and has been untouched since 2026-07-31. The other two are the larger problem: even with the SWA fix, node-llama-cpp exposes no per-position hidden states (`ControlledEvaluateIndexOutput` is vocabulary space; `getEmbeddingFor` probes as mean-like pooling), so there would still be nothing to feed the projector. Also off by one: the base repo's `config.json` marks **16** of the 28 layers `sliding_attention`, not 17. Counted from the config, not remembered. The base repo is not an alternative route either: safetensors + `custom_code` (`JinaForRanking` in `modeling.py`), no `onnx/`, so transformers.js cannot load it and there is nothing to export from without Python at install time. Behaviour note: an install that already fetched only the .gguf now reads as `partial` and resumes with the 13 MB delta. That is correct — it was partial. Verified: 55 reranker tests + 8 renderer tests green, electron typecheck clean; `projector.safetensors` and `tokenizer.json` both fetched from the pinned revision 884f7c67 and the projector's sha256 confirmed byte-for-byte. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01APVi12T9vuKqiHqpSFv9eP
Overview
Add SWA (sliding window attention) support to Qwen3 type models.
Currently, a per-layer
sliding_window_patternbool array is ignored at model loading time for Qwen3 models. This PR adds functionality to enable converting, and loading Qwen3-based architectures (e.g. jina-reranker-v3.5) with custom/irregular SWA patterns. Fallback (no sliding window) ensures model runs with full attention.Also optimizes :
--output-token-ids)Additional information
tested:
[pass] llama-embedding --pooling last and --pooling none on Qwen3-0.6B produces correct embeddings
[pass] jina-reranker-v3.5 BF16 GGUF: NDCG@10 = 0.4604 vs HF reference 0.4616 on 50 queries of AILAStatutes
[fail] llama-bench/llama-perplexity failure due to pre-existing bug: GGML_ASSERT(buffer) in llm_graph_input_out_ids::set_input, llama-graph.cpp:201
Requirements