Repository navigation
Conversation
Contributor
Author
|
CI failures match the current upstream
This PR introduces no additional test failures. |
tc3oliver
force-pushed
the
fix/specprefill-static-prefix-boundary
branch
from
October 13, 2026 10:07
859218d to
872a240
Compare
specprefill_system_end was the full prompt length minus a render without system messages. Qwen3.5-family templates add a default system block to that render, so the result came out short. BatchedEngine also passed tools to both renders, which cancels the tool block and leaves the boundary at the system text length. VLMBatchedEngine rendered without tools or chat_template_kwargs. Measure it instead, with the two-probe render jundot#4994 added for the shared system prompt, and keep what the real prompt also starts with. Both engines now share that in BaseEngine.
tc3oliver
force-pushed
the
fix/specprefill-static-prefix-boundary
branch
from
November 7, 2026 07:34
872a240 to
936cad9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
specprefill_system_endis the full prompt length minus a render without system/developer messages, and the scheduler uses it as a prefix length. On Qwen3.5-family templates that second render gets a default reasoning-effort system block, so the result comes out short.The two engines are off in different ways. BatchedEngine passes tools to both renders, so the tool block cancels out. VLMBatchedEngine renders without tools or
chat_template_kwargs, so it counts the tool block but still loses the default block. Whatever falls outside the boundary can be sparsified.Measured on main 2165e75 with the Qwen3.8-27B tokenizer only, one short system message and one user turn. "User content starts at" is where the user's text begins in the prompt.
Fix
#4994 added a two-probe render for the shared system prompt, so I reused it instead of adding a second one. The render moves out of
_system_prefix_idsinto_render_system_prefix_ids, and_system_prefix_idsbehaves as before._specprefill_static_prefix_endthen keeps the part of that render the real prompt also starts with. It can't run past the first turn, because the two probes diverge there.With no leading system message neither main nor this branch sets a boundary. I left that alone.
A longer protected prefix leaves fewer tokens to score, so a request close to the SpecPrefill threshold may now prefill normally. That comes from reading
policy.py, I didn't measure it.This costs one more render per request while SpecPrefill is on. Time spent in
_inject_specprefill_system_endon the VLM engine, about 2.7k prompt tokens with 16 tools: 0.08 ms on main, 2.5 ms here. With 64 tools and about 6k tokens it's 5 ms.Tests: the two updated
test_chat_injects_specprefill_system_endtests fail on main on the boundary itself (assert 8 < 8and aKeyErrorbecause it isn't set). The newtests/test_specprefill_boundary.pyruns both engines against a small fake template that adds a default system block, but on main it only fails on the changed signature. I ran the tests matching specprefill, boundary, preflight, vlm or batched: 2208 passed, 118 skipped. I didn't run this on a loaded model.