[Speculative Decoding] Validate vocabulary compatibility in STANDALONE mode by jmamou · Pull Request #23838 · sgl-project/sglang

jmamou · 2026-04-27T11:51:40Z

Motivation

STANDALONE speculative decoding requires the draft model to share the same vocabulary as the target model. If the vocabularies differ, draft token IDs map to different strings in the target vocabulary, making the speculative decoding lossy — the output distribution no longer matches what the target model would produce alone. Previously, SGLang silently accepted any draft model with no error or warning.

Concrete example with real models (google/gemma-2-9b as target, bigcode/tiny_starcoder_py as draft):

The STANDALONE token-ID data flow is: (1) draft forward pass → token IDs, (2) target verifies those IDs, (3) target decodes accepted IDs into the final string. If the two tokenizers map the same integer to different strings, every accepted draft token is silently wrong.

Without the fix (main branch)

$ python scripts/demo_vocab_mismatch_bug.py --mode buggy

======================================================================
Target : google/gemma-2-9b
Draft  : bigcode/tiny_starcoder_py  (WRONG — different tokenizer family)
======================================================================
  target vocab_size : 256,000
  draft  vocab_size : 49,152

Sample token IDs that decode to different strings in each vocabulary:
      ID           gemma-2-9b                tiny_starcoder_py
  ------  ----------------------------  ----------------------------
       4            '<mask>'                         ''
       5           '<2mass>'                         ''
       6           '[@BOS@]'                         ''
       7          '<unused0>'                        ''
       8          '<unused1>'                        ''
       9          '<unused2>'                        ''

======================================================================
Simulating the STANDALONE token-ID data flow
  step 1 — draft encodes prompt    (uses draft vocabulary)
  step 2 — target verifies IDs     (skipped in this simulation)
  step 3 — target decodes accepted IDs  (uses TARGET vocabulary)
======================================================================
  prompt         : 'def add(a, b): return a + b'
  draft IDs      : [589, 1015, 26, 83, 30, 323, 711, 442, 312, 474, 323]
  target decodes : ' = imp<unused19><unused76><unused23>jure…'  <<< SILENT CORRUPTION

  prompt         : 'The capital of France is'
  draft IDs      : [1318, 18926, 432, 45600, 438]
  target decodes : 'те वि… Pix…'  <<< SILENT CORRUPTION

  prompt         : 'import numpy as np'
  draft IDs      : [465, 6436, 619, 2065]
  target decodes : '…Showue Be'  <<< SILENT CORRUPTION

======================================================================
MODE: buggy  —  main branch, no vocab validation

  SGLang accepts the launch arguments without any error or warning.
  Inference runs; every accepted draft token is silently corrupted.

  The command that would silently succeed on main:

    python -m sglang.launch_server \
        --model-path google/gemma-2-9b \
        --speculative-algorithm STANDALONE \
        --speculative-draft-model-path bigcode/tiny_starcoder_py \
        --speculative-num-steps 5 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 5

  (no ValueError, no warning — server starts and corrupts silently)

With the fix (standalone-vocab-check branch)

$ python scripts/demo_vocab_mismatch_bug.py --mode fixed

  [... same corruption output as above ...]

======================================================================
MODE: fixed  —  standalone-vocab-check branch, validation active

  StandaloneWorker._validate_vocab_compatibility is called during
  __init__, after both models are loaded, before serving any request.

  ValueError raised at startup:

    STANDALONE speculative decoding requires the draft model to share the
    same vocabulary as the target model, but got target vocab_size=256000
    and draft vocab_size=49152. To use draft models with different
    vocabularies, use --speculative-algorithm TLI instead.

  Server exits immediately; no corrupted output is ever produced.

Modifications

python/sglang/srt/speculative/standalone_worker.py

Adds StandaloneWorker._validate_vocab_compatibility(), a static method called at the end of __init__ (after both models are loaded). It raises ValueError in two cases:

vocab_size mismatch — fast necessary check; catches obvious cross-family pairs (e.g. Llama 128k vs. Qwen 152k).
get_vocab() dict mismatch — catches the subtler case where both models have the same vocab_size but different token-to-ID mappings (e.g. different tokenizer families that happen to be the same size). Skipped when either tokenizer is None (--skip-tokenizer-init). This is the same approach used by HuggingFace Transformers (PR #35029) to distinguish homogeneous from heterogeneous vocabulary pairs.

Both error messages point users to --speculative-algorithm TLI for cross-family pairs (see companion PR #22883).

test/registered/unit/spec/test_standalone_vocab_check.py

8 CPU-only unit tests (no server, no GPU) registered in stage-a-test-cpu:

Test	What it checks
`test_identical_vocab_passes`	Same size + same mapping → no error
`test_mismatched_vocab_size_raises`	Different sizes → `ValueError` with sizes in message
`test_same_size_different_mapping_raises`	Same size, different token strings → `ValueError`
`test_none_target_tokenizer_skips_mapping_check`	`None` target tokenizer → only size checked
`test_none_draft_tokenizer_skips_mapping_check`	`None` draft tokenizer → only size checked
`test_both_tokenizers_none_skips_mapping_check`	Both `None` → only size checked
`test_error_message_contains_vocab_sizes`	Size-mismatch error includes both sizes
`test_error_message_suggests_tli_for_size_mismatch`	Both error types suggest `TLI`

No behaviour change for existing valid STANDALONE configurations (same-family, same-vocab pairs).

Accuracy Tests

N/A — this change only adds a startup validation; it does not modify any model forward pass or token selection logic.

Speed Tests and Profiling

N/A — the validation runs once at server startup and has no impact on inference throughput.

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

CI States

Latest PR Test (Base): ✅ Run #28356370156
Latest PR Test (Extra): ❌ Run #28356369823

…coding STANDALONE requires the draft model to share the same vocabulary as the target model, but previously silently accepted any draft model, producing lossy outputs when vocabularies differed. Changes: - Extract a static _validate_vocab_compatibility() method on StandaloneWorker that performs two checks: 1. vocab_size must match (fast necessary check). 2. Full get_vocab() dicts must match (catches same-size but different-content tokenizers, e.g. different tokenizer families that happen to be the same size). Either tokenizer being None (--skip-tokenizer-init) skips the deep check. Both error messages point users to --speculative-algorithm TLI for cross-family pairs. - Add test/registered/unit/spec/test_standalone_vocab_check.py: 8 CPU- only unit tests covering identical vocab, size mismatch, same-size mapping mismatch, None tokenizer handling, and error message content.

gemini-code-assist

Code Review

This pull request introduces vocabulary compatibility validation for standalone speculative decoding, ensuring that draft and target models share identical token-to-ID mappings. The implementation adds a validation check during StandaloneWorker initialization, updates documentation to reflect this requirement, and includes unit tests. Feedback suggests that using get_vocab() for full vocabulary comparison may be computationally expensive for large models and could lead to an AttributeError if the method is not supported by all tokenizer types.

…tibility - Add hasattr(tokenizer, 'get_vocab') guards before calling get_vocab() to avoid AttributeError on tokenizer types that do not implement the method (e.g. TiktokenTokenizer). - Update docstring to mention TiktokenTokenizer and document the measured O(vocab_size) cost (~300 ms on a 256 k-token vocabulary, negligible at startup). - Add test_tokenizer_without_get_vocab_skips_mapping_check to cover this case.

kpham-sgl

@jmamou Can you put all the TLI changes in #22883. Thanks~

jmamou · 2026-05-26T10:11:57Z

@kpham-sgl

@jmamou Can you put all the TLI changes in #22883. Thanks~

Thanks for the review! I'd like to suggest keeping these as two PRs, if you're open to it.

#23838 is a small, self-contained safety fix that prevents silent output corruption today on main, independent of TLI.

The one valid coupling is the error message, which currently points users to --speculative-algorithm TLI. I'm happy to soften that wording so #23838 doesn't reference TLI by name — e.g.:

STANDALONE speculative decoding requires the draft model to share the same vocabulary as the target model, but got target vocab_size=… and draft vocab_size=…. Use a draft model with a matching vocabulary, or a speculative algorithm that supports heterogeneous vocabularies.

Then once #22883 lands, I can do a one-line follow-up to add the TLI hint back.

Happy to fold them if you'd still prefer a single PR — just wanted to flag the trade-off. Let me know which you'd like.

Per review feedback on sgl-project#23838, move the documentation note describing the vocab requirement (and the pointer to TLI) out of this PR. It will land in sgl-project#22883 alongside TLI itself. This PR keeps only the startup validation and error message.

Per review feedback on sgl-project#23838, StandaloneWorker owns the draft side (draft_tokenizer and draft_vocab_size), so reading them through self is clearer than threading them through the call site. Target side is still passed in since it lives on the target_worker. Test helper updated to pass a fake self carrying draft_model_runner. model_config.vocab_size and tokenizer; all 9 unit tests pass.

Adds a startup-validation note above the STANDALONE example so users who hit the vocab-mismatch ValueError have an in-doc explanation and a pointer to TLI for cross-family pairs. Originally proposed in PR sgl-project#23838 and moved here per review feedback. Co-Authored-By: Claude Opus 4.7 <[email protected]>

kpham-sgl · 2026-05-26T19:22:43Z

@kpham-sgl

@jmamou Can you put all the TLI changes in #22883. Thanks~

Thanks for the review! I'd like to suggest keeping these as two PRs, if you're open to it.

#23838 is a small, self-contained safety fix that prevents silent output corruption today on main, independent of TLI.

The one valid coupling is the error message, which currently points users to --speculative-algorithm TLI. I'm happy to soften that wording so #23838 doesn't reference TLI by name — e.g.:

STANDALONE speculative decoding requires the draft model to share the same vocabulary as the target model, but got target vocab_size=… and draft vocab_size=…. Use a draft model with a matching vocabulary, or a speculative algorithm that supports heterogeneous vocabularies.

Then once #22883 lands, I can do a one-line follow-up to add the TLI hint back.

Happy to fold them if you'd still prefer a single PR — just wanted to flag the trade-off. Let me know which you'd like.

Ah no I still want to keep it as two separate PRs. I just mean to avoid referring to TLI in this PR. The new error message wording makes sense to me

jmamou · 2026-05-26T19:32:34Z

@kpham-sgl

@jmamou Can you put all the TLI changes in #22883. Thanks~

Thanks for the review! I'd like to suggest keeping these as two PRs, if you're open to it.
#23838 is a small, self-contained safety fix that prevents silent output corruption today on main, independent of TLI.
The one valid coupling is the error message, which currently points users to --speculative-algorithm TLI. I'm happy to soften that wording so #23838 doesn't reference TLI by name — e.g.:
STANDALONE speculative decoding requires the draft model to share the same vocabulary as the target model, but got target vocab_size=… and draft vocab_size=…. Use a draft model with a matching vocabulary, or a speculative algorithm that supports heterogeneous vocabularies.
Then once #22883 lands, I can do a one-line follow-up to add the TLI hint back.
Happy to fold them if you'd still prefer a single PR — just wanted to flag the trade-off. Let me know which you'd like.

Ah no I still want to keep it as two separate PRs. I just mean to avoid referring to TLI in this PR. The new error message wording makes sense to me

Thanks @kpham-sgl, that works!

The docs note that mentioned TLI has been moved into TLI PR #22883 — see commit e4f0899.

Address review feedback on sgl-project#23838 — replace the "--speculative-algorithm TLI instead" hint in both ValueErrors with neutral wording, and remove TLI assertions from the unit tests so this PR no longer references TLI by name.

kpham-sgl · 2026-05-26T19:52:30Z

/tag-and-rerun-ci

jmamou · 2026-05-26T20:05:20Z

@kpham-sgl
failing checks are not caused by changes in the PR

kpham-sgl · 2026-05-26T20:46:52Z

@kpham-sgl failing checks are not caused by changes in the PR

It does. Can you take a look at this failure? https://github.com/sgl-project/sglang/actions/runs/26471486639/job/77946051688?pr=23838

FYI we recently update our CI definition. You can point your agent to the test writing skills

ValueError: Tests registered to invalid suites:
  /home/runner/work/sglang/sglang/test/registered/unit/spec/test_standalone_vocab_check.py: backend=CPU, suite='stage-a-test-cpu'

… test The 'stage-' prefix is reserved for AMD/NPU suites. CPU per-commit suites are 'base-a-test-cpu' / 'base-b-test-cpu', so registering to 'stage-a-test-cpu' caused run_suite.py validation to fail with 'Tests registered to invalid suites'.

jmamou · 2026-05-27T08:20:13Z

/tag-and-rerun-ci

jmamou · 2026-05-27T08:56:11Z

/rerun-failed-ci

jmamou · 2026-05-27T10:31:41Z

/rerun-failed-ci

jmamou · 2026-05-27T20:17:53Z

/tag-and-rerun-ci extra

jmamou · 2026-05-27T20:21:19Z

@kpham-sgl
the failing check is PR Test Extra / call-gate / pr-gate, which is a label gate, not a test failure:

Missing required label 'run-ci-extra'. Add the label (e.g. via /tag-and-rerun-ci extra) to opt this PR into the extra test workflow.

I tried /tag-and-rerun-ci extra myself but the label wasn't applied (looks like the slash command requires maintainer permissions). Could one of you add the run-ci-extra label so the extra test workflow can run?

Thanks

kpham-sgl · 2026-05-30T06:51:31Z

/rerun-failed-ci

standalone_worker.py was deleted upstream (Spec V1 deprecated in sgl-project#25464). Port _validate_vocab_compatibility to StandaloneWorkerV2 in standalone_worker_v2.py and update the test accordingly.

jmamou · 2026-06-17T10:54:51Z

/rerun-failed-ci

ronhuafeng · 2026-06-29T05:53:16Z

Rechecked this after #24051 was auto-closed by the inactivity bot. On current head ab75b6132af41eb5667a884b9c6d93af6eadf723, the focused PR validation looks good:

Docker lmsysorg/sglang:latest, with imports forced to the checked-out PR source: test/registered/unit/spec/test_standalone_vocab_check.py -q passes (8 passed).
py_compile passes for python/sglang/srt/speculative/standalone_worker_v2.py and the new test file.
Changed-file git diff --check against the PR merge base passes for the two touched files.

One small validation note: my first container run accidentally imported the image preinstalled SGLang package rather than the PR checkout, which reproduced AttributeError: StandaloneWorkerV2 has no _validate_vocab_compatibility. After pinning imports to the PR source, the focused tests passed, so that failure was an environment/path issue rather than a PR failure.

CI status also looks consistent with this: lint plus Arm64/Xeon build-test are green, and the upstream Base matrix includes successful H100/H200/B200 jobs on this head. The remaining visible red Extra gates are missing run-ci-extra/gate fallout; I did not classify the older NPU/XPU/AMD failures as PR-specific from this local check.

Drop test/registered/unit/spec/test_standalone_vocab_check.py. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

jmamou · 2026-06-29T07:41:33Z

Thanks for the re-validation @ronhuafeng! Good to know everything passes cleanly on current head.
Could a maintainer please merge this when convenient?

kpham-sgl · 2026-06-29T07:42:52Z

Thanks for the re-validation @ronhuafeng! Good to know everything passes cleanly on current head. Could a maintainer please merge this when convenient?

Sorry I have been too busy recently... Merging this in soon

kpham-sgl · 2026-06-29T08:33:40Z

/rerun-test test/registered/spec/test_spec_standalone.py test/registered/spec/test_spec_standalone_extra.py

github-actions · 2026-06-29T08:34:04Z

Results for /rerun-test test/registered/spec/test_spec_standalone.py test/registered/spec/test_spec_standalone_extra.py:

🚀 1-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/spec/test_spec_standalone.py
cd test/ && python3 registered/spec/test_spec_standalone_extra.py

…E mode (sgl-project#23838) Co-authored-by: kpham-sgl <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

jmamou requested review from Qiaolin-Yu, Ying1123, hnyls2002, merrymercy and wisclmy0611 as code owners April 27, 2026 11:51

github-actions Bot added documentation Improvements or additions to documentation speculative-decoding labels Apr 27, 2026

gemini-code-assist Bot reviewed Apr 27, 2026

View reviewed changes

Comment thread python/sglang/srt/speculative/standalone_worker.py Outdated

jmamou mentioned this pull request Apr 29, 2026

[Bug] STANDALONE speculative decoding silently corrupts output when draft/target vocabularies differ #24051

Closed

5 tasks

kpham-sgl reviewed May 26, 2026

View reviewed changes

Comment thread docs/advanced_features/speculative_decoding.md Outdated

Comment thread python/sglang/srt/speculative/standalone_worker.py Outdated

Merge branch 'main' into standalone-vocab-check

c46be68

jmamou requested a review from zijiexia as a code owner May 26, 2026 09:41

Merge branch 'main' into standalone-vocab-check

d0de6ad

jmamou added 2 commits May 26, 2026 03:17

kpham-sgl reviewed May 26, 2026

View reviewed changes

Comment thread test/registered/unit/spec/test_standalone_vocab_check.py Outdated

jmamou force-pushed the standalone-vocab-check branch from 0e62f2d to 29b1054 Compare May 26, 2026 19:48

kpham-sgl approved these changes May 26, 2026

View reviewed changes

Merge branch 'main' into standalone-vocab-check

52ea486

github-actions Bot added the run-ci label May 26, 2026

kpham-sgl self-assigned this May 27, 2026

jmamou added 2 commits May 27, 2026 01:15

Merge remote-tracking branch 'upstream/main' into standalone-vocab-check

b6f7004

Merge branch 'main' into standalone-vocab-check

638eb9d

Merge branch 'main' into standalone-vocab-check

020bc39

Merge branch 'main' into standalone-vocab-check

d8c386d

Merge upstream/main: port vocab validation to standalone_worker_v2

ab75b61

standalone_worker.py was deleted upstream (Spec V1 deprecated in sgl-project#25464). Port _validate_vocab_compatibility to StandaloneWorkerV2 in standalone_worker_v2.py and update the test accordingly.

kpham-sgl and others added 2 commits June 28, 2026 23:43

test(spec): remove STANDALONE vocab-check unit test

7dd4547

Drop test/registered/unit/spec/test_standalone_vocab_check.py. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Merge branch 'main' into standalone-vocab-check

d22ae9a

kpham-sgl merged commit 5556631 into sgl-project:main Jun 29, 2026
167 of 181 checks passed

jmamou deleted the standalone-vocab-check branch June 30, 2026 05:30

Uh oh!

Conversation

jmamou commented Apr 27, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

CI States

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

kpham-sgl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jmamou commented May 26, 2026

Uh oh!

kpham-sgl commented May 26, 2026

Uh oh!

jmamou commented May 26, 2026

Uh oh!

Uh oh!

kpham-sgl commented May 26, 2026

Uh oh!

jmamou commented May 26, 2026

Uh oh!

kpham-sgl commented May 26, 2026

Uh oh!

jmamou commented May 27, 2026

Uh oh!

jmamou commented May 27, 2026

Uh oh!

jmamou commented May 27, 2026

Uh oh!

jmamou commented May 27, 2026

Uh oh!

jmamou commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kpham-sgl commented May 30, 2026

Uh oh!

jmamou commented Jun 17, 2026

Uh oh!

ronhuafeng commented Jun 29, 2026

Uh oh!

jmamou commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kpham-sgl commented Jun 29, 2026

Uh oh!

kpham-sgl commented Jun 29, 2026

Uh oh!

github-actions Bot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jmamou commented Apr 27, 2026 •

edited by github-actions Bot

Loading

jmamou commented May 27, 2026 •

edited

Loading

jmamou commented Jun 29, 2026 •

edited

Loading

github-actions Bot commented Jun 29, 2026 •

edited

Loading