Repository navigation
[Fix] Don't let a failed deep_ep import-time check kill servers that never use DeepEP - #40671
Merged
Merged
Conversation
LiRunGuo
requested review from
BBuf,
Edwardf0t1,
Fridge003,
HaiShaw,
Ying1123,
ch-wan,
ispobock and
merrymercy
as code owners
September 22, 2026 03:27
LiRunGuo
force-pushed
the
fix/deepep-import-guard
branch
from
September 29, 2026 18:57
474805f to
b660bcc
Compare
ch-wan
approved these changes
Oct 5, 2026
Collaborator
|
/rerun-test test_deepep_import_guard.py |
Contributor
|
LiRunGuo
force-pushed
the
fix/deepep-import-guard
branch
from
October 5, 2026 12:26
b660bcc to
a002894
Compare
…never use DeepEP sgl-deep-ep is a hard dependency since sgl-project#33932, and deep_ep/__init__.py runs host checks at import time (NCCL runtime match, CUDA home lookup, JIT init) that raise AssertionError/OSError, not only ImportError. Every server process imports the MoE token dispatchers, so one failed check (for example a second libnccl loaded through LD_PRELOAD) stopped the launcher even for dense models. Treat any exception from the optional deep_ep import as "DeepEP unavailable" in both the v1 and v2 dispatchers, keep the original error, and chain it into the ImportError raised when a DeepEP dispatcher is actually created.
LiRunGuo
force-pushed
the
fix/deepep-import-guard
branch
from
October 5, 2026 12:27
a002894 to
40205da
Compare
Contributor
Author
|
Thanks! Trimmed the comments down to the two lines that carry the rationale, and removed my earlier CI-trigger requests from this thread. Also rebased onto current |
Collaborator
|
/rerun-test test_deepep_import_guard.py |
Contributor
|
Collaborator
|
/rerun-test test_deepep_import_guard.py |
Contributor
|
🚀 |
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Since #33932 (2026-08-07),
sgl-deep-epis a hard dependency of thesglangwheel, sodeep_epis always installed.deep_ep/__init__.pyruns host checks at import time:check_nccl_so()asserts that exactly one libnccl is mapped and that it is byte-identical to the pip NCCL (AssertionError, orFileNotFoundErrorwhen a mapping reads(deleted)).find_cuda_home()asserts that a CUDA home is found (AssertionError).init_jit()initializes the C++ JIT runtime.The guard in
token_dispatcher/deepep.pydates from when "not installed" was the only failure mode, and catches onlyImportError;deepep_v2.pycatches(ImportError, OSError).Every server process imports these modules, including the launcher:
So one failed check stops the server at import time, even for dense models where DeepEP is never requested.
Reproduction (H200, Qwen3-1.7B, dense, no DeepEP flags)
Before (main @ 00986c8): the launcher exits with code 1 during import.
After: the server starts with the custom NCCL mapped, and
/generatereturnsParis. The capital of the United States is Washington, Dfor "The capital of France is".The same failure also happened without
LD_PRELOADon a shared network filesystem, where the detokenizer saw the NCCL mapping as(deleted)andfilecmpraisedFileNotFoundError.Modifications
deepep.py: catch any exception from the optional import, keep it in_deepep_import_error, and chain it into theImportErrorraised when a DeepEP dispatcher is created. The message now says "not available" instead of "not installed", because the package usually is installed.deepep_v2.py: widen(ImportError, OSError)toExceptionfor thedeep_epimport. The v2 module already stores and reports the original error.Users who request DeepEP still get a hard error, now with the root cause instead of a misleading "not installed".
The v1
tryblock also covers thesglang_per_token_group_quant_fp8import, as before; a failure there is reported the same way.Follow-up, not in this PR:
runner_utils/__init__.pyeagerly re-exportsDeepEPCudaGraphRunnerAdapter, so importingrunner_utils.poolfromlogprob_processorpulls every MoE dispatcher (anddeep_ep) into the launcher, tokenizer and detokenizer. Making that re-export lazy would stop these processes from loading DeepEP at all.Accuracy Tests
No model forward, kernel or sampling code is touched; only the import guard of the two DeepEP dispatcher modules changes.
test/registered/unit/layers/moe/test_deepep_import_guard.pyruns a fresh interpreter with a stand-indeep_epwhose import raisesAssertionError. It checks that the dispatcher package imports, that bothuse_deepepflags are false, and that creating a DeepEP dispatcher raisesImportErrorchained to the original error. It fails on main and passes with this change.test_deepep_v2_buffer_lifecycle.py,test_deepep_v2_masked_slab.py,test_deepep_v2_wire_dtype.py,test_w4afp8_deepep_dtype.py,test_w4afp8_deepep_post_reorder.py, plus the new test: 39 passed.pre-commit run --files <changed files>: all hooks pass.Speed Tests and Profiling
Not applicable. The change only affects module import, not any runtime path.
Checklist
CI States
Latest PR Test (Base): ❌ Run #37439252608
Latest PR Test (Extra): ❌ Run #37439252212
Latest PR Test (AMD ROCm 10): ❌ Run #37439252555