Skip to content

[Fix] Don't let a failed deep_ep import-time check kill servers that never use DeepEP - #40671

Merged
ch-wan merged 2 commits into
sgl-project:mainfrom
LiRunGuo:fix/deepep-import-guard
Oct 6, 2026
Merged

ch-wan merged 2 commits into
sgl-project:mainfrom
LiRunGuo:fix/deepep-import-guard

Conversation

@LiRunGuo

@LiRunGuo LiRunGuo commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Since #33932 (2026-08-07), sgl-deep-ep is a hard dependency of the sglang wheel, so deep_ep is always installed.
deep_ep/__init__.py runs host checks at import time:

  • check_nccl_so() asserts that exactly one libnccl is mapped and that it is byte-identical to the pip NCCL (AssertionError, or FileNotFoundError when a mapping reads (deleted)).
  • find_cuda_home() asserts that a CUDA home is found (AssertionError).
  • init_jit() initializes the C++ JIT runtime.

The guard in token_dispatcher/deepep.py dates from when "not installed" was the only failure mode, and catches only ImportError; deepep_v2.py catches (ImportError, OSError).
Every server process imports these modules, including the launcher:

launch_server -> http_server -> engine -> data_parallel_controller -> scheduler
  -> disaggregation/decode -> managers/utils -> layers/logits_processor
  -> layers/logprob_processor -> model_executor/runner_utils/__init__
  -> runner_utils/deepep_adapter -> layers/moe/token_dispatcher -> deepep -> deep_ep

So one failed check stops the server at import time, even for dense models where DeepEP is never requested.

Reproduction (H200, Qwen3-1.7B, dense, no DeepEP flags)

# any second libnccl on the host; here a source build of NCCL 2.32.3 vs the pip 2.29.7
export LD_PRELOAD=/path/to/nccl/build/lib/libnccl.so.2
python -m sglang.launch_server --model-path Qwen/Qwen3-1.7B --port 30006

Before (main @ 00986c8): the launcher exits with code 1 during import.

File "python/sglang/srt/layers/moe/token_dispatcher/deepep.py", line 53, in <module>
File ".venv/lib/python3.12/site-packages/deep_ep/__init__.py", line 91, in <module>
AssertionError: Duplicate NCCL runtime found in the current system: .../nccl/build/lib/libnccl.so.2.32.3 and .../site-packages/nvidia/nccl/lib/libnccl.so.2

After: the server starts with the custom NCCL mapped, and /generate returns Paris. The capital of the United States is Washington, D for "The capital of France is".
The same failure also happened without LD_PRELOAD on a shared network filesystem, where the detokenizer saw the NCCL mapping as (deleted) and filecmp raised FileNotFoundError.

Modifications

  • deepep.py: catch any exception from the optional import, keep it in _deepep_import_error, and chain it into the ImportError raised when a DeepEP dispatcher is created. The message now says "not available" instead of "not installed", because the package usually is installed.
  • deepep_v2.py: widen (ImportError, OSError) to Exception for the deep_ep import. The v2 module already stores and reports the original error.

Users who request DeepEP still get a hard error, now with the root cause instead of a misleading "not installed".
The v1 try block also covers the sglang_per_token_group_quant_fp8 import, as before; a failure there is reported the same way.

Follow-up, not in this PR: runner_utils/__init__.py eagerly re-exports DeepEPCudaGraphRunnerAdapter, so importing runner_utils.pool from logprob_processor pulls every MoE dispatcher (and deep_ep) into the launcher, tokenizer and detokenizer. Making that re-export lazy would stop these processes from loading DeepEP at all.

Accuracy Tests

No model forward, kernel or sampling code is touched; only the import guard of the two DeepEP dispatcher modules changes.

  • New CPU test test/registered/unit/layers/moe/test_deepep_import_guard.py runs a fresh interpreter with a stand-in deep_ep whose import raises AssertionError. It checks that the dispatcher package imports, that both use_deepep flags are false, and that creating a DeepEP dispatcher raises ImportError chained to the original error. It fails on main and passes with this change.
  • Existing DeepEP unit tests: test_deepep_v2_buffer_lifecycle.py, test_deepep_v2_masked_slab.py, test_deepep_v2_wire_dtype.py, test_w4afp8_deepep_dtype.py, test_w4afp8_deepep_post_reorder.py, plus the new test: 39 passed.
  • pre-commit run --files <changed files>: all hooks pass.
  • No model code is touched; the dense end-to-end run above is the functional check.

Speed Tests and Profiling

Not applicable. The change only affects module import, not any runtime path.

Checklist


CI States

Latest PR Test (Base): ❌ Run #37439252608
Latest PR Test (Extra): ❌ Run #37439252212
Latest PR Test (AMD ROCm 10): ❌ Run #37439252555

@ch-wan

ch-wan commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator

/rerun-test test_deepep_import_guard.py

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

⚠️ Rebase Required Before Re-run

A major update has landed on main. Your PR is diverged relative to required base commit fea3c088a141.

Re-run was not dispatched. What to do:

  • Rebase your branch onto the latest main and push again
  • Follow issue #21065 for context
  • CI-fix PRs may request the bypass-maintenance label to skip this check

@LiRunGuo
LiRunGuo force-pushed the fix/deepep-import-guard branch from b660bcc to a002894 Compare October 5, 2026 12:26
…never use DeepEP

sgl-deep-ep is a hard dependency since sgl-project#33932, and deep_ep/__init__.py runs
host checks at import time (NCCL runtime match, CUDA home lookup, JIT init)
that raise AssertionError/OSError, not only ImportError. Every server process
imports the MoE token dispatchers, so one failed check (for example a second
libnccl loaded through LD_PRELOAD) stopped the launcher even for dense models.

Treat any exception from the optional deep_ep import as "DeepEP unavailable"
in both the v1 and v2 dispatchers, keep the original error, and chain it into
the ImportError raised when a DeepEP dispatcher is actually created.
@LiRunGuo
LiRunGuo force-pushed the fix/deepep-import-guard branch from a002894 to 40205da Compare October 5, 2026 12:27
@LiRunGuo

LiRunGuo commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Thanks! Trimmed the comments down to the two lines that carry the rationale, and removed my earlier CI-trigger requests from this thread.

Also rebased onto current main (83a6e1d3), which clears the fea3c088a141 base requirement that blocked your /rerun-test dispatch — it should go through now.

@ch-wan

ch-wan commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

/rerun-test test_deepep_import_guard.py

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

⚠️ Rebase Required Before Re-run

A major update has landed on main. Your PR is diverged relative to required base commit 60878c3fba50.

Re-run was not dispatched. What to do:

  • Rebase your branch onto the latest main and push again
  • Follow issue #21065 for context
  • CI-fix PRs may request the bypass-maintenance label to skip this check

@ch-wan

ch-wan commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

/rerun-test test_deepep_import_guard.py

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

🚀 ubuntu-latest (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/layers/moe/test_deepep_import_guard.py

@ch-wan
ch-wan merged commit b563bbd into sgl-project:main Oct 6, 2026
100 of 116 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants