/tag-and-rerun-ci

CI failure: base-b-test-1-gpu-large (1) (PR Test Base, B200, 80 GB)

Job log

Failing test: test/registered/spec/eagle/test_eagle_infer_b.py::TestEAGLEServerAdditional::test_radix_attention

Symptom: 11/12 EAGLE tests pass, then test_radix_attention fails with ConnectionRefusedError: [Errno 111] Connection refused on http://127.0.0.1:11000/generate. The server died — a 59 MB cuda-coredumps-run-1.zip artifact was produced (artifact 7073114703).

File ".../test/registered/spec/eagle/test_eagle_infer_b.py", line 104, in test_radix_attention
    run_radix_attention_test(self.base_url)
File ".../python/sglang/test/kits/radix_cache_server_kit.py", line 49, in run_radix_attention_test
    res = requests.post(base_url + "/generate", json=data)
...
urllib3.exceptions.NewConnectionError: ... [Errno 111] Connection refused
Exception: retry() exceed maximum number of retries.

Classification: this PR is a main-CI sandbox (HEAD = latest upstream/main + a no-op python/sglang/version.py comment touch, labels run-ci + run-ci-extra + bypass-fastfail), so the failure IS a main failure. 11/12 EAGLE tests on the same base_url passed and the server emitted a CUDA coredump during test_radix_attention — points to an EAGLE-specific server crash, almost certainly a flake unless it repeats.

Next step: leaving the run untouched to see whether other lanes hit the same EAGLE / coredump pattern. If this stays isolated, will classify as flake and /rerun-test test/registered/spec/eagle/test_eagle_infer_b.py.

CI failure: extra-a-test-1-gpu-large (0) (PR Test Extra, NVIDIA)

Job log

Failing test: test/registered/lora/test_lora_qwen3_8b_logprob_diff.py::TestLoRAQwen3_8BLogprobDiff::test_lora_qwen3_8b_logprob_accuracy

Symptom: scheduler dies during init with exit code -6 (SIGABRT — not SIGKILL/-9, so not the OS OOM-killer). Retry exhausted after the engine construction throws.

File ".../test/registered/lora/test_lora_qwen3_8b_logprob_diff.py", line 134, in test_lora_qwen3_8b_logprob_accuracy
    engine = sgl.Engine(...)
File ".../python/sglang/srt/entrypoints/engine.py", line 236, in __init__
    ) = self._launch_subprocesses(
File ".../python/sglang/srt/entrypoints/engine.py", line 856, in _launch_subprocesses
    scheduler_init_result.wait_for_ready()
File ".../python/sglang/srt/entrypoints/engine.py", line 651, in wait_for_ready
    infos = _wait_for_scheduler_ready(scheduler_pipe_readers, scheduler_procs)
File ".../python/sglang/srt/entrypoints/engine.py", line 1337, in _wait_for_scheduler_ready
    raise _scheduler_died_error(i, scheduler_procs[i])
RuntimeError: Rank 0 scheduler died during initialization (exit code: -6). If exit code is -9 (SIGKILL), a common cause is the OS OOM killer. Run `dmesg -T | grep -i oom` to check.
...
Exception: retry() exceed maximum number of retries.

Classification: this PR is a main-CI sandbox (HEAD = latest upstream/main + a no-op python/sglang/version.py comment touch, labels run-ci + run-ci-extra + bypass-fastfail), so the failure IS a main failure on the NVIDIA extra-a-1-gpu-large lane. Exit -6 = SIGABRT during scheduler init — could be a CUDA-kernel crash, a model-loading assertion in the LoRA path, or transient infra. Posting a separate /rerun-test for this file to differentiate flake vs persistent.

CI failure: base-b-test-1-gpu-small (5) (PR Test Base, NVIDIA, 32 GB)

Job log

Failing test: test/registered/core/test_srt_endpoint.py::TestSRTEndpoint::test_get_server_info_concurrent ("Make sure the concurrent get_server_info doesn't crash the server.")

Symptom: server returns non-JSON on concurrent /server_info calls because the server-side handler hits an AssertionError inside communicator.queueing_call. The client then dies with JSONDecodeError: Expecting value: line 1 column 1 (char 0), retries are exhausted, test errors.

Server-side traceback:

File ".../python/sglang/srt/entrypoints/http_server.py", line 635, in server_info
    await _global_state.tokenizer_manager.get_internal_state()
File ".../python/sglang/srt/managers/tokenizer_control_mixin.py", line 788, in get_internal_state
    await self.get_internal_state_communicator(req)
File ".../python/sglang/srt/managers/communicator.py", line 79, in __call__
    return await self.queueing_call(obj)
File ".../python/sglang/srt/managers/communicator.py", line 40, in queueing_call
    assert self._result_event is None
AssertionError

Client-side:

File ".../test/registered/core/test_srt_endpoint.py", line 635, in s
    server_info.json()
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Exception: retry() exceed maximum number of retries.

Classification: this PR is a main-CI sandbox (HEAD = latest upstream/main + a no-op python/sglang/version.py comment touch, labels run-ci + run-ci-extra + bypass-fastfail), so the failure IS a main failure on the NVIDIA base-b-1-gpu-small (32 GB) lane. The assertion assert self._result_event is None in communicator.queueing_call is a concurrency race in the internal-state communicator — the test (test_get_server_info_concurrent) is specifically designed to catch exactly this class of bug. Smells like a real race, not a flake, but posting /rerun-test to confirm reproducibility before escalating.

/rerun-test test/registered/spec/eagle/test_eagle_infer_b.py

🚀 1-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/spec/eagle/test_eagle_infer_b.py

/rerun-test test/registered/lora/test_lora_qwen3_8b_logprob_diff.py

/rerun-test test/registered/core/test_srt_endpoint.py

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/core/test_srt_endpoint.py

🚀 1-gpu-h100 (1 test): ❌ View workflow run

cd test/ && python3 registered/lora/test_lora_qwen3_8b_logprob_diff.py

/rerun-test test/registered/lora/test_lora_qwen3_8b_logprob_diff.py result: ❌ FAIL (reproducible)

Rerun job log

The same test fails on rerun with the same stack as the original extra-a-test-1-gpu-large (0) failure → this is NOT a flake.

Actual root cause (the SIGABRT in extra-a-test-1-gpu-large (0) was just the post-mortem; pre-coredump output reveals):

coredump: Starting GPU coredump generation
coredump: Detected an exception of type CUDBG_EXCEPTION_WARP_ILLEGAL_ADDRESS (14)
coredump:   - Device: 0

Triggered during CUDA graph capture of TestLoRAQwen3_8BLogprobDiff::test_lora_qwen3_8b_logprob_accuracy. The C++/CUDA stack from the Python faulthandler after the coredump (the abort thread):

File ".../python/sglang/srt/layers/quantization/unquant.py", line 161 in apply
File ".../python/sglang/srt/lora/layers.py", line 724 in forward
...
File ".../python/sglang/srt/model_executor/cuda_graph_runner.py", line 1112 in run_once
File ".../python/sglang/srt/model_executor/cuda_graph_runner.py", line 1134 in capture_one_batch_size
File ".../python/sglang/srt/model_executor/cuda_graph_runner.py", line 854 in _capture_one_stream
File ".../python/sglang/srt/model_executor/cuda_graph_runner.py", line 867 in capture
File ".../python/sglang/srt/model_executor/cuda_graph_runner.py", line 707 in __init__
File ".../python/sglang/srt/model_executor/model_runner.py", line 2776 in init_device_graphs

Classification: real bug on main HEAD 4a451128…, in the LoRA layer's forward path under CUDA graph capture. Likely a bad index / out-of-bounds memory access in lora/layers.py:724 (or in the unquant apply at unquant.py:161) when running Qwen3-8B with LoRA. Two-run reproducibility on the same commit confirms it's not a transient flake.

(This is the main-CI sandbox PR; the failing commit 4a451128… = latest upstream/main + a no-op python/sglang/version.py comment touch, so this bug is on main proper.)

/rerun-test test/registered/core/test_srt_endpoint.py result: ✅ PASS (flake)

Rerun job → SUCCESS.

The original failure on base-b-test-1-gpu-small (5) (test_get_server_info_concurrent, AssertionError self._result_event is None in communicator.queueing_call) did not reproduce. Classifying as flake — likely a transient race in the internal-state communicator under concurrent server_info that didn't hit the timing window on rerun. Not pursuing further.

Final per-file rerun verdicts on this main-CI sandbox:

Bisect probe: d90bc65e30 ([NPU] Fix TypeError in get_state_buf_infos when index_head_dim is None on MLA (#25383) — pre-chain, HEAD-28)

• File: test/registered/lora/test_lora_qwen3_8b_logprob_diff.py
• rerun-test run: 26073712614 — FAIL
• Tree date: 2026-05-19 (the commit on main directly preceding Tom's 23-commit refactor chain)

Verbatim CUDA error fingerprint:

coredump: Detected an exception of type CUDBG_EXCEPTION_WARP_ILLEGAL_ADDRESS (14)
coredump:   - Device: 0
Fatal Python error: Aborted
RuntimeError: Rank 0 scheduler died during initialization (exit code: -6).

Same CUDBG_EXCEPTION_WARP_ILLEGAL_ADDRESS during CUDA graph capture as on c2a212bf… / 4a451128… (current main HEAD).

→ bug PRE-EXISTS Tom's chain. The 28 PRs between d90bc65e30 and current HEAD (PRs #25703–#25728 — Tom's scheduler refactor chain — plus #25282 DeepSeek V4 host pool, #25596 LTX2 diffusion fix, #25699 PD/NIXL aux, #25689 spec_verify metric, #24710 RMSNorm dispatch) are NOT the cause.

Bisect bound moves to last-good < d90bc65e30. Next probe: ba214ef3d3 (file-move point, 5 days ago) in flight; also dispatching 229cadec04 (midpoint of ba214ef3d3..d90bc65e30, 2026-05-16) to narrow in parallel.

Bisect probes: ba214ef3d3 + 229cadec04

PROBE B: ba214ef3d3 (ci: tag-gated nightly migration — foundation + 40 whole-file moves (#24725) — file-move point, 2026-05-14)

• File: test/registered/lora/test_lora_qwen3_8b_logprob_diff.py
• rerun-test run: 26073728329 — PASS ✅

PROBE C: 229cadec04 (Update logging for inplace setting in MoE layer (#25499) — midpoint of (ba214ef3d3..d90bc65e30), 2026-05-16)

• rerun-test run: 26074082226 — PASS ✅

→ Bisect bound collapses to bug introduced in 229cadec04..d90bc65e30 (92 commits, 2026-05-16 → 2026-05-19).

Next probe: c58b47bc86 (Move PoolStats dataclass to scheduler_components.pool_stats_observer (#25618) — midpoint of the new range, 2026-05-18) — in flight as run 26075022728.

Bisect state so far:

Bisect probe: c58b47bc86 (Move PoolStats dataclass to scheduler_components.pool_stats_observer (#25618) — 2026-05-18)

• File: test/registered/lora/test_lora_qwen3_8b_logprob_diff.py
• rerun-test run: 26075022728 — PASS ✅

→ Bisect bound collapses to bug introduced in c58b47bc86..d90bc65e30 (46 commits, 2026-05-18 → 2026-05-19).

Next probe: f04c522534 ([PD] Add conclude_state to fake KV backend (#25599) — midpoint of the new range, 2026-05-18).

Bisect state:

Bisect probe: f04c522534 ([PD] Add conclude_state to fake KV backend (#25599) — 2026-05-18)

• File: test/registered/lora/test_lora_qwen3_8b_logprob_diff.py
• rerun-test run: 26075390772 — PASS ✅

→ Bisect bound collapses to bug introduced in f04c522534..d90bc65e30 (23 commits, both same-day 2026-05-18 / 2026-05-19).

Next probe: f5049709b3 (fix(eagle3): drop +1 offset on aux layer ids when first id != 1 (#25454) — midpoint, 2026-05-18).

Bisect state:

Bisect probe: f5049709b3 (fix(eagle3): drop +1 offset on aux layer ids when first id != 1 (#25454) — 2026-05-18)

• File: test/registered/lora/test_lora_qwen3_8b_logprob_diff.py
• rerun-test run: 26075730388 — PASS ✅

→ Bisect bound: bug introduced in f5049709b3..d90bc65e30 (12 commits, 2026-05-18 → 2026-05-19).

Full range (suspicion-worthy commits highlighted):

1f185c6ba8 Support draft extend cuda graph for tokenspeed_mla attention backend (#25489)  ← CUDA graph
b7267e8fce [CI] Enable weight prefetch for 8-gpu-h200 basic tests (#25684)
9e3bb9a307 [Spec] fold can_run_cuda_graph into EagleVerifyOutput (#25566)                  ← CUDA graph
c904fdd20e ci: pr-states match renamed "PR Test Base" workflow_run (#25687)
6f892047ec [misc] Throw error when single batch overlap is enabled on Hopper (#25509)      ← Hopper
878e6b8886 [SP] Fix runtime_max_tokens_per_rank for sequence parallelism (#25685)          ← midpoint
745abd6cc0 Add no_combine support to cutlass_moe_fp4 (#25688)
314dedf7c6 Use SGLANG_CACHE_DIR env for gpu_p2p_access_cache path (#25686)
b79e4b1e68 [Fix] Try to fix error caused by latest cutedsl packages (#25690)                ← cutedsl
dbac464726 [Spec]: Make Triton standalone spec test deterministic (#25303)
d028697d17 [NPU][Docs] Add Kimi-K2.5-W4A8 instance doc on NPU (#25269)
d90bc65e30 [NPU] Fix TypeError in get_state_buf_infos when index_head_dim is None on MLA (#25383)

Next probe: 878e6b8886 (midpoint).

Bisect state:

Bisect probe: 878e6b8886 ([SP] Fix runtime_max_tokens_per_rank for sequence parallelism (#25685) — 2026-05-18)

• File: test/registered/lora/test_lora_qwen3_8b_logprob_diff.py
• rerun-test run: 26076099919 — PASS ✅

→ Bisect bound: bug introduced in 878e6b8886..d90bc65e30 (6 commits).

Remaining range:

745abd6cc0 Add no_combine support to cutlass_moe_fp4 (#25688)
314dedf7c6 Use SGLANG_CACHE_DIR env for gpu_p2p_access_cache path (#25686)
b79e4b1e68 [Fix] Try to fix error caused by latest cutedsl packages (#25690)  ← prime suspect (CUDA-DSL packages)
dbac464726 [Spec]: Make Triton standalone spec test deterministic (#25303)
d028697d17 [NPU][Docs] Add Kimi-K2.5-W4A8 instance doc on NPU (#25269)
d90bc65e30 [NPU] Fix TypeError in get_state_buf_infos when index_head_dim is None on MLA (#25383)

Next probe (also the midpoint): b79e4b1e68 — the cutedsl-packages fix. This was the most suspicious commit in the wider range too (touches CUDA-DSL builds; LoRA forward → quant unquant.apply → cuBLAS path is a plausible blast radius).

Bisect state:

Bisect probe: b79e4b1e68 ([Fix] Try to fix error caused by latest cutedsl packages (#25690) — 2026-05-18)

• File: test/registered/lora/test_lora_qwen3_8b_logprob_diff.py
• rerun-test run: 26076486815 — FAIL ❌

Same CUDBG_EXCEPTION_WARP_ILLEGAL_ADDRESS (14) fingerprint as on HEAD:

coredump: Detected an exception of type CUDBG_EXCEPTION_WARP_ILLEGAL_ADDRESS (14)
RuntimeError: Rank 0 scheduler died during initialization (exit code: -6).

→ Bisect bound: bug introduced in 878e6b8886..b79e4b1e68 (3 commits inclusive of b79e4b1e68):

745abd6cc0 Add no_combine support to cutlass_moe_fp4 (#25688)
314dedf7c6 Use SGLANG_CACHE_DIR env for gpu_p2p_access_cache path (#25686)
b79e4b1e68 [Fix] Try to fix error caused by latest cutedsl packages (#25690)  ← FAIL ❌

Next probe: 314dedf7c6 (midpoint of the 3-commit range, 2026-05-18).

• If PASS → offender is b79e4b1e68 itself (the cutedsl fix).
• If FAIL → offender is 745abd6cc0 (cutlass_moe_fp4) or 314dedf7c6 (SGLANG_CACHE_DIR env path).

Bisect state:

🤖 Posted autonomously by Claude Code acting on Tom's behalf. The 9-probe bisect (PROBE_A..I) below was driven by the agent — each probe pushed a temp branch on upstream, dispatched rerun-test.yml against it, classified the result, and narrowed the range. The @-mentions are programmatic, not Tom's personal request; please push back if anything is off.

Bisect result: test_lora_qwen3_8b_logprob_diff.py regressed at b79e4b1e68 (PR #25690, [Fix] Try to fix error caused by latest cutedsl packages)

PROBE I (the deciding probe): 314dedf7c6 (Use SGLANG_CACHE_DIR env for gpu_p2p_access_cache path (#25686)) → rerun-test 26076870779 — PASS ✅

With 314dedf7c6 PASS and b79e4b1e68 FAIL on the immediately-following commit, the regression lands on b79e4b1e68 exactly.

Final bisect table

Offending change

• PR: [Fix] Try to fix error caused by latest cutedsl packages  #25690 — [Fix] Try to fix error caused by latest cutedsl packages
• Author: @Fridge003 (Co-authored-by @hnyls2002)
• Merged: 2026-05-18 23:51 UTC
• Diff: 21 +, 4 -. Touches python/pyproject.toml (switches flashinfer_python and nvidia-cutlass-dsl to the [cu13] extras variant) and scripts/ci/cuda/ci_install_dependency.sh (regex-update for [extras] notation + new purge_cutlass_libs_base() step that uninstalls nvidia-cutlass-dsl-libs-base then force-reinstalls nvidia-cutlass-dsl-libs-cu13).

The PR's own commit message explains the original bug it was fixing:

nvidia-cutlass-dsl[cu13] extras are additive on PyPI: requires_dist always pulls -libs-base AND -libs-cu13 when [cu13] is requested. Both wheels write to the same site-packages paths with different content, leaving the wrapper (cutlass.py, cu13 style) mismatched with the binding (_gpu_ops_gen.py, base style) -> GPUModuleOp signature TypeError.

The fix correctly purges -libs-base in the install script, but the LoRA Qwen3-8B forward path with CUDA graph capture now hits a kernel-side illegal address — so either the cu13 wheel's compiled kernel is broken for this path, or the purge_cutlass_libs_base step doesn't actually win in all install orderings.

Failure fingerprint (every FAIL probe + current HEAD)

coredump: Detected an exception of type CUDBG_EXCEPTION_WARP_ILLEGAL_ADDRESS (14)
Fatal Python error: Aborted
RuntimeError: Rank 0 scheduler died during initialization (exit code: -6).

Python call stack at the abort thread:
  File ".../python/sglang/srt/layers/quantization/unquant.py", line 161 in apply
  File ".../python/sglang/srt/lora/layers.py", line 724 in forward
  ...
  File ".../python/sglang/srt/model_executor/cuda_graph_runner.py", line 1112 in run_once
  File ".../python/sglang/srt/model_executor/cuda_graph_runner.py", line 1134 in capture_one_batch_size
  File ".../python/sglang/srt/model_executor/cuda_graph_runner.py", line 707 in __init__
  File ".../python/sglang/srt/model_executor/model_runner.py", line 2776 in init_device_graphs

Reproduce

# Probe latest good (PASS):
git push upstream 314dedf7c6:refs/heads/tmp-good
gh workflow run rerun-test.yml --repo sgl-project/sglang --ref tmp-good \
  -f mode=cuda -f test_command="registered/lora/test_lora_qwen3_8b_logprob_diff.py" \
  -f runs_on="1-gpu-h100" -f install_script="scripts/ci/cuda/ci_install_dependency.sh"

# Probe first bad (FAIL):
git push upstream b79e4b1e68:refs/heads/tmp-bad
gh workflow run rerun-test.yml --repo sgl-project/sglang --ref tmp-bad \
  -f mode=cuda -f test_command="registered/lora/test_lora_qwen3_8b_logprob_diff.py" \
  -f runs_on="1-gpu-h100" -f install_script="scripts/ci/cuda/ci_install_dependency.sh"

cc @Fridge003 @hnyls2002 — could you take a look? This regression has been on main since 2026-05-18 and is currently surfacing as extra-a-test-1-gpu-large (0) on the main-CI sandbox.

Diagnostic revert PR opened for verification: #25743 — /rerun-test of the failing LoRA file is pending there.

🤖 Posted autonomously by Claude Code acting on Tom's behalf. Bidirectional confirmation of the bisect result via paired diagnostic PRs.

Bisect confirmed via paired diagnostic PRs

Two sibling PRs were opened to nail down b79e4b1e68 (#25690) as the root cause:

Together with the per-commit bisect probes above, that's three independent lines of evidence:

1. Walking from a known-good 2026-05-14 down to b79e4b1e68 (9 probes, all consistent with PASS-then-FAIL at the exact commit boundary).
2. Revert-the-commit → PASS on the same test file.
3. Don't-revert (plain main + harmless touch) → FAIL on the same test file with identical fingerprint.

The regression is unambiguously b79e4b1e68 (#25690) — independent of Tom's #25703–#25728 chain.

cc @Fridge003 @hnyls2002 — could you take a look? Closing the two diagnostic PRs now.

🤖 Posted autonomously by Claude Code acting on the user's behalf. First full round complete: 2 real B200 regressions (both #27063), and 11 CUDA jobs that died at install on a transient GitHub-releases 504 — those never ran their tests, so I'm re-running the failed set to actually verify those suites on main. Please push back if any conclusion is off.

Round 1 complete — head d16481061d (107 pass / 25 fail / 25 skip / 2 cancelled)

🔴 Real main regressions (2, both from #27063):

Both reported on #27063. They will re-fail on rerun (deterministic).

🟡 Transient install 504 — 11 CUDA jobs whose tests never ran: extra-a, base-b-small (0/4/5/7), base-b-large (8), base-b-b200 (1), base-c-b200 (1), base-c-h100 (0), base-c-gb300 (0), extra-b-b200 (1). Same Failed to download sglang-kernel==0.4.3+cu130 → 504 across all; 107 other jobs pulled the same wheel fine, so the CDN incident is over. Re-running these to actually exercise their suites.

⚪ h20 ignored. 🔵 Non-CUDA (not ours): Xeon req_lens int64, XPU exit-137, NPU container, AMD mi325 / mi35x / mi35x-disagg. ⚫ Cancelled (not failures): base-c-h100 (4), notify-pr-states.

Triggering /rerun-failed-ci next. After the rerun settles I'll confirm the 504 suites pass, leave the two #27063 regressions as the standing verdict, and close this sandbox PR (do not merge).

/rerun-failed-ci

🤖 Posted autonomously by Claude Code acting on the user's behalf. Final verdict for this tip-of-main CI verification: after clearing a transient GitHub-releases 504 install incident (all 11 affected CUDA jobs passed on rerun), the only real CUDA failures on main are 2 deterministic B200 regressions, both from #27063. Closing this sandbox PR (do not merge). Please push back if any conclusion is off.

Final verdict — head d16481061d (latest upstream/main + version.py sentinel)

123 pass / 25 skip / 11 fail after rerunning the transient-504 batch.

🔴 Real main regressions — 2, both from #27063 ("[AMD] Optimize gpt-oss-120B performance", 1c73ff8ad3):

Both reported on #27063. Root cause: #27063's hidden_dim_unpadded = self.experts.hidden_size assumes that attribute is the experts' unpadded output width, but it's the padded width for the B200 MXFP4 FusedMoE (3072 vs 2880), and the LoRA wrapper (FusedMoEWithLoRA) doesn't expose it at all. pr-test-finish (Base) and pr-test-extra-finish are red only because of these two.

✅ Transient infra, fully resolved: 11 CUDA jobs initially died at install with Failed to download sglang-kernel==0.4.3+cu130 → 504 Gateway Timeout (GitHub-releases CDN incident). All 11 passed on rerun (including base-c-h20), confirming their suites are green on main and the 504 was purely transient.

🔵 Non-CUDA lanes (pre-existing, not gpt-oss-related, not chased):

• PR Test (Xeon) base-b-test-cpu: RuntimeError: decode: expect req_lens to be int64, got Int (test_external_models.py).
• PR Test (NPU) stage-b-test-1-npu-a2: self-hosted-runner/container failure.
• PR Test (AMD) stage-c mi325 + stage-c mi35x + stage-b mi35x-disaggregation (+ AMD/NPU finish aggregates).

Net: main CUDA is green except the 2 #27063 B200 regressions. Closing this sandbox PR — not merged.

🤖 Posted autonomously by Claude Code acting on the user's behalf. Triaged the first failure in this main-CI verification run; classified it as infra (dependency-install timeout, test never ran) and will rerun failed jobs after the round completes. Please push back if any conclusion is off.

CI triage: base-b-test-1-gpu-small (1) — infra, not a code failure

• Job log: https://github.com/sgl-project/sglang/actions/runs/27758414047/job/82126256382
• Failing step: Install dependencies
• Verbatim fingerprint:

  ##[error]The action 'Install dependencies' has timed out after 20 minutes.
  

Classification: infra / timeout. The failure is in the pip install setup step (20-minute cap), so the test suite never started — this is a runner/network flake, not a regression. CUDA test code on this shard did not execute.

Plan: Let the rest of the round finish (the bypass-fastfail label prevents this from cascade-cancelling siblings), then /rerun-failed-ci once all jobs settle.

🤖 Posted autonomously by Claude Code acting on the user's behalf. Triaged the AMD-lane failure in this main-CI verification run; classified it as a tip-of-main AMD issue unrelated to this sandbox diff (version.py comment only). Flagging for AMD owners. Please push back if any conclusion is off.

CI triage: stage-c-test-large-8-gpu-amd (..., 3) — AMD lane, unrelated to this PR

• Job log: https://github.com/sgl-project/sglang/actions/runs/27758414047/job/82126286136
• Failing test: test/registered/ops/test_aiter_allreduce_fusion_amd.py
• Verbatim fingerprint:

  FAILED: fused residual output diverges from unfused path for some shapes.
  torch.distributed.elastic.multiprocessing.errors.ChildFailedError
  /sglang-checkout/test/registered/ops/test_aiter_allreduce_fusion_amd.py FAILED (exit code 1)
  

Classification: non-CUDA (AMD mi325) failure on tip-of-main. This sandbox PR's only change is a comment in python/sglang/version.py, so the divergence cannot originate from this diff — it is a property of current main on the AMD aiter allreduce-fusion path (real AMD numerical regression or a tolerance flake). Per babysit policy, non-CUDA lanes are not chased here; flagging for AMD owners.

Plan: Continue watching the CUDA scope. This AMD job will be included in the end-of-round /rerun-failed-ci to check whether it reproduces (flake vs persistent).

🤖 Posted autonomously by Claude Code acting on the user's behalf. Triaged the NPU-lane failure in this main-CI verification run; classified it as a non-CUDA perf-threshold flake unrelated to this sandbox diff. Please push back if any conclusion is off.

CI triage: stage-b-test-1-npu-a2 (0) — NPU lane, perf-threshold flake, unrelated to this PR

• Failing test: test/registered/ascend/basic_function/quant/test_npu_w8a8_quantization.py
• Verbatim fingerprint:

  AssertionError: Output throughput of .../Qwen2.5-0.5B-Instruct-quantized.w8a8 is 487.6356529757384,
  is lower than 700  (not >= 700)
  

Classification: non-CUDA (Ascend NPU) perf-threshold flake. The test asserts a throughput floor (≥700 tok/s) and the NPU runner only hit 487 tok/s this round — a machine-load / perf-variance failure, not a correctness regression. The sandbox diff is a python/sglang/version.py comment only, so it cannot affect NPU throughput. Per babysit policy, non-CUDA lanes are not chased here.

Plan: Continue watching the CUDA scope; this NPU job will be covered by the end-of-round /rerun-failed-ci.

🤖 Posted autonomously by Claude Code acting on the user's behalf. Triaged a second CUDA-lane failure; it shares root cause with the earlier base-b-test-1-gpu-small (1) install-timeout (same infra issue), so noting it here rather than as a separate investigation. Please push back if any conclusion is off.

CI triage: extra-a-test-1-gpu-small (1) — same install-timeout infra as base-b-test-1-gpu-small (1)

• Job log: https://github.com/sgl-project/sglang/actions/runs/27758414047/job/82126226917
• Failing step: Install dependencies
• Verbatim fingerprint:

  ##[error]The action 'Install dependencies' has timed out after 20 minutes.
  

Classification: infra / timeout (recurring). Two 1-gpu-small shards (base-b ... (1) and now extra-a ... (1)) both died in the pip install setup at the 20-minute cap — the CUDA test code never ran. This is a runner/registry/network slowness pattern on the small-GPU pool, not a code regression.

Plan: Roll both into the end-of-round /rerun-failed-ci. If the install timeout reproduces on rerun, it's a persistent infra slowdown worth flagging to infra owners (not a main-branch code issue).

🤖 Posted autonomously by Claude Code acting on the user's behalf. Triaged the XPU-lane timeout in this main-CI verification run and noting that the finish gate red is just the aggregate of the already-triaged non-CUDA failures. Please push back if any conclusion is off.

CI triage: stage-b-test-1-gpu-xpu (timeout) + finish (aggregate gate)

• stage-b-test-1-gpu-xpu: test/registered/xpu/test_topk.py timed out after 1200s

  ✗ FAILED: /sglang-checkout/test/registered/xpu/test_topk.py (timeout after 1200s)
  

  Classification: non-CUDA (XPU) timeout. XPU lane, unrelated to this sandbox diff (version.py comment only). Per babysit policy, non-CUDA lanes are not chased.
• finish: the aggregate pr-test-*-finish gate — it goes red whenever any child job failed/cancelled. Here it simply reflects the already-triaged non-CUDA + install-timeout failures; not an independent signal.

Running tally of this round's failures (all non-CUDA-test):

1. base-b-test-1-gpu-small (1) — install-deps 20-min timeout (infra)
2. extra-a-test-1-gpu-small (1) — same install-deps timeout (infra)
3. stage-c-test-large-8-gpu-amd (3) — AMD aiter allreduce-fusion numerical divergence
4. stage-b-test-1-npu-a2 (0) — NPU throughput-threshold flake
5. stage-b-test-1-gpu-xpu — XPU test_topk.py 1200s timeout
6. finish — aggregate gate (reflects the above)

CUDA test lanes: 0 real failures. Plan unchanged: end-of-round /rerun-failed-ci.

🤖 Posted autonomously by Claude Code acting on the user's behalf. Logging a third install-dependency timeout sharing root cause with the prior two; this is now a clear systemic pattern on the 1-gpu-small pool, flagging for infra. Please push back if any conclusion is off.

CI triage: base-b-test-1-gpu-small (4) — same install-timeout pattern (3rd occurrence)

• Job log: https://github.com/sgl-project/sglang/actions/runs/27758414047/job/82126256400
• Failing step: Install dependencies

  ##[error]The action 'Install dependencies' has timed out after 20 minutes.
  

Classification: infra / timeout — systemic. Three 1-gpu-small shards this round (base-b (1), base-b (4), extra-a (1)) all died at the 20-minute pip install cap before any test ran. That's a persistent dependency-install slowdown on the small-GPU runner pool (registry/network), not a code issue — the sandbox diff is a version.py comment only. Worth raising with CI/infra owners if it keeps recurring.

CUDA test lanes still 0 real failures. Plan: end-of-round /rerun-failed-ci — expect the install-timeout shards to pass on a fresh runner.

🤖 Posted autonomously by Claude Code acting on the user's behalf. Main CI verification round on latest upstream/main (97e3b8998d) is essentially complete — 101 CUDA test jobs green, 0 real CUDA test failures. Triggering /rerun-failed-ci next to reconfirm the infra-timeout shards as flakes. Please push back if any conclusion is off.

Main CI verification summary (run 27758414047, head 97e3b8998d)

CUDA test lanes: 101 SUCCESS, 0 real test failures. (One last shard base-c-test-4-gpu-h100 (2) is still queued waiting for a scarce 4-GPU runner — not a failure.)

All 6 red jobs are non-test-code:

finish / pr-test-extra-finish / pr-test-npu-finish are aggregate gates reflecting the above; not independent signals. The 3 1-gpu-small install timeouts are a systemic dependency-install slowdown on the small-GPU runner pool this round (registry/network), unrelated to any code — this run's diff is a version.py comment only.

Re-running the failed jobs now to confirm the install timeouts pass on fresh runners.

/rerun-failed-ci

🤖 Posted autonomously by Claude Code acting on the user's behalf. First /rerun-failed-ci cleared the NPU/XPU/extra-a flakes (all green on retry, confirming transient infra) but did not re-trigger the two base-b-test-1-gpu-small shards; re-running once more to clear them. Please push back if any conclusion is off.

Rerun-1 outcome: stage-b-test-1-npu-a2, stage-b-test-1-gpu-xpu, and extra-a-test-1-gpu-small (1) all passed on retry — confirms the install-dependency timeout / NPU-throughput / XPU-topk reds were transient flakes. The base-b-test-1-gpu-small (1)/(4) shards (same install-deps 20-min timeout) weren't picked up by rerun-1, so triggering rerun-2 for them. AMD stage-c-test-large-8-gpu-amd (3) is non-CUDA and left as-is.

/rerun-failed-ci

🤖 Posted autonomously by Claude Code acting on the user's behalf. Final verdict for this main-CI verification of upstream/main 97e3b8998d, after two /rerun-failed-ci cycles. Please push back if any conclusion is off.

Final verdict: tip-of-main CUDA test suite is GREEN — no code regressions

122 CUDA test jobs SUCCESS, 0 real CUDA test failures across base-a/b/c + extra-a/b on 1/2/4/8-GPU h100/h200/b200/gb300.

All reds across the run + 2 reruns were non-test-code:

Conclusion: No code-level breakage on main. The one persistent CUDA-lane red (base-b (1)) is purely the Install dependencies step hitting its 20-minute cap on the small-GPU runner pool — a systemic dependency-install / registry slowness worth raising with CI/infra owners (3 distinct 1-gpu-small shards hit it this run; two cleared on a fresh runner, one keeps losing the dice). The NPU throughput flake and AMD aiter-fusion divergence are non-CUDA, pre-existing on tip-of-main.

Not rerunning base-b (1) again — it would just re-gamble against the same congested install pool, not exercise any new test coverage.

🤖 Posted autonomously by Claude Code acting on the user's behalf. This is a CI-status comment on the sandbox-verify-main-ci PR: I triaged the current failures on the in-flight run (head = latest upstream/main, no functional diff) and classified them. Please push back if any conclusion is off.

CI triage — run still in progress (rollup: 77 ✅ / 56 ⏳ / 42 skipped / 6 ❌)

This PR carries no functional diff (only a version.py sentinel comment), so every failure here reflects the state of tip-of-main, not a code change.

1. Four H200 CUDA failures — all the same root cause: OOM at CUDA context creation

All four die with the identical fingerprint during scheduler init — before any model weights are loaded:

File ".../srt/model_executor/model_runner.py", line 1168, in init_torch_distributed
    torch.get_device_module(self.device).set_device(self.gpu_id)
File ".../torch/cuda/__init__.py", line 638, in set_device
    torch._C._cuda_setDevice(device)
torch.AcceleratorError: CUDA error: out of memory
RuntimeError: Rank 0 scheduler died during initialization (exit code: -3)

Classification: infra (dirty/over-subscribed runner), not a main regression. An OOM at cudaSetDevice means the GPU is already full before this process touches it — i.e. leftover allocation from a prior test/process on the same 8-GPU H200 box, not model-size or a code bug (the model load hasn't even started). Four unrelated large-model tests OOMing at the same context-creation step on the same lane is the signature of a runner-pool cleanup issue.

2. One XPU failure (non-CUDA lane) — timeout, not chasing

• stage-b-test-1-gpu-xpu: test_intel_xpu_backend.py::test_attention_backend ✗ TIMEOUT after 1200s (the other 4 XPU files passed). The finish FAILURE cascades from this. Non-CUDA lane on a no-diff PR → noted, not chased.

Next step

Let the in-flight run finish exposing all failures, then /rerun-failed-ci to get a clean-runner data point on the four H200 jobs. If they pass on rerun → confirmed dirty-runner flake; if they OOM again at set_device → escalate (persistent runner contamination or a leak in an earlier test in the same shard).

🤖 Posted autonomously by Claude Code acting on the user's behalf. Update after more of the run completed: two new CUDA failures + 15 cancelled CUDA jobs all trace to the same dirty/over-subscribed GPU runner pool, not a main regression. Please push back if any conclusion is off.

Update — failures grew to 9 (rollup: 90 ✅ / 42 skipped / 27 ⏳ / 15 cancelled / 9 ❌)

Two new CUDA failures — same infra root cause

The B200 timeout has a smoking gun for the runner being dirty before the test even started:

Available memory 10.6138916015625GB is less than required memory 13.0GB for warmup,
reducing max_m to 8192 to avoid out of memory

A B200 has ~180 GB. Seeing only 10.6 GB free at warmup means ~170 GB was already occupied by something else before this test touched the GPU — the same leftover-allocation signature as the five H200 set_device OOMs.

15 CUDA jobs cancelled (not real failures)

base-b-test-1-gpu-small, base-b-test-2-gpu-large, base-c-test-4-gpu-b200, base-c-test-4-gpu-gb300, base-c-test-4-gpu-h100, base-c-test-8-gpu-h200 shards were CANCELLED. Sampled one — it ran ~15 min then ended with ##[error]The operation was canceled., i.e. GitHub-level cancellation, not a test error.

Verdict so far

This run is compromised by a CUDA runner-pool infra incident — leftover GPU memory across H200/B200 boxes (5× set_device OOM, 1× B200 with 10.6 GB free → timeout) plus 15 cancellations. None of it points at main code: every OOM is at CUDA context creation, before any model weights load. It is not a verification that main is broken, and not a verification that it is green either — the GPU lanes never got a clean shot.

Next step

Wait for the in-flight run to finish (2 CUDA shards still pending), then /rerun-failed-ci to re-run the failed+cancelled CUDA jobs on (hopefully) clean runners. If they pass → confirmed dirty-runner flake, main clean. If they OOM again at set_device → persistent runner-pool contamination; will escalate to @fzyzcjy rather than keep burning reruns.

🤖 Posted autonomously by Claude Code acting on the user's behalf. Final state of the first run + the recovery action I'm taking. I force-cancelled a stuck sub-workflow and am about to trigger a rerun of the failed jobs; reasoning below. Please push back if any conclusion is off.

First run: verdict = CUDA runner-pool infra incident (not a main regression)

Final tally of the original run: 6 CUDA test-job failures, all the same root cause, plus runner starvation:

Plus 15 CUDA jobs CANCELLED (The operation was canceled) and 2 extra-b CUDA shards stuck queued for 2h45m with no runner ever assigned (extra-b-test-4-gpu-h100 (1), extra-b-test-4-gpu-b200 (1)).

Every failure is at CUDA context creation, before any model weights load — the GPU was already full/occupied when the test process started. Combined with the runner starvation, this is an infra incident on the H200/B200 pool, not a code problem on main. (Non-CUDA stage-b-test-1-gpu-xpu timeout and stage-b-test-1-gpu-small-amd-nondeterministic are separate non-CUDA lanes, not chased.)

Action taken

• Force-cancelled the stuck PR Test Extra run (28140040929) so it reached a terminal state (the 2 ghost-queued shards were never going to get a runner).
• Now triggering /rerun-failed-ci to re-run the failed + cancelled CUDA jobs on hopefully-clean runners.

Escalation criterion: if the rerun passes → confirmed transient dirty-runner flake, main clean. If it OOMs again at set_device → persistent runner-pool contamination; I'll stop burning reruns and flag it to @fzyzcjy to decide (infra fix vs. retry-later) rather than loop.

/rerun-failed-ci

🤖 Posted autonomously by Claude Code acting on the user's behalf. Rerun results so far confirm the base-c H200 OOMs were a dirty-runner flake (not a main regression); I've now also re-triggered the extra-b jobs that the first /rerun-failed-ci left untouched. Please push back if any conclusion is off.

Rerun update — base-c OOMs confirmed as transient dirty-runner flake

/rerun-failed-ci re-ran the PR Test Base jobs. After ~1h of runner-queue wait, the base-c reruns are landing green, including the exact lanes that OOM'd at set_device in the first run:

Zero base-c reruns have failed. The H200 set_device OOMs reproduced only on the first run's dirty/over-subscribed runners and clear on clean ones → confirmed infra flake, main is clean for these tests. (Two 8-gpu-h200 glm5 shards are still finishing; will confirm when they land.)

Extra-b jobs re-triggered

The first /rerun-failed-ci only re-ran PR Test Base (the PR Test Extra run had been force-cancelled, so the bot didn't pick it up). I've now re-run its failed jobs directly (gh run rerun <extra-run> --failed), covering the 3 extra-b CUDA failures:

• extra-b-test-8-gpu-h200 (1) — test_dsa_glm5_hisparse.py (OOM @ set_device)
• extra-b-test-deepep-8-gpu-h200 (0) — test_deepseek_v32_cp_single_node.py (OOM @ set_device)
• extra-b-test-deepep-4-gpu-b200 (0) — test_deepseek_v4_flash_fp4_b200_cp.py (timeout / 10.6 GB free)

Expectation given the base-c evidence: these too were dirty-runner flakes and should pass on clean runners. Still watching; will post the final base-c + extra-b verdict once both rerun cycles complete.

🤖 Posted autonomously by Claude Code acting on the user's behalf. All 6 original CUDA failures now pass on rerun; flagging one new, different infra failure that surfaced during the rerun and re-triggering it. Please push back if any conclusion is off.

All 6 original CUDA failures → confirmed flake (pass on clean runners)

Every one of the first run's CUDA failures passed when re-run on clean runners — including the B200 one:

• base-c-test-8-gpu-h200 (0) (test_pcg_glm5_fp8_tp8.py) ✅
• base-c-test-8-gpu-h200 (1) (test_dsa_glm5_tp_mtp.py) ✅
• base-c-test-deepep-8-gpu-h200 (test_disaggregation_dsv4.py) ✅
• extra-b-test-8-gpu-h200 (1) (test_dsa_glm5_hisparse.py) ✅
• extra-b-test-deepep-8-gpu-h200 (test_deepseek_v32_cp_single_node.py) ✅
• extra-b-test-deepep-4-gpu-b200 (0) (test_deepseek_v4_flash_fp4_b200_cp.py) ✅

→ The H200/B200 set_device OOMs and the B200 timeout were all dirty/over-subscribed-runner infra flakes, not main regressions.

One new (different) failure surfaced on rerun — also infra, re-triggered

extra-b-test-4-gpu-b200 (1) (test_flashinfer_a2a_cutedsl_v2.py) — this shard was CANCELLED in run 1, and on rerun it failed with a container-capability error (not OOM, not main code):

RuntimeError: pidfd_getfd(pidfd=184, fd=198) failed with errno 1: Operation not permitted.
Permission denied. If running in a container, try adding --cap-add=SYS_PTRACE to your docker run command.

The test uses cross-process fd sharing (pidfd_getfd) which needs SYS_PTRACE; this B200 runner's container didn't grant it. Shard (0) of the same test passed on the same config → a per-runner container-config flake, not a code issue. Re-triggered it (gh run rerun --failed); 3 base-c-test-4-gpu-b200 shards are still draining the slow B200 queue.

Net: no main regression found — every failure this run has been infra (dirty-runner OOM, runner starvation, or container caps). Will post the final summary once the last B200 reruns land.