[None][infra] Handle sacct error when checking slurm job status by yuanjingx87 · Pull Request #14367 · NVIDIA/TensorRT-LLM

yuanjingx87 · 2026-05-20T17:13:52Z

Summary by CodeRabbit

Improvements
- Enhanced test execution robustness by adding transient failure tolerance to job status tracking. The test runner now retries on temporary infrastructure failures instead of immediately failing, reducing false test failures and improving overall test reliability.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

yuanjingx87 · 2026-05-20T17:20:07Z

/bot run --stage-list "GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-1"

tensorrt-cicd · 2026-05-20T17:30:48Z

PR_Github #49462 [ run ] triggered by Bot. Commit: a98a776 Link to invocation

yuanjingx87 · 2026-05-20T18:45:40Z

/bot run --stage-list "GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-1"

tensorrt-cicd · 2026-05-20T18:54:24Z

PR_Github #49471 [ run ] triggered by Bot. Commit: 709b0ed Link to invocation

tensorrt-cicd · 2026-05-20T18:59:45Z

PR_Github #49462 [ run ] completed with state ABORTED. Commit: a98a776

Link to invocation

yuanjingx87 · 2026-05-20T19:23:52Z

/bot run --stage-list "GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-1"

tensorrt-cicd · 2026-05-20T19:28:46Z

PR_Github #49471 [ run ] completed with state FAILURE. Commit: 709b0ed
/LLM/main/L0_MergeRequest_PR pipeline #39112 (Partly Tested) completed with status: 'ABORTED'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

tensorrt-cicd · 2026-05-20T19:30:08Z

PR_Github #49474 [ run ] triggered by Bot. Commit: 73a7c8b Link to invocation

Signed-off-by: Yuanjing Xue <[email protected]>

yuanjingx87 · 2026-05-20T20:53:55Z

/bot run --stage-list "GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-1"

tensorrt-cicd · 2026-05-20T20:59:35Z

PR_Github #49488 [ run ] triggered by Bot. Commit: 3ec344c Link to invocation

tensorrt-cicd · 2026-05-20T21:03:37Z

PR_Github #49474 [ run ] completed with state ABORTED. Commit: 73a7c8b

Link to invocation

Signed-off-by: Yuanjing Xue <[email protected]>

yuanjingx87 · 2026-05-20T22:41:07Z

/bot run --stage-list "GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-1"

yuanjingx87 · 2026-05-20T22:42:10Z

/bot run --stage-list "DGX_B200-PyTorch-1"

tensorrt-cicd · 2026-05-20T22:48:21Z

PR_Github #49501 [ run ] triggered by Bot. Commit: a0188c5 Link to invocation

tensorrt-cicd · 2026-05-20T22:48:25Z

PR_Github #49500 [ ] completed with state ABORTED. Commit: a0188c5

Link to invocation

tensorrt-cicd · 2026-05-20T22:52:11Z

PR_Github #49488 [ run ] completed with state ABORTED. Commit: 3ec344c

Link to invocation

coderabbitai · 2026-05-20T23:23:54Z

📝 Walkthrough

Walkthrough

The runLLMTestlistWithSbatch function's Slurm job status polling loop now handles transient sacct command failures by logging a warning and retrying after 60 seconds instead of failing or continuing with invalid state.

Changes

Slurm job status polling resilience

Layer / File(s)	Summary
sacct failure handling in Slurm tracking loop `jenkins/L0_Test.groovy`	When `sacct` exits non-zero during Slurm status polling, the script logs a warning message, sleeps 60 seconds, and continues the polling loop without altering downstream state logic.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The PR description lacks substantive content in all key sections: no actual description of the issue or solution, no test coverage documentation, and the PR checklist is completed without verification of required items.	Author should provide: (1) a clear explanation of the sacct error issue and why error-handling is needed, (2) specific test cases that verify the new error-handling behavior, and (3) verification that each checklist item has been genuinely addressed.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes the main change: handling sacct errors when checking Slurm job status, which directly relates to the code modification.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

tensorrt-cicd · 2026-05-21T01:10:06Z

PR_Github #49501 [ run ] completed with state SUCCESS. Commit: a0188c5
/LLM/main/L0_MergeRequest_PR pipeline #39138 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

yuanjingx87 · 2026-05-21T05:44:43Z

/bot skip --comment "Single sbatch slurm run is sufficient"

tensorrt-cicd · 2026-05-21T05:50:21Z

PR_Github #49599 [ skip ] triggered by Bot. Commit: a0188c5 Link to invocation

tensorrt-cicd · 2026-05-21T05:56:25Z

PR_Github #49599 [ skip ] completed with state SUCCESS. Commit: a0188c5
Skipping testing for commit a0188c5

Link to invocation

…IA#14367) Signed-off-by: Yuanjing Xue <[email protected]>

yuanjingx87 requested review from a team as code owners May 20, 2026 17:13

yuanjingx87 requested review from EmmaQiaoCh and zeroepoch May 20, 2026 17:13

yuanjingx87 marked this pull request as draft May 20, 2026 17:13

github-actions Bot assigned yuanjingx87 May 20, 2026

yuanjingx87 force-pushed the user/yuanjingx/handle_sacct_error branch 2 times, most recently from f017770 to 709b0ed Compare May 20, 2026 18:45

yuanjingx87 force-pushed the user/yuanjingx/handle_sacct_error branch from 709b0ed to 73a7c8b Compare May 20, 2026 19:23

Handle sacct exception

3ec344c

Signed-off-by: Yuanjing Xue <[email protected]>

yuanjingx87 force-pushed the user/yuanjingx/handle_sacct_error branch from 73a7c8b to 3ec344c Compare May 20, 2026 20:52

zeroepoch approved these changes May 20, 2026

View reviewed changes

Revert change for verifying

a0188c5

Signed-off-by: Yuanjing Xue <[email protected]>

yuanjingx87 marked this pull request as ready for review May 20, 2026 23:20

EmmaQiaoCh approved these changes May 21, 2026

View reviewed changes

yuanjingx87 merged commit c7d609b into NVIDIA:main May 21, 2026
12 of 14 checks passed

xxi-nv pushed a commit to xxi-nv/TensorRT-LLM that referenced this pull request May 22, 2026

[None][infra] Handle sacct error when checking slurm job status (NVID…

6c361f4

…IA#14367) Signed-off-by: Yuanjing Xue <[email protected]>

bmarimuthu-nv pushed a commit to nv-auto-deploy/TensorRT-LLM that referenced this pull request May 28, 2026

[None][infra] Handle sacct error when checking slurm job status (NVID…

2a62249

…IA#14367) Signed-off-by: Yuanjing Xue <[email protected]>

Conversation

yuanjingx87 commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

yuanjingx87 commented May 20, 2026

Uh oh!

tensorrt-cicd commented May 20, 2026

Uh oh!

yuanjingx87 commented May 20, 2026

Uh oh!

tensorrt-cicd commented May 20, 2026

Uh oh!

tensorrt-cicd commented May 20, 2026

Uh oh!

yuanjingx87 commented May 20, 2026

Uh oh!

tensorrt-cicd commented May 20, 2026

Uh oh!

tensorrt-cicd commented May 20, 2026

Uh oh!

yuanjingx87 commented May 20, 2026

Uh oh!

tensorrt-cicd commented May 20, 2026

Uh oh!

tensorrt-cicd commented May 20, 2026

Uh oh!

yuanjingx87 commented May 20, 2026

Uh oh!

yuanjingx87 commented May 20, 2026

Uh oh!

tensorrt-cicd commented May 20, 2026

Uh oh!

tensorrt-cicd commented May 20, 2026

Uh oh!

tensorrt-cicd commented May 20, 2026

Uh oh!

coderabbitai Bot commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

❌ Failed checks (1 warning)

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

yuanjingx87 commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yuanjingx87 commented May 20, 2026 •

edited

Loading

coderabbitai Bot commented May 20, 2026 •

edited

Loading