[https://nvbugs/6076767][fix] Add barrier before warmup to prevent PP hang with guided decoding by ziyixiong-nv · Pull Request #13132 · NVIDIA/TensorRT-LLM

ziyixiong-nv · 2026-04-16T19:55:27Z

Summary

Fix for NVBugs 6076767: [TensorRT-LLM][release/1.2.1]: accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding_4gpus[llguidance] stably hang
Root cause: Add barrier before warmup to prevent PP hang with guided decoding
Fix: (auto-detected from git commit)
Automated fix generated by repair-bot

Test plan

Verify fix on the same GPU type as the original failure
Check for regressions in related tests

Links

Bug: https://nvbugs/6076767

Summary by CodeRabbit

Chores
- Added global synchronization during initialization for pipeline-parallel execution stages to ensure coordinated startup behavior across all ranks.

coderabbitai · 2026-04-16T20:02:51Z

📝 Walkthrough

Walkthrough

Added a distributed barrier synchronization in PyExecutor.__init__ before warmup initialization. All pipeline-parallel ranks now wait at this point during initialization to ensure synchronized warmup startup across all stages.

Changes

Cohort / File(s)	Summary
Synchronization Coordination `tensorrt_llm/_torch/pyexecutor/py_executor.py`	Added `self.dist.barrier()` call immediately before entering warmup phase to enforce all-rank synchronization in pipeline-parallel execution.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description covers the Summary, Test Coverage sections well and includes links, but lacks specific details in some sections and omits verification of PR checklist items.
Title check	✅ Passed	The title clearly and specifically identifies the main change: adding a barrier before warmup to prevent pipeline-parallel hangs with guided decoding, which directly addresses the root cause described in the PR objectives.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

ziyixiong-nv · 2026-04-17T00:14:14Z

/bot run

tensorrt-cicd · 2026-04-17T00:20:40Z

PR_Github #43855 [ run ] triggered by Bot. Commit: f250fef Link to invocation

tensorrt-cicd · 2026-04-17T01:22:21Z

PR_Github #43855 [ run ] completed with state FAILURE. Commit: f250fef
/LLM/main/L0_MergeRequest_PR pipeline #34312 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

ziyixiong-nv · 2026-04-17T08:01:04Z

/bot run

ziyixiong-nv · 2026-04-20T02:23:47Z

/bot run

tensorrt-cicd · 2026-04-20T02:29:53Z

PR_Github #44257 [ run ] triggered by Bot. Commit: 03e073c Link to invocation

tensorrt-cicd · 2026-04-20T03:30:42Z

PR_Github #44257 [ run ] completed with state SUCCESS. Commit: 03e073c
/LLM/main/L0_MergeRequest_PR pipeline #34678 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

ziyixiong-nv · 2026-04-20T05:57:27Z

/bot run

tensorrt-cicd · 2026-04-20T06:03:32Z

PR_Github #44340 [ run ] triggered by Bot. Commit: 03e073c Link to invocation

tensorrt-cicd · 2026-04-20T06:13:57Z

PR_Github #44340 [ run ] completed with state FAILURE. Commit: 03e073c
/LLM/main/L0_MergeRequest_PR pipeline #34757 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

ziyixiong-nv · 2026-04-28T00:21:11Z

/bot run

tensorrt-cicd · 2026-04-28T00:27:41Z

PR_Github #45804 [ run ] triggered by Bot. Commit: d94ffb1 Link to invocation

tensorrt-cicd · 2026-04-28T07:20:33Z

PR_Github #45804 [ run ] completed with state SUCCESS. Commit: d94ffb1
/LLM/main/L0_MergeRequest_PR pipeline #35994 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

ziyixiong-nv · 2026-04-28T07:23:20Z

/bot run

tensorrt-cicd · 2026-04-28T07:29:10Z

PR_Github #45891 [ run ] triggered by Bot. Commit: d94ffb1 Link to invocation

tensorrt-cicd · 2026-04-28T11:05:57Z

PR_Github #45891 [ run ] completed with state FAILURE. Commit: d94ffb1
/LLM/main/L0_MergeRequest_PR pipeline #36059 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

ziyixiong-nv · 2026-04-29T00:23:43Z

/bot run

tensorrt-cicd · 2026-04-29T00:32:28Z

PR_Github #46002 [ run ] triggered by Bot. Commit: ab66d5a Link to invocation

tensorrt-cicd · 2026-04-29T01:49:01Z

PR_Github #46002 [ run ] completed with state FAILURE. Commit: ab66d5a
/LLM/main/L0_MergeRequest_PR pipeline #36151 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

ziyixiong-nv · 2026-04-29T02:20:03Z

/bot run

tensorrt-cicd · 2026-04-29T02:27:01Z

PR_Github #46028 [ run ] triggered by Bot. Commit: ab66d5a Link to invocation

tensorrt-cicd · 2026-05-09T15:38:54Z

PR_Github #47511 [ run ] completed with state SUCCESS. Commit: a4853e4
/LLM/main/L0_MergeRequest_PR pipeline #37429 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

ziyixiong-nv · 2026-05-10T07:11:57Z

/bot run

tensorrt-cicd · 2026-05-10T07:18:05Z

PR_Github #47571 [ run ] triggered by Bot. Commit: 3f15170 Link to invocation

tensorrt-cicd · 2026-05-10T12:14:58Z

PR_Github #47571 [ run ] completed with state SUCCESS. Commit: 3f15170
/LLM/main/L0_MergeRequest_PR pipeline #37482 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

ziyixiong-nv · 2026-05-10T12:51:44Z

/bot run

tensorrt-cicd · 2026-05-10T12:58:02Z

PR_Github #47595 [ run ] triggered by Bot. Commit: 3f15170 Link to invocation

tensorrt-cicd · 2026-05-10T15:11:01Z

PR_Github #47595 [ run ] completed with state SUCCESS. Commit: 3f15170
/LLM/main/L0_MergeRequest_PR pipeline #37503 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

ziyixiong-nv · 2026-05-11T00:01:41Z

/bot run

tensorrt-cicd · 2026-05-11T00:07:02Z

PR_Github #47625 [ run ] triggered by Bot. Commit: 6680107 Link to invocation

tensorrt-cicd · 2026-05-11T03:46:20Z

PR_Github #47625 [ run ] completed with state FAILURE. Commit: 6680107
/LLM/main/L0_MergeRequest_PR pipeline #37530 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

ziyixiong-nv · 2026-05-11T04:37:27Z

/bot run

tensorrt-cicd · 2026-05-11T04:42:39Z

PR_Github #47660 [ run ] triggered by Bot. Commit: 6680107 Link to invocation

tensorrt-cicd · 2026-05-11T15:09:20Z

PR_Github #47660 [ run ] completed with state SUCCESS. Commit: 6680107
/LLM/main/L0_MergeRequest_PR pipeline #37562 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

ziyixiong-nv · 2026-05-12T06:33:14Z

/bot run

tensorrt-cicd · 2026-05-12T06:38:42Z

PR_Github #47912 [ run ] triggered by Bot. Commit: 664b783 Link to invocation

tensorrt-cicd · 2026-05-12T08:55:25Z

PR_Github #47912 [ run ] completed with state SUCCESS. Commit: 664b783
/LLM/main/L0_MergeRequest_PR pipeline #37759 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…th guided decoding When pipeline parallelism is used with guided decoding (llguidance backend), ranks on the last PP stage can be delayed by the LLGuidanceMatcherFactory / LLTokenizer initialization while earlier PP stages skip guided-decoder creation entirely. Without synchronization, the earlier stages enter warmup forward passes that issue pp_send operations expecting matching pp_recv on the later stages — but those stages have not entered warmup yet, causing a permanent NCCL communication deadlock. Add a dist.barrier() call in PyExecutor.__init__ immediately before the warmup phase so that all ranks are synchronized before any PP communication begins. Signed-off-by: Ziyi Xiong <[email protected]>

ziyixiong-nv · 2026-05-12T23:28:35Z

/bot run

tensorrt-cicd · 2026-05-12T23:34:02Z

PR_Github #48045 [ run ] triggered by Bot. Commit: ab6eb2d Link to invocation

tensorrt-cicd · 2026-05-13T05:52:34Z

PR_Github #48045 [ run ] completed with state SUCCESS. Commit: ab6eb2d
/LLM/main/L0_MergeRequest_PR pipeline #37879 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

ziyixiong-nv · 2026-05-13T05:54:48Z

/bot run

tensorrt-cicd · 2026-05-13T06:00:18Z

PR_Github #48114 [ run ] triggered by Bot. Commit: ab6eb2d Link to invocation

tensorrt-cicd · 2026-05-13T10:01:33Z

PR_Github #48114 [ run ] completed with state SUCCESS. Commit: ab6eb2d
/LLM/main/L0_MergeRequest_PR pipeline #37941 completed with status: 'SUCCESS'

CI Report

Link to invocation

ziyixiong-nv requested a review from a team as a code owner April 16, 2026 19:55

ziyixiong-nv requested a review from lfr-0531 April 16, 2026 19:55

github-actions Bot assigned ziyixiong-nv Apr 16, 2026

ziyixiong-nv force-pushed the repair-bot-bug6076767 branch from f250fef to abaa191 Compare April 17, 2026 08:00

ziyixiong-nv force-pushed the repair-bot-bug6076767 branch from abaa191 to 03e073c Compare April 20, 2026 02:23

ziyixiong-nv force-pushed the repair-bot-bug6076767 branch from 03e073c to d94ffb1 Compare April 28, 2026 00:20

ziyixiong-nv force-pushed the repair-bot-bug6076767 branch from d94ffb1 to ab66d5a Compare April 29, 2026 00:23

ziyixiong-nv force-pushed the repair-bot-bug6076767 branch from a4853e4 to 3f15170 Compare May 10, 2026 07:11

ziyixiong-nv force-pushed the repair-bot-bug6076767 branch from 3f15170 to 6680107 Compare May 11, 2026 00:01

ziyixiong-nv force-pushed the repair-bot-bug6076767 branch from 6680107 to 664b783 Compare May 12, 2026 06:33

ziyixiong-nv force-pushed the repair-bot-bug6076767 branch from 664b783 to ab6eb2d Compare May 12, 2026 23:28

ziyixiong-nv enabled auto-merge (squash) May 13, 2026 11:38

yuxianq approved these changes May 14, 2026

View reviewed changes

ziyixiong-nv merged commit 54c5c4b into NVIDIA:main May 14, 2026
7 checks passed

Conversation

ziyixiong-nv commented Apr 16, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Links

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

❌ Failed checks (1 warning)

Uh oh!

ziyixiong-nv commented Apr 17, 2026

Uh oh!

tensorrt-cicd commented Apr 17, 2026

Uh oh!

tensorrt-cicd commented Apr 17, 2026

Uh oh!

ziyixiong-nv commented Apr 17, 2026

Uh oh!

ziyixiong-nv commented Apr 20, 2026

Uh oh!

tensorrt-cicd commented Apr 20, 2026

Uh oh!

tensorrt-cicd commented Apr 20, 2026

Uh oh!

ziyixiong-nv commented Apr 20, 2026

Uh oh!

tensorrt-cicd commented Apr 20, 2026

Uh oh!

tensorrt-cicd commented Apr 20, 2026

Uh oh!

ziyixiong-nv commented Apr 28, 2026

Uh oh!

tensorrt-cicd commented Apr 28, 2026

Uh oh!

tensorrt-cicd commented Apr 28, 2026

Uh oh!

ziyixiong-nv commented Apr 28, 2026

Uh oh!

tensorrt-cicd commented Apr 28, 2026

Uh oh!

tensorrt-cicd commented Apr 28, 2026

Uh oh!

ziyixiong-nv commented Apr 29, 2026

Uh oh!

tensorrt-cicd commented Apr 29, 2026

Uh oh!

tensorrt-cicd commented Apr 29, 2026

Uh oh!

ziyixiong-nv commented Apr 29, 2026

Uh oh!

tensorrt-cicd commented Apr 29, 2026

Uh oh!

tensorrt-cicd commented May 9, 2026

Uh oh!

ziyixiong-nv commented May 10, 2026

Uh oh!

tensorrt-cicd commented May 10, 2026

Uh oh!

tensorrt-cicd commented May 10, 2026

Uh oh!

ziyixiong-nv commented May 10, 2026

Uh oh!

tensorrt-cicd commented May 10, 2026

Uh oh!

tensorrt-cicd commented May 10, 2026

Uh oh!

ziyixiong-nv commented May 11, 2026

Uh oh!

tensorrt-cicd commented May 11, 2026

Uh oh!

tensorrt-cicd commented May 11, 2026

Uh oh!

ziyixiong-nv commented May 11, 2026

Uh oh!

tensorrt-cicd commented May 11, 2026

ziyixiong-nv commented Apr 16, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Apr 16, 2026 •

edited

Loading