[TRTLLM-10076][feat] Serve CLI improvements: renames, new flags, and mm_embedding_serve enhancements by JunyiXu-nv · Pull Request #12105 · NVIDIA/TensorRT-LLM

JunyiXu-nv · 2026-03-11T07:45:09Z

TRTLLM-10076: Update --tokenizer description for PyTorch backend, add --hf_revision alias for --revision with deprecation warning, support hf_revision key in YAML config, add --enable_attention_dp flag
TRTLLM-10079: mm_embedding_serve: add --config alias for --extra_encoder_options, expose --hf_revision, --free_gpu_memory_fraction, --tensor_parallel_size
TRTLLM-10229: Add --config alias for --config_file in disaggregated and disaggregated_mpi_worker commands
TRTLLM-10078: Improve --server_role help message with role descriptions

Made-with: Cursor

Summary by CodeRabbit

Release Notes

New Features
- Added --enable_attention_dp flag to the serve command for distributed attention processing.
Deprecations
- Use --hf_revision instead of --revision.
- Use --config instead of --extra_encoder_options and --config_file.
- Deprecation warnings will guide migration to new parameter names.
Improvements
- Enhanced help documentation for tokenizer and server role configuration.
- Extended serve_encoder with additional parameters: revision, free_gpu_memory_fraction, and tensor_parallel_size.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

coderabbitai · 2026-03-11T07:48:55Z

📝 Walkthrough

Walkthrough

The changes add support for a new enable_attention_dp feature flag through CLI argument plumbing in the serve command and related functions. Additionally, deprecation warnings are introduced for legacy options (--revision, --extra_encoder_options, --config_file), and new parameters are added to serve_encoder. A preprocessing step maps hf_revision to revision in llm_args handling.

Changes

Cohort / File(s)	Summary
CLI and Argument Plumbing `tensorrt_llm/commands/serve.py`	Added `--enable_attention_dp` CLI option to serve command; extended serve_encoder signature with revision, free_gpu_memory_fraction, and tensor_parallel_size parameters; introduced deprecation warnings for legacy flags (--revision → --hf_revision, --extra_encoder_options → --config, --config_file → --config); enhanced help texts for tokenizer and server role clarification; propagated enable_attention_dp through _serve_llm and launch_server paths (+84/-24).
Argument Preprocessing `tensorrt_llm/llmapi/llm_args.py`	Added preprocessing step to map `hf_revision` to `revision` in llm_args_dict before downstream processing using pop/setdefault (+3/-0).

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	⚠️ Warning	The PR description lists specific JIRA tickets and changes but lacks proper PR title format, missing sections for Description, Test Coverage details, and incomplete PR Checklist verification.	Add a properly formatted PR title following [JIRA][type] format, provide detailed Description and Test Coverage sections, and verify all PR Checklist items are addressed.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main changes: adding new CLI flags (enable_attention_dp, hf_revision), improving rename/alias support, and enhancing mm_embedding_serve.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tensorrt_llm/commands/serve.py`:
- Around line 1122-1127: The disaggregated_mpi_worker entrypoint (function
disaggregated_mpi_worker) should mirror the deprecation behavior of
disaggregated by detecting if "--config_file" was passed and emitting a
DeprecationWarning; add a check using sys.argv to see if "--config_file" is
present and call warnings.warn(..., DeprecationWarning, stacklevel=2)
immediately after the disaggregated_mpi_worker docstring/entry log so users see
the same deprecation message as the disaggregated command.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: c1f1f5fa-f1ab-4275-9a8a-9c393cfda74f

📥 Commits

Reviewing files that changed from the base of the PR and between f7255e0 and b398e02.

📒 Files selected for processing (2)

tensorrt_llm/commands/serve.py
tensorrt_llm/llmapi/llm_args.py

JunyiXu-nv · 2026-03-11T08:15:09Z

/bot run

tensorrt-cicd · 2026-03-11T08:21:07Z

PR_Github #38560 [ run ] triggered by Bot. Commit: 1386e53 Link to invocation

tensorrt-cicd · 2026-03-11T11:19:06Z

PR_Github #38560 [ run ] completed with state SUCCESS. Commit: 1386e53
/LLM/main/L0_MergeRequest_PR pipeline #29902 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

JunyiXu-nv · 2026-03-11T12:17:53Z

/bot run

tensorrt-cicd · 2026-03-11T12:24:24Z

PR_Github #38585 [ run ] triggered by Bot. Commit: 1386e53 Link to invocation

tensorrt-cicd · 2026-03-11T16:19:13Z

PR_Github #38585 [ run ] completed with state SUCCESS. Commit: 1386e53
/LLM/main/L0_MergeRequest_PR pipeline #29921 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

QiJune

LGTM

JunyiXu-nv · 2026-03-12T02:05:18Z

/bot run

tensorrt-cicd · 2026-03-12T02:14:58Z

PR_Github #38651 [ run ] triggered by Bot. Commit: 1386e53 Link to invocation

tensorrt-cicd · 2026-03-12T03:59:09Z

PR_Github #38651 [ run ] completed with state SUCCESS. Commit: 1386e53
/LLM/main/L0_MergeRequest_PR pipeline #29979 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

JunyiXu-nv · 2026-03-12T11:21:44Z

/bot run

tensorrt-cicd · 2026-03-12T11:28:20Z

PR_Github #38719 [ run ] triggered by Bot. Commit: 1386e53 Link to invocation

tensorrt-cicd · 2026-03-12T11:51:54Z

PR_Github #38719 [ run ] completed with state FAILURE. Commit: 1386e53
/LLM/main/L0_MergeRequest_PR pipeline #30037 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

JunyiXu-nv · 2026-03-12T12:47:38Z

/bot run

tensorrt-cicd · 2026-03-12T12:53:34Z

PR_Github #38732 [ run ] triggered by Bot. Commit: 1386e53 Link to invocation

tensorrt-cicd · 2026-03-12T16:05:33Z

PR_Github #38732 [ run ] completed with state SUCCESS. Commit: 1386e53
/LLM/main/L0_MergeRequest_PR pipeline #30050 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

JunyiXu-nv · 2026-03-17T02:33:24Z

/bot run

JunyiXu-nv · 2026-03-17T06:10:26Z

/bot run

tensorrt-cicd · 2026-03-17T06:19:12Z

PR_Github #39202 [ run ] triggered by Bot. Commit: 26af669 Link to invocation

tensorrt-cicd · 2026-03-17T10:36:03Z

PR_Github #39202 [ run ] completed with state SUCCESS. Commit: 26af669
/LLM/main/L0_MergeRequest_PR pipeline #30453 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

JunyiXu-nv · 2026-03-18T02:15:20Z

/bot run

tensorrt-cicd · 2026-03-18T02:21:01Z

PR_Github #39358 [ run ] triggered by Bot. Commit: 26af669 Link to invocation

tensorrt-cicd · 2026-03-18T03:09:23Z

PR_Github #39358 [ run ] completed with state SUCCESS. Commit: 26af669
/LLM/main/L0_MergeRequest_PR pipeline #30602 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

JunyiXu-nv · 2026-03-18T03:13:18Z

/bot run

tensorrt-cicd · 2026-03-18T03:19:56Z

PR_Github #39373 [ run ] triggered by Bot. Commit: fa0bba4 Link to invocation

tensorrt-cicd · 2026-03-18T10:46:44Z

PR_Github #39373 [ run ] completed with state SUCCESS. Commit: fa0bba4
/LLM/main/L0_MergeRequest_PR pipeline #30614 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

… improvements: renames, new flags, and mm_embedding_serve enhancements - TRTLLM-10076: Update --tokenizer description for PyTorch backend, add --hf_revision alias for --revision with deprecation warning, support hf_revision key in YAML config, add --enable_attention_dp flag - TRTLLM-10079: mm_embedding_serve: add --config alias for --extra_encoder_options, expose --hf_revision, --free_gpu_memory_fraction, --tensor_parallel_size - TRTLLM-10229: Add --config alias for --config_file in disaggregated and disaggregated_mpi_worker commands - TRTLLM-10078: Improve --server_role help message with role descriptions Signed-off-by: Junyi Xu <[email protected]> Made-with: Cursor Signed-off-by: Junyi Xu <[email protected]> Made-with: Cursor Signed-off-by: Junyi Xu <[email protected]> Made-with: Cursor

JunyiXu-nv · 2026-03-19T06:21:43Z

/bot help

github-actions · 2026-03-19T06:21:51Z

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental) --high-priority]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

--high-priority (OPTIONAL) : Run the pipeline with high priority. This option is restricted to authorized users only and will route the job to a high-priority queue.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

JunyiXu-nv · 2026-03-19T06:22:28Z

/bot reuse-pipeline

tensorrt-cicd · 2026-03-19T06:29:00Z

PR_Github #39553 [ reuse-pipeline ] triggered by Bot. Commit: 62c3050 Link to invocation

tensorrt-cicd · 2026-03-19T06:37:00Z

PR_Github #39553 [ reuse-pipeline ] completed with state SUCCESS. Commit: 62c3050
Reusing PR_Github #39373 for commit 62c3050

Link to invocation

…mm_embedding_serve enhancements (NVIDIA#12105) Signed-off-by: Junyi Xu <[email protected]>

JunyiXu-nv requested a review from a team as a code owner March 11, 2026 07:45

JunyiXu-nv requested a review from Superjomn March 11, 2026 07:45

github-actions Bot assigned JunyiXu-nv Mar 11, 2026

JunyiXu-nv requested review from QiJune and arysef March 11, 2026 07:45

coderabbitai Bot reviewed Mar 11, 2026

View reviewed changes

Comment thread tensorrt_llm/commands/serve.py

JunyiXu-nv force-pushed the dev-junyix-feat-serve-cli-misc-improvements branch 2 times, most recently from 57a6655 to 1386e53 Compare March 11, 2026 08:14

QiJune approved these changes Mar 12, 2026

View reviewed changes

JunyiXu-nv force-pushed the dev-junyix-feat-serve-cli-misc-improvements branch from 1386e53 to 26af669 Compare March 17, 2026 06:06

JunyiXu-nv force-pushed the dev-junyix-feat-serve-cli-misc-improvements branch from 26af669 to fa0bba4 Compare March 18, 2026 03:13

JunyiXu-nv force-pushed the dev-junyix-feat-serve-cli-misc-improvements branch from fa0bba4 to e228e96 Compare March 19, 2026 06:16

JunyiXu-nv force-pushed the dev-junyix-feat-serve-cli-misc-improvements branch from e228e96 to 62c3050 Compare March 19, 2026 06:19

JunyiXu-nv enabled auto-merge (squash) March 19, 2026 06:29

JunyiXu-nv merged commit 9db8487 into NVIDIA:main Mar 19, 2026
5 checks passed

longcheng-nv pushed a commit to longcheng-nv/TensorRT-LLM that referenced this pull request Mar 31, 2026

[TRTLLM-10076][feat] Serve CLI improvements: renames, new flags, and …

cc9d8f9

…mm_embedding_serve enhancements (NVIDIA#12105) Signed-off-by: Junyi Xu <[email protected]>

Conversation

JunyiXu-nv commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Release Notes

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

coderabbitai Bot commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

❌ Failed checks (2 warnings)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JunyiXu-nv commented Mar 11, 2026

Uh oh!

tensorrt-cicd commented Mar 11, 2026

Uh oh!

tensorrt-cicd commented Mar 11, 2026

Uh oh!

JunyiXu-nv commented Mar 11, 2026

Uh oh!

tensorrt-cicd commented Mar 11, 2026

Uh oh!

tensorrt-cicd commented Mar 11, 2026

Uh oh!

QiJune left a comment

Choose a reason for hiding this comment

Uh oh!

JunyiXu-nv commented Mar 12, 2026

Uh oh!

tensorrt-cicd commented Mar 12, 2026

Uh oh!

tensorrt-cicd commented Mar 12, 2026

Uh oh!

JunyiXu-nv commented Mar 12, 2026

Uh oh!

tensorrt-cicd commented Mar 12, 2026

Uh oh!

tensorrt-cicd commented Mar 12, 2026

Uh oh!

JunyiXu-nv commented Mar 12, 2026

Uh oh!

tensorrt-cicd commented Mar 12, 2026

Uh oh!

tensorrt-cicd commented Mar 12, 2026

Uh oh!

JunyiXu-nv commented Mar 17, 2026

Uh oh!

JunyiXu-nv commented Mar 17, 2026

Uh oh!

tensorrt-cicd commented Mar 17, 2026

Uh oh!

tensorrt-cicd commented Mar 17, 2026

Uh oh!

JunyiXu-nv commented Mar 18, 2026

Uh oh!

tensorrt-cicd commented Mar 18, 2026

Uh oh!

tensorrt-cicd commented Mar 18, 2026

Uh oh!

JunyiXu-nv commented Mar 18, 2026

Uh oh!

tensorrt-cicd commented Mar 18, 2026

Uh oh!

tensorrt-cicd commented Mar 18, 2026

Uh oh!

JunyiXu-nv commented Mar 19, 2026

Uh oh!

github-actions Bot commented Mar 19, 2026

GitHub Bot Help

kill

skip

JunyiXu-nv commented Mar 11, 2026 •

edited

Loading

coderabbitai Bot commented Mar 11, 2026 •

edited

Loading