[https://nvbugs/5495789][feat] Optionally disable server GC and worker GC #7995

yuantailing · 2025-09-25T11:02:34Z

In disagg serving with high concurrency, #7858 is the major bottleneck, and this PR is minor bottleneck.

Compared to the disagg server, server and worker are reused by more modules and projects. Besides, the worker changes fast, cyclic refs may be introduced in the future. We need more investigation on drawbacks of disabling GC, so GC is not disabled by default.

Summary by CodeRabbit

New Features
- Added optional environment variables to disable Python garbage collection for the server and worker processes, enabling advanced performance tuning under high load. Defaults remain unchanged unless explicitly set.
- Potential benefits include reduced latency and improved throughput for long‑running sessions.
Chores
- Introduced runtime toggles for garbage collection without altering public APIs or error handling.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md
and the scripts/test_to_stage_mapping.py helper.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

Signed-off-by: Tailing Yuan <[email protected]>

yuantailing · 2025-09-25T11:04:52Z

/bot run

coderabbitai · 2025-09-25T11:06:21Z

📝 Walkthrough

Pre-merge checks and finishing touches

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Description Check	⚠️ Warning	The PR description fails to follow the required template because the structured “## Description” and “## Test Coverage” sections are left empty while the explanatory text appears above the template, leaving critical details about the issue, solution, and relevant tests unspecified.	Please move the explanation of the problem and your solution into the designated “## Description” section, add a clear list of tests under “## Test Coverage,” and ensure every required template heading is populated with appropriate content.
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run `@coderabbitai generate docstrings` to improve docstring coverage.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Title Check	✅ Passed	The title correctly follows the repository’s convention by including the NVBugs ticket, `[feat]` type, and a concise summary that matches the primary change of optionally disabling garbage collection for both the server and worker processes.

✨ Finishing touches

📝 Generate Docstrings

🧪 Generate unit tests

Create PR with unit tests
Post copyable unit tests in a comment

Tip

👮 Agentic pre-merge checks are now available in preview!

Pro plan users can now enable pre-merge checks in their settings to enforce checklists before merging PRs.

Built-in checks – Quickly apply ready-made checks to enforce title conventions, require pull request descriptions that follow templates, validate linked issues for compliance, and more.
Custom agentic checks – Define your own rules using CodeRabbit’s advanced agentic capabilities to enforce organization-specific policies and workflows. For example, you can instruct CodeRabbit’s agent to verify that API documentation is updated whenever API schema files are modified in a PR. Note: Upto 5 custom checks are currently allowed during the preview period. Pricing for this feature will be announced in a few weeks.

Please see the documentation for more information.

Example:

reviews:
  pre_merge_checks:
    custom_checks:
      - name: "Undocumented Breaking Changes"
        mode: "warning"
        instructions: |
          Pass/fail criteria: All breaking changes to public APIs, CLI flags, environment variables, configuration keys, database schemas, or HTTP/GraphQL endpoints must be documented in the "Breaking Change" section of the PR description and in CHANGELOG.md. Exclude purely internal or private changes (e.g., code not exported from package entry points or explicitly marked as internal).

Please share your feedback with us on this Discord post.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 2

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between e30d9ac and 9118b8e.

📒 Files selected for processing (2)

tensorrt_llm/commands/serve.py (1 hunks)
tensorrt_llm/executor/worker.py (2 hunks)

🧰 Additional context used

📓 Path-based instructions (3)

**/*.{h,hpp,hh,hxx,cpp,cxx,cc,cu,cuh,py}

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

Use only spaces, no tabs; indent with 4 spaces.

Files:

tensorrt_llm/executor/worker.py
tensorrt_llm/commands/serve.py

**/*.py

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

**/*.py: Python code must target Python 3.8+.
Indent Python code with 4 spaces; do not use tabs.
Maintain module namespace when importing; prefer 'from package.subpackage import foo' then 'foo.SomeClass()' instead of importing the class directly.
Python filenames should be snake_case (e.g., some_file.py).
Python classes use PascalCase names.
Functions and methods use snake_case names.
Local variables use snake_case; prefix 'k' for variables that start with a number (e.g., k_99th_percentile).
Global variables use upper SNAKE_CASE prefixed with 'G' (e.g., G_MY_GLOBAL).
Constants use upper SNAKE_CASE (e.g., MY_CONSTANT).
Avoid shadowing variables from an outer scope.
Initialize all externally visible members of a class in the constructor.
Prefer docstrings for interfaces that may be used outside a file; comments for in-function or file-local interfaces.
Use Google-style docstrings for classes and functions (Sphinx-parsable).
Document attributes and variables inline so they render under the class/function docstring.
Avoid reflection when a simpler, explicit approach suffices (e.g., avoid dict(**locals()) patterns).
In try/except, catch the most specific exceptions possible.
For duck-typing try/except, keep the try body minimal and use else for the main logic.

Files:

tensorrt_llm/executor/worker.py
tensorrt_llm/commands/serve.py

**/*.{cpp,cxx,cc,h,hpp,hh,hxx,cu,cuh,py}

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

Prepend the NVIDIA Apache-2.0 copyright header with current year to the top of all source files (e.g., .cpp, .h, .cu, .py).

Files:

tensorrt_llm/executor/worker.py
tensorrt_llm/commands/serve.py

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

GitHub Check: Pre-commit Check

tensorrt_llm/commands/serve.py

tensorrt_llm/executor/worker.py

tensorrt-cicd · 2025-09-25T11:10:44Z

PR_Github #19947 [ run ] triggered by Bot

tensorrt-cicd · 2025-09-25T17:21:31Z

PR_Github #19947 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #15017 completed with status: 'ABORTED'

kaiyux · 2025-09-26T00:08:52Z

/bot run --disable-fail-fast

tensorrt-cicd · 2025-09-26T00:22:20Z

PR_Github #20003 [ run ] triggered by Bot

tensorrt-cicd · 2025-09-26T00:53:33Z

PR_Github #20003 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #15065 completed with status: 'SUCCESS'

Signed-off-by: Tailing Yuan <[email protected]>

yuantailing · 2025-09-26T02:39:34Z

/bot run

tensorrt-cicd · 2025-09-26T02:44:38Z

PR_Github #20024 [ run ] triggered by Bot

tensorrt-cicd · 2025-09-26T13:34:20Z

PR_Github #20024 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #15083 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

…r GC (NVIDIA#7995) Signed-off-by: Tailing Yuan <[email protected]>

Signed-off-by: Balaram Buddharaju <[email protected]> add ds-lite tllm-gen based disagg test Signed-off-by: Matthias Jouanneaux <[email protected]> initial support for helix parallelism Signed-off-by: Matthias Jouanneaux <[email protected]> fixed mapping tests, added working MLA module test, added disagg test for helix (WIP) Signed-off-by: Matthias Jouanneaux <[email protected]> Helix MLA module test: added more scenarios, removed unnecessary code Signed-off-by: Matthias Jouanneaux <[email protected]> MLA Helix test: restricting number of tests, better output Signed-off-by: Matthias Jouanneaux <[email protected]> test MLA helix: remove OOM test scenario Signed-off-by: Matthias Jouanneaux <[email protected]> test MLA helix: fix scenario max position embeddings Signed-off-by: Matthias Jouanneaux <[email protected]> test Helix MLA: try to fix NaNs Signed-off-by: Matthias Jouanneaux <[email protected]> added all-to-all impl Signed-off-by: Matthias Jouanneaux <[email protected]> fix thop lib Signed-off-by: Matthias Jouanneaux <[email protected]> fix alltoall Signed-off-by: Matthias Jouanneaux <[email protected]> attention MLA: remove kv heads (unused), improve heads naming, fix tests Signed-off-by: Matthias Jouanneaux <[email protected]> test Helix MLA: minor fixes Signed-off-by: Matthias Jouanneaux <[email protected]> test Helix MLA: disable numeric test Signed-off-by: Matthias Jouanneaux <[email protected]> test Helix MLA: add TODOs to MLA module Signed-off-by: Matthias Jouanneaux <[email protected]> test Helix MLA: fix MLA module Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> fully working MLA test Signed-off-by: Matthias Jouanneaux <[email protected]> attempt to make latent cache work Signed-off-by: Matthias Jouanneaux <[email protected]> debugging numerical issue Signed-off-by: Matthias Jouanneaux <[email protected]> debugging numerical issue Signed-off-by: Matthias Jouanneaux <[email protected]> debugging numerical issue Signed-off-by: Matthias Jouanneaux <[email protected]> debugging numerical issue Signed-off-by: Matthias Jouanneaux <[email protected]> debugging numerical issue Signed-off-by: Matthias Jouanneaux <[email protected]> adding additional test for further numerical debugging Signed-off-by: Matthias Jouanneaux <[email protected]> fixing tests & correction Signed-off-by: Matthias Jouanneaux <[email protected]> remove debug output from tests Signed-off-by: Matthias Jouanneaux <[email protected]> fix tests Signed-off-by: Matthias Jouanneaux <[email protected]> further debugging with multiple sequences Signed-off-by: Matthias Jouanneaux <[email protected]> further debugging with multiple sequences Signed-off-by: Matthias Jouanneaux <[email protected]> further debugging with multiple sequences Signed-off-by: Matthias Jouanneaux <[email protected]> fixed multiple sequences tests Signed-off-by: Matthias Jouanneaux <[email protected]> automated review comments Signed-off-by: Matthias Jouanneaux <[email protected]> debugging of latent cache Signed-off-by: Matthias Jouanneaux <[email protected]> debugging of latent cache Signed-off-by: Matthias Jouanneaux <[email protected]> further debugging of pe values Signed-off-by: Matthias Jouanneaux <[email protected]> further debugging of latent cache Signed-off-by: Matthias Jouanneaux <[email protected]> fixed latent cache, remove flaky test Signed-off-by: Matthias Jouanneaux <[email protected]> better reporting Signed-off-by: Matthias Jouanneaux <[email protected]> better reporting Signed-off-by: Matthias Jouanneaux <[email protected]> finalized test scenarios Signed-off-by: Matthias Jouanneaux <[email protected]> better perf measurements, added graph support Signed-off-by: Matthias Jouanneaux <[email protected]> added helix post process kernel Signed-off-by: Matthias Jouanneaux <[email protected]> added unit test, minor fix for helix kernel Signed-off-by: Matthias Jouanneaux <[email protected]> fixing helix kernels Signed-off-by: Matthias Jouanneaux <[email protected]> better tests, minor fixes Signed-off-by: Matthias Jouanneaux <[email protected]> better tests, minor fixes Signed-off-by: Matthias Jouanneaux <[email protected]> debugging helix test Signed-off-by: Matthias Jouanneaux <[email protected]> debugging helix test Signed-off-by: Matthias Jouanneaux <[email protected]> debugging helix test Signed-off-by: Matthias Jouanneaux <[email protected]> fixed helix post process kernel: main kernel had perf issue/flaw Signed-off-by: Matthias Jouanneaux <[email protected]> fixed helix post process test Signed-off-by: Matthias Jouanneaux <[email protected]> added helix full layer test Signed-off-by: Matthias Jouanneaux <[email protected]> fix full layer helix test/bench Signed-off-by: Matthias Jouanneaux <[email protected]> added correct mapping to ds helix Signed-off-by: Matthias Jouanneaux <[email protected]> further improvements for fp8 init Signed-off-by: Matthias Jouanneaux <[email protected]> debugging quantization config Signed-off-by: Matthias Jouanneaux <[email protected]> better debug output Signed-off-by: Matthias Jouanneaux <[email protected]> fixes for fp8 Signed-off-by: Matthias Jouanneaux <[email protected]> fix fp8 runs Signed-off-by: Matthias Jouanneaux <[email protected]> attempt to fix fp8 context Signed-off-by: Matthias Jouanneaux <[email protected]> fix context phase: just randomly gen kv cache values. fix scenario sizes Signed-off-by: Matthias Jouanneaux <[email protected]> fix tp size config in helix layer test Signed-off-by: Matthias Jouanneaux <[email protected]> minor changes for test get trtllm-serve working with BF16 for gen with cp - v_b_proj weight loading needs to be revisited $ CUDA_VISIBLE_DEVICES=0,1 trtllm-serve /home/scratch.trt_llm_data/llm-models/DeepSeek-V3-Lite/bf16/ --host localhost --port 8002 --cp_size 2 --extra_llm_api_options ./gen_extra-llm-api-config.yaml end-to-end test in disagg works $ pytest tests/integration/defs/disaggregated/test_disaggregated.py::test_disaggregated_deepseek_v3_lite_bf16_tllm_gen_helix -s -v Switch to contiguous block dist among CP rank save changes to _merge_requests() undo changes to prepare_inputs() Raise exception for blocks fewer than num_cp_ranks save intermediate changes attempt to fix attention tests Signed-off-by: Matthias Jouanneaux <[email protected]> save changes for minimal test save minor dev comments added helix inactive rank option to MLA kernels Signed-off-by: Matthias Jouanneaux <[email protected]> pass the right seq_lens_kv - test with seqlen 64 works $ pytest tests/unittest/_torch/modules/test_mla_helix_expt.py -s -v is_inactive_helix at request level cp_allgather for position_id helix: make inactive rank a bool tensor Signed-off-by: Matthias Jouanneaux <[email protected]> undo mapping changes to modeling_deepseek Failed attempt to replace model_config.mapping fill in helix_is_inactive for each request update position_id logic better way to package mapping - repurpose comms creation too save disagg gen-only benchmark test prep for integration test improvements to position_id, num_cached_tokens_per_seq and tokens_per_block changes to save blocks at prefill changes to save blocks at decode add changes to read KV from disk updates to save and read KV blocks for all layers over-allocate at prefill to get cache transmission right prune saved KV cache files updates to avoid over-allocation on gen side in disagg Revert "over-allocate at prefill to get cache transmission right" This reverts commit af7d000. save disagg configs for DSV3 - currently goes OOM verifying tests on 8 GPUs helix: added (working) DS R1 8-GPU integration test Signed-off-by: Matthias Jouanneaux <[email protected]> helix: added large prompt + ds lite config using large prompt Signed-off-by: Matthias Jouanneaux <[email protected]> save intermediate changes for fixes fix debug printing Signed-off-by: Matthias Jouanneaux <[email protected]> Mention cache_transceiver_config.max_tokens_in_buffer for disagg servers save initial changes to benchmarking script added mjoux specific submit script, tighter timeouts, better defaults Signed-off-by: Matthias Jouanneaux <[email protected]> helix slurm: increase timeouts slightly, use deepgemm moe backend for smaller models Signed-off-by: Matthias Jouanneaux <[email protected]> helix slurm: add dataset caching path Signed-off-by: Matthias Jouanneaux <[email protected]> fix padding when input_len is divisible by tokens_per_block save changes to test varying prompt len fix_kvcache_split Signed-off-by: Chuang Zhu <[email protected]> avoid fabric memory and print send and recv sizes auto-determine transceiver size Signed-off-by: Matthias Jouanneaux <[email protected]> remove verbose print output Signed-off-by: Matthias Jouanneaux <[email protected]> attempt to fix DS R1 run Signed-off-by: Matthias Jouanneaux <[email protected]> helix slurm: fix parameters for DS R1 up to 256K tokens Signed-off-by: Matthias Jouanneaux <[email protected]> minor updates to reduce memory footprint and bring back warmup enable cudagraph and add some debug prints ugly hack to get results with 512k updates to benchmark 1M seqlen updates to benchmark 2M seqlen updates for passing down moe properly minor changes to get nsys profiles test helix layer: support for slurm call, support for fp4 Signed-off-by: Matthias Jouanneaux <[email protected]> test helix layer: added sbatch script Signed-off-by: Matthias Jouanneaux <[email protected]> add minimal cache transmission test for 1M seqlen minor bug fix changes to benchmark 4M seqlen skip launch/wait of context servers when TRTLLM_DISAGG_BENCHMARK_GEN_ONLY=1 remove hacks; skip profiling; gpu_mem_frac test helix layer: fix nvfp4 config to fit high perf mode Signed-off-by: Matthias Jouanneaux <[email protected]> helix single layer: improved timing, added arg parsing, added output parsing Signed-off-by: Matthias Jouanneaux <[email protected]> helix single layer: add dense option Signed-off-by: Matthias Jouanneaux <[email protected]> helix slurm: fix gen_only config, support EP config, add submit script for multiple configs, remove build_wheel by default for array benchmarking Signed-off-by: Matthias Jouanneaux <[email protected]> helix slurm: added parse script for results Signed-off-by: Matthias Jouanneaux <[email protected]> helix single layer: fixed test, added config submit script, improved parsing Signed-off-by: Matthias Jouanneaux <[email protected]> helix single layer: fix segment for sbatch script Signed-off-by: Matthias Jouanneaux <[email protected]> helix: fixed TP-only runs (removed hack to make higher seq len work), improved sbatch scripts Signed-off-by: Matthias Jouanneaux <[email protected]> helix: fix high node count runs, move back to e2e mode, improve parse script Signed-off-by: Matthias Jouanneaux <[email protected]> longer prompt for DSV3 Lite & DSR1 FP4 integration test disaggregated/test_disaggregated.py::test_disaggregated_deepseek_v3_lite_bf16_tllm_gen_helix disaggregated/test_disaggregated.py::test_disaggregated_deepseek_v3_lite_fp8_tllm_gen_helix disaggregated/test_disaggregated.py::test_disaggregated_deepseek_r1_fp4_tllm_gen_helix helix: added initial README for testing/benchmarking Signed-off-by: Matthias Jouanneaux <[email protected]> helix slurm: remove references to internal clusters Signed-off-by: Matthias Jouanneaux <[email protected]> minor updates to README minor updates helix: improve transpose/split for alltoall Signed-off-by: Matthias Jouanneaux <[email protected]> Revert "helix: improve transpose/split for alltoall" This reverts commit c8b24b9. helix: improve alltoall perf Signed-off-by: Matthias Jouanneaux <[email protected]> [https://nvbugs/5495789][feat] Optionally disable server GC and worker GC (NVIDIA#7995) Signed-off-by: Tailing Yuan <[email protected]> save changes for custom logging

Signed-off-by: Balaram Buddharaju <[email protected]> add ds-lite tllm-gen based disagg test Signed-off-by: Matthias Jouanneaux <[email protected]> initial support for helix parallelism Signed-off-by: Matthias Jouanneaux <[email protected]> fixed mapping tests, added working MLA module test, added disagg test for helix (WIP) Signed-off-by: Matthias Jouanneaux <[email protected]> Helix MLA module test: added more scenarios, removed unnecessary code Signed-off-by: Matthias Jouanneaux <[email protected]> MLA Helix test: restricting number of tests, better output Signed-off-by: Matthias Jouanneaux <[email protected]> test MLA helix: remove OOM test scenario Signed-off-by: Matthias Jouanneaux <[email protected]> test MLA helix: fix scenario max position embeddings Signed-off-by: Matthias Jouanneaux <[email protected]> test Helix MLA: try to fix NaNs Signed-off-by: Matthias Jouanneaux <[email protected]> added all-to-all impl Signed-off-by: Matthias Jouanneaux <[email protected]> fix thop lib Signed-off-by: Matthias Jouanneaux <[email protected]> fix alltoall Signed-off-by: Matthias Jouanneaux <[email protected]> attention MLA: remove kv heads (unused), improve heads naming, fix tests Signed-off-by: Matthias Jouanneaux <[email protected]> test Helix MLA: minor fixes Signed-off-by: Matthias Jouanneaux <[email protected]> test Helix MLA: disable numeric test Signed-off-by: Matthias Jouanneaux <[email protected]> test Helix MLA: add TODOs to MLA module Signed-off-by: Matthias Jouanneaux <[email protected]> test Helix MLA: fix MLA module Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> debugging Signed-off-by: Matthias Jouanneaux <[email protected]> fully working MLA test Signed-off-by: Matthias Jouanneaux <[email protected]> attempt to make latent cache work Signed-off-by: Matthias Jouanneaux <[email protected]> debugging numerical issue Signed-off-by: Matthias Jouanneaux <[email protected]> debugging numerical issue Signed-off-by: Matthias Jouanneaux <[email protected]> debugging numerical issue Signed-off-by: Matthias Jouanneaux <[email protected]> debugging numerical issue Signed-off-by: Matthias Jouanneaux <[email protected]> debugging numerical issue Signed-off-by: Matthias Jouanneaux <[email protected]> adding additional test for further numerical debugging Signed-off-by: Matthias Jouanneaux <[email protected]> fixing tests & correction Signed-off-by: Matthias Jouanneaux <[email protected]> remove debug output from tests Signed-off-by: Matthias Jouanneaux <[email protected]> fix tests Signed-off-by: Matthias Jouanneaux <[email protected]> further debugging with multiple sequences Signed-off-by: Matthias Jouanneaux <[email protected]> further debugging with multiple sequences Signed-off-by: Matthias Jouanneaux <[email protected]> further debugging with multiple sequences Signed-off-by: Matthias Jouanneaux <[email protected]> fixed multiple sequences tests Signed-off-by: Matthias Jouanneaux <[email protected]> automated review comments Signed-off-by: Matthias Jouanneaux <[email protected]> debugging of latent cache Signed-off-by: Matthias Jouanneaux <[email protected]> debugging of latent cache Signed-off-by: Matthias Jouanneaux <[email protected]> further debugging of pe values Signed-off-by: Matthias Jouanneaux <[email protected]> further debugging of latent cache Signed-off-by: Matthias Jouanneaux <[email protected]> fixed latent cache, remove flaky test Signed-off-by: Matthias Jouanneaux <[email protected]> better reporting Signed-off-by: Matthias Jouanneaux <[email protected]> better reporting Signed-off-by: Matthias Jouanneaux <[email protected]> finalized test scenarios Signed-off-by: Matthias Jouanneaux <[email protected]> better perf measurements, added graph support Signed-off-by: Matthias Jouanneaux <[email protected]> added helix post process kernel Signed-off-by: Matthias Jouanneaux <[email protected]> added unit test, minor fix for helix kernel Signed-off-by: Matthias Jouanneaux <[email protected]> fixing helix kernels Signed-off-by: Matthias Jouanneaux <[email protected]> better tests, minor fixes Signed-off-by: Matthias Jouanneaux <[email protected]> better tests, minor fixes Signed-off-by: Matthias Jouanneaux <[email protected]> debugging helix test Signed-off-by: Matthias Jouanneaux <[email protected]> debugging helix test Signed-off-by: Matthias Jouanneaux <[email protected]> debugging helix test Signed-off-by: Matthias Jouanneaux <[email protected]> fixed helix post process kernel: main kernel had perf issue/flaw Signed-off-by: Matthias Jouanneaux <[email protected]> fixed helix post process test Signed-off-by: Matthias Jouanneaux <[email protected]> added helix full layer test Signed-off-by: Matthias Jouanneaux <[email protected]> fix full layer helix test/bench Signed-off-by: Matthias Jouanneaux <[email protected]> added correct mapping to ds helix Signed-off-by: Matthias Jouanneaux <[email protected]> further improvements for fp8 init Signed-off-by: Matthias Jouanneaux <[email protected]> debugging quantization config Signed-off-by: Matthias Jouanneaux <[email protected]> better debug output Signed-off-by: Matthias Jouanneaux <[email protected]> fixes for fp8 Signed-off-by: Matthias Jouanneaux <[email protected]> fix fp8 runs Signed-off-by: Matthias Jouanneaux <[email protected]> attempt to fix fp8 context Signed-off-by: Matthias Jouanneaux <[email protected]> fix context phase: just randomly gen kv cache values. fix scenario sizes Signed-off-by: Matthias Jouanneaux <[email protected]> fix tp size config in helix layer test Signed-off-by: Matthias Jouanneaux <[email protected]> minor changes for test get trtllm-serve working with BF16 for gen with cp - v_b_proj weight loading needs to be revisited $ CUDA_VISIBLE_DEVICES=0,1 trtllm-serve /home/scratch.trt_llm_data/llm-models/DeepSeek-V3-Lite/bf16/ --host localhost --port 8002 --cp_size 2 --extra_llm_api_options ./gen_extra-llm-api-config.yaml end-to-end test in disagg works $ pytest tests/integration/defs/disaggregated/test_disaggregated.py::test_disaggregated_deepseek_v3_lite_bf16_tllm_gen_helix -s -v Switch to contiguous block dist among CP rank save changes to _merge_requests() undo changes to prepare_inputs() Raise exception for blocks fewer than num_cp_ranks save intermediate changes attempt to fix attention tests Signed-off-by: Matthias Jouanneaux <[email protected]> save changes for minimal test save minor dev comments added helix inactive rank option to MLA kernels Signed-off-by: Matthias Jouanneaux <[email protected]> pass the right seq_lens_kv - test with seqlen 64 works $ pytest tests/unittest/_torch/modules/test_mla_helix_expt.py -s -v is_inactive_helix at request level cp_allgather for position_id helix: make inactive rank a bool tensor Signed-off-by: Matthias Jouanneaux <[email protected]> undo mapping changes to modeling_deepseek Failed attempt to replace model_config.mapping fill in helix_is_inactive for each request update position_id logic better way to package mapping - repurpose comms creation too save disagg gen-only benchmark test prep for integration test improvements to position_id, num_cached_tokens_per_seq and tokens_per_block changes to save blocks at prefill changes to save blocks at decode add changes to read KV from disk updates to save and read KV blocks for all layers over-allocate at prefill to get cache transmission right prune saved KV cache files updates to avoid over-allocation on gen side in disagg Revert "over-allocate at prefill to get cache transmission right" This reverts commit af7d000. save disagg configs for DSV3 - currently goes OOM verifying tests on 8 GPUs helix: added (working) DS R1 8-GPU integration test Signed-off-by: Matthias Jouanneaux <[email protected]> helix: added large prompt + ds lite config using large prompt Signed-off-by: Matthias Jouanneaux <[email protected]> save intermediate changes for fixes fix debug printing Signed-off-by: Matthias Jouanneaux <[email protected]> Mention cache_transceiver_config.max_tokens_in_buffer for disagg servers save initial changes to benchmarking script added mjoux specific submit script, tighter timeouts, better defaults Signed-off-by: Matthias Jouanneaux <[email protected]> helix slurm: increase timeouts slightly, use deepgemm moe backend for smaller models Signed-off-by: Matthias Jouanneaux <[email protected]> helix slurm: add dataset caching path Signed-off-by: Matthias Jouanneaux <[email protected]> fix padding when input_len is divisible by tokens_per_block save changes to test varying prompt len fix_kvcache_split Signed-off-by: Chuang Zhu <[email protected]> avoid fabric memory and print send and recv sizes auto-determine transceiver size Signed-off-by: Matthias Jouanneaux <[email protected]> remove verbose print output Signed-off-by: Matthias Jouanneaux <[email protected]> attempt to fix DS R1 run Signed-off-by: Matthias Jouanneaux <[email protected]> helix slurm: fix parameters for DS R1 up to 256K tokens Signed-off-by: Matthias Jouanneaux <[email protected]> minor updates to reduce memory footprint and bring back warmup enable cudagraph and add some debug prints ugly hack to get results with 512k updates to benchmark 1M seqlen updates to benchmark 2M seqlen updates for passing down moe properly minor changes to get nsys profiles test helix layer: support for slurm call, support for fp4 Signed-off-by: Matthias Jouanneaux <[email protected]> test helix layer: added sbatch script Signed-off-by: Matthias Jouanneaux <[email protected]> add minimal cache transmission test for 1M seqlen minor bug fix changes to benchmark 4M seqlen skip launch/wait of context servers when TRTLLM_DISAGG_BENCHMARK_GEN_ONLY=1 remove hacks; skip profiling; gpu_mem_frac test helix layer: fix nvfp4 config to fit high perf mode Signed-off-by: Matthias Jouanneaux <[email protected]> helix single layer: improved timing, added arg parsing, added output parsing Signed-off-by: Matthias Jouanneaux <[email protected]> helix single layer: add dense option Signed-off-by: Matthias Jouanneaux <[email protected]> helix slurm: fix gen_only config, support EP config, add submit script for multiple configs, remove build_wheel by default for array benchmarking Signed-off-by: Matthias Jouanneaux <[email protected]> helix slurm: added parse script for results Signed-off-by: Matthias Jouanneaux <[email protected]> helix single layer: fixed test, added config submit script, improved parsing Signed-off-by: Matthias Jouanneaux <[email protected]> helix single layer: fix segment for sbatch script Signed-off-by: Matthias Jouanneaux <[email protected]> helix: fixed TP-only runs (removed hack to make higher seq len work), improved sbatch scripts Signed-off-by: Matthias Jouanneaux <[email protected]> helix: fix high node count runs, move back to e2e mode, improve parse script Signed-off-by: Matthias Jouanneaux <[email protected]> longer prompt for DSV3 Lite & DSR1 FP4 integration test disaggregated/test_disaggregated.py::test_disaggregated_deepseek_v3_lite_bf16_tllm_gen_helix disaggregated/test_disaggregated.py::test_disaggregated_deepseek_v3_lite_fp8_tllm_gen_helix disaggregated/test_disaggregated.py::test_disaggregated_deepseek_r1_fp4_tllm_gen_helix helix: added initial README for testing/benchmarking Signed-off-by: Matthias Jouanneaux <[email protected]> helix slurm: remove references to internal clusters Signed-off-by: Matthias Jouanneaux <[email protected]> minor updates to README minor updates helix: improve transpose/split for alltoall Signed-off-by: Matthias Jouanneaux <[email protected]> Revert "helix: improve transpose/split for alltoall" This reverts commit c8b24b9. helix: improve alltoall perf Signed-off-by: Matthias Jouanneaux <[email protected]> [https://nvbugs/5495789][feat] Optionally disable server GC and worker GC (NVIDIA#7995) Signed-off-by: Tailing Yuan <[email protected]> save changes for custom logging redo cherry-pick of attention.py save more changes for build and pipe-cleaning save more changes clean up - 1 clean up - 2 reuse mla_tensor_params instead of using helix_tensor_params undo all_tp_rank_num_tokens update test_disaggregated.py updates to dsv3RopeOp more cleanup save fp8 disagg test [https://nvbugs/5637012][fix] Fix helix unit tests Signed-off-by: Balaram Buddharaju <[email protected]> minor updates to attention.py updates to test - seqlen 64 works get integration test working

Optionally disable server GC and worker GC

9118b8e

Signed-off-by: Tailing Yuan <[email protected]>

yuantailing requested a review from a team as a code owner September 25, 2025 11:02

yuantailing requested a review from hchings September 25, 2025 11:02

coderabbitai bot reviewed Sep 25, 2025

View reviewed changes

tensorrt_llm/commands/serve.py Show resolved Hide resolved

tensorrt_llm/executor/worker.py Outdated Show resolved Hide resolved

kaiyux approved these changes Sep 25, 2025

View reviewed changes

Cancel int conversion

203cefb

Signed-off-by: Tailing Yuan <[email protected]>

yuantailing requested a review from pcastonguay September 26, 2025 02:39

pcastonguay approved these changes Sep 26, 2025

View reviewed changes

yuantailing merged commit b11ee86 into NVIDIA:main Sep 26, 2025
5 checks passed

yuantailing deleted the optionally_disable_gc branch September 28, 2025 10:42

kaiyux mentioned this pull request Sep 29, 2025

[TRTLLM-7292][feat] Support manual gc in trtllm-serve #7518

Closed

1 task

[https://nvbugs/5495789][feat] Optionally disable server GC and worker GC #7995

[https://nvbugs/5495789][feat] Optionally disable server GC and worker GC #7995

Uh oh!

Conversation

yuantailing commented Sep 25, 2025 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Description

Test Coverage

PR Checklist

GitHub Bot Help

kill

skip

reuse-pipeline

Uh oh!

yuantailing commented Sep 25, 2025

Uh oh!

coderabbitai bot commented Sep 25, 2025

Pre-merge checks and finishing touches

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tensorrt-cicd commented Sep 25, 2025

Uh oh!

tensorrt-cicd commented Sep 25, 2025

Uh oh!

kaiyux commented Sep 26, 2025

Uh oh!

tensorrt-cicd commented Sep 26, 2025

Uh oh!

tensorrt-cicd commented Sep 26, 2025

Uh oh!

yuantailing commented Sep 26, 2025

Uh oh!

tensorrt-cicd commented Sep 26, 2025

Uh oh!

tensorrt-cicd commented Sep 26, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yuantailing commented Sep 25, 2025 •

edited by coderabbitai bot

Loading