[#9150][feat] AutoDeploy Nemotron-Flash support #9504
Conversation
Signed-off-by: Lucas Liebenwein <[email protected]>
📝 WalkthroughWalkthroughIntroduces support for NemotronFlash-3B-Instruct model by adding a custom model implementation with delta-rule attention kernels, registering it with a factory, and providing example deployment configuration and documentation. Changes
Sequence Diagram(s)sequenceDiagram
participant Tokenizer as NemotronFlashTokenizer
participant Model as NemotronFlashModel
participant Decoder as DecoderLayer
participant Attention as Attention/Mamba/FFN
participant Delta as DeltaRule Backend
Tokenizer->>Model: input_ids (+ memory tokens)
Model->>Model: embed tokens
Model->>Decoder: hidden_states, position_ids
loop For each decoder layer
Decoder->>Attention: hidden_states
alt Layer Type
Attention->>Delta: q, k, v, beta (via cached_delta_rule)
Delta->>Delta: prefill: chunk_delta_rule_fwd
Delta->>Delta: decode: fused_recurrent_delta_rule_fwd
Delta->>Delta: update delta_cache with final_state
Delta-->>Attention: output
else Mamba2
Attention->>Attention: apply SSM
else FFN
Attention->>Attention: apply MLP
end
Attention-->>Decoder: output
Decoder->>Decoder: residual + norm
Decoder-->>Model: next hidden_states
end
Model->>Model: final RMSNorm
Model->>Model: TruncatedLinear LM head
Model-->>Tokenizer: logits
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes
Pre-merge checks and finishing touches❌ Failed checks (2 warnings)
✅ Passed checks (1 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 16
🧹 Nitpick comments (14)
tensorrt_llm/_torch/auto_deploy/custom_ops/rms_norm.py (1)
1-1: Missing NVIDIA copyright header at top of filePer project guidelines, each TensorRT-LLM OSS Python file should start with the standard NVIDIA copyright header including the current year. Please add the project-standard header above the module docstring when you next touch this file.
tensorrt_llm/_torch/auto_deploy/custom_ops/mamba/torch_causal_conv.py (1)
1-1: Add NVIDIA copyright headerThis file also appears to be missing the standard NVIDIA copyright header at the very top. Please prepend the project-standard header comment above the module docstring to align with the TensorRT-LLM coding guidelines.
tensorrt_llm/_torch/auto_deploy/custom_ops/attention_interface.py (2)
1-10: Add NVIDIA copyright header at top of filePer TensorRT‑LLM guidelines, all open‑source
.pyfiles should start with an NVIDIA copyright header including the current year. Consider adding it above the module docstring while you are touching this file.
35-39: Re‑checkdelta_dtypedefault vsCacheConfig.__or__merge semantics
delta_dtypenow defaults totorch.float32, so it is neverNone. With__or__implemented as:merged_kwargs[field_name] = getattr(self, field_name) or getattr(other, field_name)the left‑hand
delta_dtypewill always win, and a right‑hand config cannot override it unless the left explicitly setsdelta_dtype=None.If the intended usage is “later/override configs can change
delta_dtypewhen merging”, you may wantdelta_dtypeto default toNoneand apply thefloat32default at use‑site instead, or adjust the merge logic to giveotherprecedence.Also applies to: 52-62
examples/auto_deploy/nemotron_flash.yaml (1)
8-8: Minor formatting inconsistency in the list.There's a missing space after the comma between
64and96.Apply this diff:
-cuda_graph_batch_sizes: [1, 2, 4, 8, 16, 24, 32, 64,96, 128, 256, 320, 384] +cuda_graph_batch_sizes: [1, 2, 4, 8, 16, 24, 32, 64, 96, 128, 256, 320, 384]tensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/wy_fast.py (1)
93-114: Consider adding docstrings to public functions.The
prepare_wy_repr_fwdandrecompute_w_u_fwdfunctions appear to be part of the public API. Per coding guidelines, interfaces used outside a file should prefer docstrings over comments.Example for
prepare_wy_repr_fwd:def prepare_wy_repr_fwd( k: torch.Tensor, v: torch.Tensor, beta: torch.Tensor, cu_seqlens: Optional[torch.LongTensor], ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]: """Prepare the WY representation for delta-rule attention. Args: k: Key tensor of shape [B, T, H, K]. v: Value tensor of shape [B, T, H, V]. beta: Beta tensor of shape [B, T, H]. cu_seqlens: Cumulative sequence lengths for variable-length batches. Returns: Tuple of (w, u, A) tensors for the delta-rule computation. """tensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_delta.py (1)
18-21: Prefix unused unpacked variables with underscore.The variables
Aandfinal_statereturned bychunk_delta_rule_fwdare not used. Per Python convention (and Ruff RUF059), prefix them with an underscore to indicate intentional non-use.- o, A, final_state = chunk_delta_rule_fwd( + o, _A, _final_state = chunk_delta_rule_fwd( q, k, v, beta, scale, initial_state=None, output_final_state=False, cu_seqlens=None )tensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_backend_delta.py (2)
66-68: Specify device forbatch_info_tensorto ensure consistency.The tensor is created on CPU by default while other tensors use
seq_len_sanitized.device. Whiletolist()works regardless, explicit device placement prevents potential issues and clarifies intent.batch_info_tensor = torch.tensor( - [num_prefill, num_prefill_tokens, num_decode], dtype=torch.int32 + [num_prefill, num_prefill_tokens, num_decode], + dtype=torch.int32, + device=seq_len_sanitized.device, )
91-91: Fake implementation should match device placement of real implementation.If the real implementation specifies a device for
batch_info_tensor, the fake should match for consistency during tracing.- torch.empty(3, dtype=torch.int32), # host tensor + torch.empty(3, dtype=torch.int32, device=seq_len_sanitized.device),tensorrt_llm/_torch/auto_deploy/models/custom/modeling_nemotron_flash.py (3)
162-162: UseOptional[int]instead of implicitNonedefault.PEP 484 prohibits implicit
Optional. Sincelayer_idxcan beNone, it should be explicitly typed asOptional[int].- layer_idx: int = None, + layer_idx: Optional[int] = None,
581-587: Hardcodeddevice="cuda"may fail on CPU-only systems.Consider making the device configurable or inferring it from the model's device placement.
def _init_rope(self): self.rotary_emb = LlamaRotaryEmbedding( config=self.config, dim=self.kq_head_dim, base=self.rope_theta, - device=torch.device("cuda"), + device=None, # Will be moved to correct device during model loading )
74-74: Remove return type annotation from__init__.
__init__methods should not have return type annotations (they implicitly returnNone). The same applies to lines 97 and 168.- def __init__(self, hidden_size: int, eps: float = 1e-5) -> "NemotronFlashRMSNorm": + def __init__(self, hidden_size: int, eps: float = 1e-5) -> None:tensorrt_llm/_torch/auto_deploy/custom_ops/l2norm.py (2)
5-5: Prefer importing the module to preserve namespace instead of the functionThe current import brings
l2norm_fwddirectly into this module’s namespace:from tensorrt_llm._torch.modules.fla.l2norm import l2norm_fwdGuidelines recommend maintaining module namespaces. Consider importing the submodule and calling through it:
-import torch - -from tensorrt_llm._torch.modules.fla.l2norm import l2norm_fwd +import torch + +from tensorrt_llm._torch.modules.fla import l2norm as fla_l2norm_modAnd later:
-def fla_l2norm(x: torch.Tensor, eps: float = 1e-6) -> torch.Tensor: - y = l2norm_fwd(x, eps) +def fla_l2norm(x: torch.Tensor, eps: float = 1e-6) -> torch.Tensor: + y = fla_l2norm_mod.l2norm_fwd(x, eps)As per coding guidelines, imports should preserve a module namespace.
27-29: Address unusedepsarguments in fake kernels (Ruff ARG001)Ruff correctly flags
epsas unused in the fake implementations. To keep signatures aligned with the real ops while silencing the warning, you can explicitly discardeps:@_torch_l2norm.register_fake def _torch_l2norm_fake(x: torch.Tensor, eps: float = 1e-6) -> torch.Tensor: - return torch.empty_like(x) + del eps + return torch.empty_like(x) @@ @fla_l2norm.register_fake def fla_l2norm_fake(x: torch.Tensor, eps: float = 1e-6) -> torch.Tensor: - return torch.empty_like(x) + del eps + return torch.empty_like(x)This keeps the API consistent and resolves the static-analysis warning.
Also applies to: 38-40
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (19)
docs/source/features/auto_deploy/support_matrix.md(1 hunks)examples/auto_deploy/.gitignore(1 hunks)examples/auto_deploy/nemotron_flash.yaml(1 hunks)tensorrt_llm/_torch/auto_deploy/config/default.yaml(1 hunks)tensorrt_llm/_torch/auto_deploy/custom_ops/attention_interface.py(1 hunks)tensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/chunk.py(1 hunks)tensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/fused_recurrent.py(1 hunks)tensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/utils.py(1 hunks)tensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/wy_fast.py(1 hunks)tensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_backend_delta.py(1 hunks)tensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_delta.py(1 hunks)tensorrt_llm/_torch/auto_deploy/custom_ops/l2norm.py(1 hunks)tensorrt_llm/_torch/auto_deploy/custom_ops/mamba/torch_causal_conv.py(1 hunks)tensorrt_llm/_torch/auto_deploy/custom_ops/rms_norm.py(1 hunks)tensorrt_llm/_torch/auto_deploy/models/__init__.py(1 hunks)tensorrt_llm/_torch/auto_deploy/models/custom/__init__.py(1 hunks)tensorrt_llm/_torch/auto_deploy/models/custom/modeling_nemotron_flash.py(1 hunks)tensorrt_llm/_torch/auto_deploy/models/nemotron_flash.py(1 hunks)tensorrt_llm/_torch/auto_deploy/transform/library/ssm_cache.py(1 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py
📄 CodeRabbit inference engine (CODING_GUIDELINES.md)
**/*.py: The code developed for TensorRT-LLM should conform to Python 3.8+
Indent Python code with 4 spaces; do not use tabs
Always maintain the namespace when importing in Python, even if only one class or function from a module is used (e.g., usefrom package.subpackage import fooand thenfoo.SomeClass()instead offrom package.subpackage.foo import SomeClass)
Python filenames should use snake_case (e.g.,some_file.py)
Python class names should use PascalCase (e.g.,class SomeClass)
Python function and method names should use snake_case (e.g.,def my_awesome_function():)
Python local variable names should use snake_case, with prefixkfor variable names that start with a number (e.g.,k_99th_percentile = ...)
Python global variables should use upper snake_case with prefixG(e.g.,G_MY_GLOBAL = ...)
Python constants should use upper snake_case (e.g.,MY_CONSTANT = ...)
Avoid shadowing variables declared in an outer scope in Python
Initialize all externally visible members of a Python class in the constructor
For Python interfaces that may be used outside a file, prefer docstrings over comments
Python comments should be reserved for code within a function, or interfaces that are local to a file
Use Google style docstrings for Python classes and functions, which can be parsed by Sphinx
Python attributes and variables can be documented inline with type and description (e.g.,self.x = 5followed by"""<type>: Description of 'x'""")
Avoid using reflection in Python when functionality can be easily achieved without reflection
When using try-except blocks in Python, limit the except clause to the smallest set of specific errors possible instead of catching all exceptions
When using try-except blocks in Python to handle multiple possible variable types (duck-typing), keep the body of the try as small as possible and use the else block to implement the logic
Files:
tensorrt_llm/_torch/auto_deploy/custom_ops/attention_interface.pytensorrt_llm/_torch/auto_deploy/transform/library/ssm_cache.pytensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/utils.pytensorrt_llm/_torch/auto_deploy/custom_ops/l2norm.pytensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/wy_fast.pytensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_delta.pytensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/chunk.pytensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/fused_recurrent.pytensorrt_llm/_torch/auto_deploy/models/nemotron_flash.pytensorrt_llm/_torch/auto_deploy/models/custom/__init__.pytensorrt_llm/_torch/auto_deploy/custom_ops/rms_norm.pytensorrt_llm/_torch/auto_deploy/custom_ops/mamba/torch_causal_conv.pytensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_backend_delta.pytensorrt_llm/_torch/auto_deploy/models/__init__.pytensorrt_llm/_torch/auto_deploy/models/custom/modeling_nemotron_flash.py
**/*.{cpp,h,cu,py}
📄 CodeRabbit inference engine (CODING_GUIDELINES.md)
All TensorRT-LLM Open Source Software code files should contain an NVIDIA copyright header that includes the current year at the top
Files:
tensorrt_llm/_torch/auto_deploy/custom_ops/attention_interface.pytensorrt_llm/_torch/auto_deploy/transform/library/ssm_cache.pytensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/utils.pytensorrt_llm/_torch/auto_deploy/custom_ops/l2norm.pytensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/wy_fast.pytensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_delta.pytensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/chunk.pytensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/fused_recurrent.pytensorrt_llm/_torch/auto_deploy/models/nemotron_flash.pytensorrt_llm/_torch/auto_deploy/models/custom/__init__.pytensorrt_llm/_torch/auto_deploy/custom_ops/rms_norm.pytensorrt_llm/_torch/auto_deploy/custom_ops/mamba/torch_causal_conv.pytensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_backend_delta.pytensorrt_llm/_torch/auto_deploy/models/__init__.pytensorrt_llm/_torch/auto_deploy/models/custom/modeling_nemotron_flash.py
🧠 Learnings (9)
📚 Learning: 2025-08-09T02:04:49.623Z
Learnt from: Fridah-nv
Repo: NVIDIA/TensorRT-LLM PR: 6760
File: tensorrt_llm/_torch/auto_deploy/models/quant_config_reader.py:81-98
Timestamp: 2025-08-09T02:04:49.623Z
Learning: In TensorRT-LLM's auto_deploy module, torch.dtype values in configuration dictionaries must be stored as string representations (e.g., "float16" instead of torch.float16) because OmegaConf.merge does not support torch.dtype types. These string representations are converted to actual torch.dtype objects in downstream code.
Applied to files:
tensorrt_llm/_torch/auto_deploy/custom_ops/attention_interface.py
📚 Learning: 2025-08-06T03:47:16.802Z
Learnt from: venkywonka
Repo: NVIDIA/TensorRT-LLM PR: 6650
File: tests/integration/test_lists/qa/llm_perf_cluster.yml:33-37
Timestamp: 2025-08-06T03:47:16.802Z
Learning: Ministral is a valid and distinct model family from Mistral AI, separate from their regular Mistral models. Ministral 8B is specifically designed for edge computing and on-device applications, released in October 2024. In TensorRT-LLM test configurations, "ministral_8b" and "ministral_8b_fp8" are correct model identifiers and should not be changed to "mistral_8b".
Applied to files:
docs/source/features/auto_deploy/support_matrix.md
📚 Learning: 2025-08-06T03:47:16.802Z
Learnt from: venkywonka
Repo: NVIDIA/TensorRT-LLM PR: 6650
File: tests/integration/test_lists/qa/llm_perf_cluster.yml:33-37
Timestamp: 2025-08-06T03:47:16.802Z
Learning: Ministral is a valid model name from Mistral AI, distinct from the regular Mistral models. In TensorRT-LLM test configurations, "ministral_8b" and "ministral_8b_fp8" are correct model identifiers and should not be changed to "mistral_8b".
Applied to files:
docs/source/features/auto_deploy/support_matrix.md
📚 Learning: 2025-09-09T18:31:44.336Z
Learnt from: venkywonka
Repo: NVIDIA/TensorRT-LLM PR: 7658
File: .github/CODEOWNERS:160-164
Timestamp: 2025-09-09T18:31:44.336Z
Learning: The teams NVIDIA/trt-llm-release-nim-branch-approval and NVIDIA/trt-llm-release-branch-approval exist in the NVIDIA organization and are valid for use in .github/CODEOWNERS files, even if they may not be accessible via external API queries due to permissions.
Applied to files:
docs/source/features/auto_deploy/support_matrix.md
📚 Learning: 2025-09-04T07:33:10.618Z
Learnt from: MrGeva
Repo: NVIDIA/TensorRT-LLM PR: 7219
File: tensorrt_llm/_torch/auto_deploy/compile/backends/torch_cudagraph.py:162-168
Timestamp: 2025-09-04T07:33:10.618Z
Learning: When users explicitly provide cuda_graph_batch_sizes in TorchCudagraphCompiler, respect their choices and only sanitize the values (clamp, dedupe, sort) without forcing additional batch sizes like 1 or max_batch_size. Only add commonly-used batch sizes when falling back to the heuristic.
Applied to files:
examples/auto_deploy/nemotron_flash.yaml
📚 Learning: 2025-10-20T16:54:09.824Z
Learnt from: nvchenghaoz
Repo: NVIDIA/TensorRT-LLM PR: 8469
File: tensorrt_llm/_torch/auto_deploy/custom_ops/rms_norm.py:6-6
Timestamp: 2025-10-20T16:54:09.824Z
Learning: In tensorrt_llm/_torch/auto_deploy/custom_ops/rms_norm.py, the import `from ...modules.mamba.layernorm_gated import _layer_norm_fwd` is correct and should not be changed to modules.fla.layernorm_gated. The _layer_norm_fwd function exists in both modules/mamba/layernorm_gated.py and modules/fla/layernorm_gated.py, but the mamba version is the intended implementation for this use case.
Applied to files:
tensorrt_llm/_torch/auto_deploy/custom_ops/l2norm.pytensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/wy_fast.pytensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_delta.pytensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/chunk.pytensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/fused_recurrent.pytensorrt_llm/_torch/auto_deploy/models/custom/__init__.pytensorrt_llm/_torch/auto_deploy/custom_ops/rms_norm.pytensorrt_llm/_torch/auto_deploy/custom_ops/mamba/torch_causal_conv.pytensorrt_llm/_torch/auto_deploy/models/__init__.pytensorrt_llm/_torch/auto_deploy/models/custom/modeling_nemotron_flash.py
📚 Learning: 2025-10-20T17:09:21.560Z
Learnt from: nvchenghaoz
Repo: NVIDIA/TensorRT-LLM PR: 8469
File: tensorrt_llm/_torch/auto_deploy/transform/library/rms_norm.py:180-182
Timestamp: 2025-10-20T17:09:21.560Z
Learning: In tensorrt_llm/_torch/auto_deploy/transform/library/rms_norm.py, the _gated_rmsnorm_replacement function does not need to cast the output of torch.ops.auto_deploy.torch_rmsnorm_gated back to the input dtype, even though the custom op returns fp32. The dtype handling is managed elsewhere or the fp32 output is acceptable for downstream consumers.
Applied to files:
tensorrt_llm/_torch/auto_deploy/custom_ops/l2norm.pytensorrt_llm/_torch/auto_deploy/custom_ops/rms_norm.py
📚 Learning: 2025-08-27T14:41:56.665Z
Learnt from: ixlmar
Repo: NVIDIA/TensorRT-LLM PR: 7294
File: tensorrt_llm/_torch/modules/rms_norm.py:96-99
Timestamp: 2025-08-27T14:41:56.665Z
Learning: In tensorrt_llm/_torch/modules/rms_norm.py, the RMSNorm class uses a custom sentinel (_ARGUMENT_NOT_SPECIFIED_SENTINEL) instead of Ellipsis (...) for detecting unspecified optional arguments. Other modules in the codebase may use Ellipsis as a sentinel but do not forward it to RMSNorm methods, so there's no need for backward compatibility with Ellipsis in RMSNorm.
Applied to files:
tensorrt_llm/_torch/auto_deploy/custom_ops/rms_norm.py
📚 Learning: 2025-07-22T02:20:31.841Z
Learnt from: venkywonka
Repo: NVIDIA/TensorRT-LLM PR: 6019
File: tests/unittest/llmapi/lora_test_utils.py:125-214
Timestamp: 2025-07-22T02:20:31.841Z
Learning: The .nemo format requires configuration files to have a .yaml extension even if the content is JSON format, as this is a constraint of NeMo's loading/unloading mechanisms. Changing the extension would cause complications with NeMo compatibility.
Applied to files:
examples/auto_deploy/.gitignore
🧬 Code graph analysis (13)
tensorrt_llm/_torch/auto_deploy/custom_ops/attention_interface.py (2)
tensorrt_llm/llmapi/llm_args.py (1)
Field(63-90)tensorrt_llm/builder.py (1)
default(45-50)
tensorrt_llm/_torch/auto_deploy/transform/library/ssm_cache.py (3)
tensorrt_llm/_torch/auto_deploy/transform/interface.py (1)
TransformRegistry(507-535)tensorrt_llm/_torch/auto_deploy/custom_ops/attention_interface.py (1)
register(906-914)tensorrt_llm/_torch/auto_deploy/transform/library/kvcache.py (1)
InsertCachedAttention(80-215)
tensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/utils.py (1)
tensorrt_llm/_torch/autotuner.py (1)
autotune(219-251)
tensorrt_llm/_torch/auto_deploy/custom_ops/l2norm.py (2)
tensorrt_llm/_torch/modules/fla/l2norm.py (2)
l2norm(133-136)l2norm_fwd(74-122)tensorrt_llm/functional.py (1)
sum(3253-3275)
tensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/wy_fast.py (4)
tensorrt_llm/_torch/modules/fla/chunk_scaled_dot_kkt.py (1)
chunk_scaled_dot_kkt_fwd(88-143)tensorrt_llm/_torch/modules/fla/index.py (1)
prepare_chunk_indices(17-23)tensorrt_llm/_torch/modules/fla/solve_tril.py (1)
solve_tril(360-426)tensorrt_llm/_torch/modules/fla/utils.py (1)
check_shared_mem(300-306)
tensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_delta.py (2)
tensorrt_llm/functional.py (1)
chunk(3826-3861)tensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/chunk.py (1)
chunk_delta_rule_fwd(13-42)
tensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/chunk.py (3)
tensorrt_llm/_torch/modules/fla/chunk_delta_h.py (1)
chunk_gated_delta_rule_fwd_h(236-294)tensorrt_llm/_torch/modules/fla/chunk_o.py (1)
chunk_fwd_o(124-169)tensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/wy_fast.py (1)
prepare_wy_repr_fwd(93-114)
tensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/fused_recurrent.py (1)
tests/unittest/_torch/attention/sparse/test_dsa_indexer.py (1)
cdiv(44-46)
tensorrt_llm/_torch/auto_deploy/models/nemotron_flash.py (3)
tensorrt_llm/_torch/auto_deploy/models/factory.py (1)
ModelFactoryRegistry(354-376)tensorrt_llm/_torch/auto_deploy/models/hf.py (2)
AutoModelForCausalLMFactory(100-521)_get_model_config(198-213)tensorrt_llm/_torch/auto_deploy/models/custom/modeling_nemotron_flash.py (1)
NemotronFlashPreTrainedTokenizerFast(19-62)
tensorrt_llm/_torch/auto_deploy/models/custom/__init__.py (1)
tensorrt_llm/_torch/auto_deploy/models/custom/modeling_nemotron_flash.py (2)
NemotronFlashForCausalLM(1023-1085)NemotronFlashPreTrainedTokenizerFast(19-62)
tensorrt_llm/_torch/auto_deploy/custom_ops/rms_norm.py (2)
tensorrt_llm/_torch/auto_deploy/custom_ops/attention_interface.py (1)
to(501-508)tensorrt_llm/_torch/auto_deploy/shim/interface.py (1)
to(42-46)
tensorrt_llm/_torch/auto_deploy/custom_ops/mamba/torch_causal_conv.py (1)
tensorrt_llm/functional.py (1)
conv1d(3548-3585)
tensorrt_llm/_torch/auto_deploy/models/custom/modeling_nemotron_flash.py (4)
tensorrt_llm/_torch/auto_deploy/models/nemotron_flash.py (1)
NemotronFlashForCausalLMFactory(6-20)tensorrt_llm/_torch/auto_deploy/custom_ops/rms_norm.py (4)
triton_rmsnorm_gated(87-139)_(29-40)_(59-61)_(81-83)tensorrt_llm/_torch/auto_deploy/custom_ops/torch_attention.py (1)
torch_attention(96-212)tensorrt_llm/_torch/auto_deploy/models/hf.py (1)
register_custom_model_cls(503-517)
🪛 Ruff (0.14.6)
tensorrt_llm/_torch/auto_deploy/custom_ops/l2norm.py
28-28: Unused function argument: eps
(ARG001)
39-39: Unused function argument: eps
(ARG001)
tensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_delta.py
18-18: Unpacked variable A is never used
Prefix it with an underscore or any other dummy variable pattern
(RUF059)
18-18: Unpacked variable final_state is never used
Prefix it with an underscore or any other dummy variable pattern
(RUF059)
26-26: Unused function argument: q
(ARG001)
27-27: Unused function argument: k
(ARG001)
29-29: Unused function argument: beta
(ARG001)
30-30: Unused function argument: scale
(ARG001)
tensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_backend_delta.py
31-31: Unused function argument: cache_loc
(ARG001)
32-32: Unused function argument: pages_per_seq
(ARG001)
34-34: Unused function argument: page_size
(ARG001)
35-35: Unused function argument: chunk_size
(ARG001)
77-77: Unused function argument: input_pos
(ARG001)
78-78: Unused function argument: cache_loc
(ARG001)
79-79: Unused function argument: pages_per_seq
(ARG001)
81-81: Unused function argument: page_size
(ARG001)
82-82: Unused function argument: chunk_size
(ARG001)
176-176: Unused function argument: q
(ARG001)
177-177: Unused function argument: k
(ARG001)
179-179: Unused function argument: beta
(ARG001)
181-181: Unused function argument: cu_seqlens
(ARG001)
182-182: Unused function argument: slot_idx
(ARG001)
183-183: Unused function argument: use_initial_states
(ARG001)
184-184: Unused function argument: batch_info_tensor
(ARG001)
186-186: Unused function argument: delta_cache
(ARG001)
188-188: Unused function argument: scale
(ARG001)
246-246: Unused class method argument: source_attn_node
(ARG003)
tensorrt_llm/_torch/auto_deploy/models/custom/modeling_nemotron_flash.py
47-47: Prefer TypeError exception for invalid type
(TRY004)
47-47: Avoid specifying long messages outside the exception class
(TRY003)
162-162: PEP 484 prohibits implicit Optional
Convert to T | None
(RUF013)
166-166: Unused method argument: config
(ARG002)
167-167: Unused method argument: kwargs
(ARG002)
233-233: Unused method argument: kwargs
(ARG002)
365-365: Unused method argument: kwargs
(ARG002)
443-443: Avoid specifying long messages outside the exception class
(TRY003)
548-551: Avoid specifying long messages outside the exception class
(TRY003)
594-594: Unused method argument: kwargs
(ARG002)
725-725: Unused method argument: kwargs
(ARG002)
759-759: Unused method argument: position_ids
(ARG002)
759-759: Unused method argument: kwargs
(ARG002)
810-810: Avoid specifying long messages outside the exception class
(TRY003)
826-826: Unused method argument: position_ids
(ARG002)
826-826: Unused method argument: kwargs
(ARG002)
855-855: Mutable class attributes should be annotated with typing.ClassVar
(RUF012)
917-917: Avoid specifying long messages outside the exception class
(TRY003)
948-948: Unused static method argument: local_metadata
(ARG004)
949-949: Unused static method argument: strict
(ARG004)
950-950: Unused static method argument: missing_keys
(ARG004)
951-951: Unused static method argument: unexpected_keys
(ARG004)
952-952: Unused static method argument: error_msgs
(ARG004)
977-977: Unused method argument: kwargs
(ARG002)
980-980: Avoid specifying long messages outside the exception class
(TRY003)
982-982: Avoid specifying long messages outside the exception class
(TRY003)
1024-1024: Mutable class attributes should be annotated with typing.ClassVar
(RUF012)
1026-1026: Unused method argument: kwargs
(ARG002)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Pre-commit Check
🔇 Additional comments (21)
tensorrt_llm/_torch/auto_deploy/custom_ops/rms_norm.py (1)
73-77: Contiguous output fortorch_rmsnormlooks correctThe added
.contiguous()on(weight * input.to(input_dtype))is reasonable: it preserves shape and dtype while ensuring a contiguous layout, which should help downstream consumers that assume contiguity, at the cost of at most one extra copy when needed. No correctness issues from this change.tensorrt_llm/_torch/auto_deploy/custom_ops/mamba/torch_causal_conv.py (1)
24-36: Causal conv output contiguity change is appropriateWrapping the
conv1d(...)[..., :seq_len].transpose(1, 2)path and adding.contiguous()ensures the returned tensor is contiguous after the transpose, without changing semantics. This is consistent with similar contiguity guarantees elsewhere and should make downstream code more robust to layout assumptions.tensorrt_llm/_torch/auto_deploy/custom_ops/attention_interface.py (1)
41-49: Validator extension fordelta_dtypelooks consistent with existing patternIncluding
"delta_dtype"in_coerce_dtypekeeps behavior aligned withdtypeandmamba_dtypeand supports string values like"float16"/"float32"coming from configuration (e.g., OmegaConf) that are then converted totorch.dtype. This matches the existing approach in auto_deploy where dtypes are stored as strings in configs and coerced later. Based on learnings, this change looks correct and complete.examples/auto_deploy/.gitignore (1)
8-8: LGTM!The unignore pattern follows the existing convention for tracking deployment config files.
examples/auto_deploy/nemotron_flash.yaml (1)
1-11: Configuration looks well-structured.The settings appropriately configure the NemotronFlash model with chunked prefill, cuda graph compilation, and correctly disables block reuse for hybrid/SSM models.
The
max_seq_len: 2097152(2M tokens) is very large. Please verify this is the intended context length for the Nemotron-Flash-3B-Instruct model.tensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/wy_fast.py (2)
28-91: Kernel implementation is well-structured.The Triton kernel correctly handles both fixed-length and variable-length sequences with proper boundary checks. Using
allow_tf32=Falsein the dot operations ensures numerical precision for the delta-rule computation.
117-152: Function implementation looks correct.The dynamic tiling based on shared memory availability (
check_shared_mem()) is a good approach for hardware adaptability. The kernel launch parameters align properly with the kernel signature.docs/source/features/auto_deploy/support_matrix.md (1)
86-86: LGTM!The new model entry is correctly placed in alphabetical order within the nvidia models section.
tensorrt_llm/_torch/auto_deploy/models/__init__.py (1)
1-4: LGTM!The import changes correctly register the
nemotron_flashandcustommodules, ensuring the model factory and custom implementations are available when the package is loaded. This follows the pattern described in the TODO comment.tensorrt_llm/_torch/auto_deploy/transform/library/ssm_cache.py (1)
16-20: LGTM!The new
InsertCachedDeltaRuletransform follows the established pattern of its sibling classes (SSMCacheTransform,InitializeCausalConvCache) and is correctly registered with theTransformRegistry.tensorrt_llm/_torch/auto_deploy/config/default.yaml (1)
155-157: LGTM!The new
insert_cached_delta_ruletransform entry is correctly configured with thecache_initstage andfla_deltabackend, following the established pattern for other cache initialization transforms in this file.tensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/chunk.py (1)
13-42: LGTM!The
chunk_delta_rule_fwdfunction correctly orchestrates the delta-rule forward pass by:
- Computing the WY representation via
prepare_wy_repr_fwd- Running the gated delta-rule forward pass via
chunk_gated_delta_rule_fwd_h- Computing the output via
chunk_fwd_oThe implementation follows the expected pattern from the FLA library.
tensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_delta.py (1)
24-32: Fake implementation is correct.The fake variant correctly returns an empty tensor matching
v's shape for shape inference purposes. The unused arguments (q,k,beta,scale) are expected—fake implementations only need to produce correctly shaped output tensors.tensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/fused_recurrent.py (2)
9-109: Triton kernel implementation looks correct.The kernel properly handles:
- Variable-length sequences via
cu_seqlens- Initial and final state management
- Headwise vs scalar beta modes
- Correct mask handling for partial tiles
One minor note: the variable
all(lines 47, 51) shadows the Python built-in, but this is acceptable within Triton kernel scope and is common practice in adapted FLA code.
121-160: Wrapper implementation is correct.The wrapper correctly:
- Computes block sizes using
triton.next_power_of_2- Validates that
NK == 1(single K-block constraint)- Allocates output tensors with appropriate shapes
- Launches the kernel with proper grid dimensions
tensorrt_llm/_torch/auto_deploy/models/nemotron_flash.py (1)
5-20: I'll verify the concerns raised in this review comment. The reviewer wants to ensure thatnum_memory_tokensandvocab_sizeattributes are always present in the NemotronFlash model config to avoid runtimeAttributeError.Let me start by exploring the codebase structure and locating the relevant configuration files.
<function_calls>
#!/bin/bashFind files related to NemotronFlash configuration and modeling
fd -e py -path "nemotron" | grep -E "(config|model)" | head -20
</function_calls>Now let me search for where the model config is defined and used:
<function_calls>
#!/bin/bashSearch for NemotronFlash config class definition
rg -n "class.*NemotronFlash.*Config" --type py
</function_calls>Let me also check the specific file mentioned in the review:
<function_calls>
#!/bin/bashList files in the custom directory related to Nemotron
fd -e py "nemotron" tensorrt_llm/_torch/auto_deploy/models/custom/
</function_calls>Let me examine the init_tokenizer method and trace where model_config comes from:
<function_calls>
#!/bin/bashSearch for _get_model_config method
rg -n "_get_model_config" --type py -B 2 -A 5
</function_calls>Let me search for the custom imports and custom.py file:
<function_calls>
#!/bin/bashFind the custom module/file
fd -type f "custom*" tensorrt_llm/_torch/auto_deploy/models/ | grep -v pycache
</function_calls>Let me check the structure of the auto_deploy/models directory:
<function_calls>
#!/bin/bashList directory structure
find tensorrt_llm/_torch/auto_deploy/models -type f -name "*.py" | grep -v pycache | head -30
</function_calls>Let me search more broadly for num_memory_tokens attribute:
<function_calls>
#!/bin/bashSearch for num_memory_tokens in the entire codebase
rg -n "num_memory_tokens" --type py -B 3 -A 3
</function_calls><function_calls>
#!/bin/bashSearch for vocab_size in model configs and NemotronFlash related code
rg -n "vocab_size" --type py tensorrt_llm/_torch/auto_deploy/ -B 2 -A 2
</function_calls>Let me look at the file under review more closely:
<function_calls>
#!/bin/bashRead the nemotron_flash.py file
cat -n tensorrt_llm/_torch/auto_deploy/models/nemotron_flash.py
</function_calls><function_calls>
#!/bin/bashSearch for AutoModelForCausalLMFactory parent class
rg -n "class AutoModelForCausalLMFactory" --type py -A 10
</function_calls>Let me check if there's a init.py or custom.py in the models directory:
<function_calls>
#!/bin/bashCheck the directory contents
ls -la tensorrt_llm/_torch/auto_deploy/models/
</function_calls><function_calls>
#!/bin/bashIf custom is a directory, list its contents
ls -la tensorrt_llm/_torch/auto_deploy/models/custom/ 2>/dev/null || echo "custom directory not found or not accessible"
</function_calls>Let me search for where model_config attributes are accessed in the codebase:
<function_calls>
#!/bin/bashSearch for model_config.num_memory_tokens usage
rg -n "model_config.num_memory_tokens" --type py -B 3 -A 3
</function_calls><function_calls>
#!/bin/bashSearch for model_config.vocab_size usage
rg -n "model_config.vocab_size" --type py -B 3 -A 3
</function_calls>Let me check the imports and base classes more carefully:
<function_calls>
#!/bin/bashFind and read auto_deploy related files
find tensorrt_llm/_torch/auto_deploy -name "*.py" -type f | grep -v pycache | sort
</function_calls>Let me check the file size first and then examine it:
<function_calls>
#!/bin/bashCheck file sizes
wc -l tensorrt_llm/_torch/auto_deploy/.py 2>/dev/null
wc -l tensorrt_llm/_torch/auto_deploy//*.py 2>/dev/null | head -20
</function_calls>Let me examine the factory base class:
<function_calls>
#!/bin/bashFind the AutoModelForCausalLMFactory definition
rg -n "class AutoModelForCausalLMFactory" --type py -A 50 | head -80
</function_calls>Let me search for where _get_model_config is implemented:
<function_calls>
#!/bin/bashFind _get_model_config implementation
rg -n "def _get_model_config" --type py -A 15
</function_calls>Let me look for how model config is loaded and check if there's any validation:
<function_calls>
#!/bin/bashSearch for pretrained config loading
rg -n "from_pretrained|PretrainedConfig" --type py tensorrt_llm/_torch/auto_deploy/ -B 1 -A 1 | head -40
</function_calls>Let me check if NemotronFlash is a Hugging Face model and search for its actual configuration:
<function_calls>
#!/bin/bashSearch for NemotronFlash references in the entire codebase
rg -n "NemotronFlash" --type py -C 3 | head -100
</function_calls>Let me examine the nemotron_flash.py file directly to understand its full content:
<function_calls>
#!/bin/bashGet full content of nemotron_flash.py
wc -l tensorrt_llm/_torch/auto_deploy/models/nemotron_flash.py
</function_calls><function_calls>
#!/bin/bashRead the entire file
cat tensorrt_llm/_torch/auto_deploy/models/nemotron_flash.py
</function_calls><function_calls>
#!/bin/bashSearch for custom module or package related to NemotronFlash
rg -n "NemotronFlashPreTrainedTokenizerFast" --type py -B 5 -A 5
</function_calls>Let me check the directory structure more carefully:
<function_calls>
#!/bin/bashList all Python files in auto_deploy/models
find tensorrt_llm/_torch/auto_deploy/models -name "*.py" -type f | sort
</function_calls>Now let me search for where the model is instantiated or used to understand what config attributes are actually required:
<function_calls>
#!/bin/bashSearch for actual config attributes initialization or assignment
rg -n "num_memory_tokens.*=" --type py -B 2 -A 2
</function_calls><function_calls>
#!/bin/bashLook for test files that might use NemotronFlash
find . -path "test" -name "nemotron" -type f 2>/dev/null
</function_calls>Let me check if there's any documentation or model card reference:
<function_calls>
#!/bin/bashSearch for README or documentation mentioning NemotronFlash
find tensorrt_llm -name "README*" -o -name "*.md" | xargs grep -l "nemotron" 2>/dev/null
</function_calls>Let me search for actual Python model configs in the repo:
<function_calls>
#!/bin/bashSearch for PretrainedConfig usage that might define num_memory_tokens
rg -n "PretrainedConfig|modeling_" --type py tensorrt_llm/_torch/ -l | head -20
</function_calls>Let me check if config.json files exist:
<function_calls>
#!/bin/bashSearch for config.json references or example configs
find tensorrt_llm -name "config.json" 2>/dev/null | head -5
</function_calls>Let me search in the imports for clues about NemotronFlash model structure:
<function_calls>
#!/bin/bashCheck if NemotronFlash config comes from transformers or is custom
rg -n "from.*transformers.*import|from.*custom.*import" tensorrt_llm/_torch/auto_deploy/models/nemotron_flash.py
</function_calls>Let me examine the parent class and understand the initialization flow:
<function_calls>
#!/bin/bashSearch for all factory classes to understand the pattern
rg -n "class.*Factory.*AutoModel" --type py -B 2 -A 15 | head -100
</function_calls>Let me search for error handling related to config attributes:
<function_calls>
#!/bin/bashSearch for try-except or getattr usage around config attributes
rg -n "getattr|hasattr|try:" --type py tensorrt_llm/_torch/auto_deploy/ -B 2 -A 2 | head -80
</function_calls>Let me check the actual NemotronFlash model on Hugging Face by searching for references in code:
<function_calls>
#!/bin/bashLook for any references to model identifiers or huggingface hub
rg -n "nvidia/Nemotron|huggingface" --type py tensorrt_llm/_torch/ -i
</function_calls>Let me check if there's a setup or configuration that defines what attributes a NemotronFlash config must have:
<function_calls>
#!/bin/bashSearch for class definitions that might define model config structure
ast-grep --pattern 'class $_ {
$$$
num_memory_tokens
$$$
}'
</function_calls>Let me try a different approach - search for how PreTrainedTokenizerFast is used:
<function_calls>
#!/bin/bashSearch for PreTrainedTokenizerFast usage and requirements
rg -n "PreTrainedTokenizerFast|from_pretrained" --type py tensorrt_llm/_torch/auto_deploy/ -B 3 -A 8 | head -120
</function_calls>Let me check if there's model validation logic anywhere:
<function_calls>
#!/bin/bashSearch for validation or requirement checks
rg -n "assert|raise|ValueError.*config" --type py tensorrt_llm/_torch/auto_deploy/ -B 1 -A 1 | head -60
</function_calls>Let me check if there's any existing attribute access to understand if this is being done safely:
<function_calls>
#!/bin/bashLook at how the factory processes model configs in general
rg -n "model_config." --type py tensorrt_llm/_torch/auto_deploy/ -C 2
</function_calls>Let me search for imports from transformers to understand if NemotronFlash is a standard Hugging Face model:
<function_calls>
#!/bin/bashCheck transformers imports and see if NemotronFlash is standard or custom
rg -n "from transformers import" --type py tensorrt_llm/_torch/auto_deploy/
</function_calls>Let me check more carefully in the repository for NemotronFlash implementation:
<function_calls>
#!/bin/bashSearch for all references to Nemotron across the entire repo
rg -r "nemotron" --type py -i | head -30
</function_calls>Let me check if there's a model registry or mapping:
<function_calls>
#!/bin/bashSearch for ModelFactoryRegistry and its definitions
rg -n "ModelFactoryRegistry" --type py -B 2 -A 5 | head -80
</function_calls>Let me examine if there's any documentation about the specific attributes needed:
<function_calls>
#!/bin/bashLook for comments or docstrings mentioning num_memory_tokens
rg -n "memory.*token" --type py -i -B 2 -A 2
</function_calls>Let me check the imports at the top of the file being reviewed:
<function_calls>
#!/bin/bashGet the first 30 lines of nemotron_flash.py
head -30 tensorrt_llm/_torch/auto_deploy/models/nemotron_flash.py
</function_calls>Let me search for the parent class factory implementation:
<function_calls>
#!/bin/bashFind where AutoModelForCausalLMFactory is defined
fd -type f -name "*.py" -path "auto_deploy" | xargs grep -l "class AutoModelForCausalLMFactory"
</function_calls><function_calls>
#!/bin/bashRead the parent factory class
find tensorrt_llm -name "*.py" -type f | xargs grep -l "class AutoModelForCausalLMFactory" | head -1 | xargs cat
</function_calls>Let me search for the actual factory pattern and base implementation:
<function_calls>
#!/bin/bashFind all factory files
find tensorrt_llm/_torch/auto_deploy -name "factory" -o -name "base" | grep -v pycache
</function_calls><function_calls>
#!/bin/bashLook at what files are in auto_deploy
ls -la tensorrt_llm/_torch/auto_deploy/
</function_calls><function_calls>
#!/bin/bashRead the init.py or main file
head -100 tensorrt_llm/_torch/auto_deploy/init.py 2>/dev/null || cat tensorrt_llm/_torch/auto_deploy.py 2>/dev/null || echo "Neither found"
</function_calls>Let me search in a different way - looking for the actual import statement:
<function_calls>
#!/bin/bashLook for all imports in nemotron_flash.py with context
cat tensorrt_llm/_torch/auto_deploy/models/nemotron_flash.py | head -50
</function_calls><function_calls>
#!/bin/bashLook at all Python files in the models directory
ls -la tensorrt_llm/_torch/auto_deploy/models/*.py 2>/dev/null | head -20
</function_calls>Let me check if these files exist at all:
<function_calls>
#!/bin/bashVerify the path exists and check full structure
if [ -d "tensorrt_llm/_torch/auto_deploy" ]; then
echo "auto_deploy exists"
find tensorrt_llm/_torch/auto_deploy -type f -name "*.py" | head -20
else
echo "Path does not exist, searching for auto_deploy"
find tensorrt_llm -type d -name "auto_deploy" 2>/dev/null
fi
</function_calls>Let me try a broader search:
<function_calls>
#!/bin/bashSearch for nemotron_flash.py anywhere
find . -name "nemotron_flash.py" -type f 2>/dev/null
</function_calls><function_calls>
#!/bin/bashSearch for NemotronFlashForCausalLMFactory
rg "NemotronFlashForCausalLMFactory" -type py
</function_calls>Let me check what Python files are actually in the repo:
<function_calls>
#!/bin/bashGet a general overview of the repo structure
git ls-files "*.py" | head -30
</function_calls><function_calls>
#!/bin/bashSearch for files with "nemotron" in any form
git ls-files | grep -i nemotron
</function_calls><function_calls>
#!/bin/bashSearch for any _torch directory
git ls-files | grep "_torch" | head -20
</function_calls><function_calls>
#!/bin/bashGet top-level directories
git ls-tree -d HEAD | head -20
</function_calls>Based on my investigation, I need to provide the rewritten review comment. The verification revealed that the files and paths referenced in the review comment do not exist in the current state of the repository. The file
tensorrt_llm/_torch/auto_deploy/models/nemotron_flash.pycannot be located, and the classNemotronFlashForCausalLMFactorydoes not appear in the codebase. This indicates the code snippet shown in the review comment may be from a feature branch, proposed change, or the paths are incorrect.The file being reviewed does not exist in the current codebase. The path
tensorrt_llm/_torch/auto_deploy/models/nemotron_flash.pyand all referenced components (NemotronFlashForCausalLMFactory,AutoModelForCausalLMFactory,NemotronFlashPreTrainedTokenizerFast) cannot be verified as they are not present in the repository. Without access to the actual implementation, the configuration class definition, and the parent factory classes, it is impossible to verify whethernum_memory_tokensandvocab_sizeattributes are guaranteed to exist in the NemotronFlash model config.tensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_backend_delta.py (2)
95-170: LGTM!The prefill/decode separation logic correctly handles both phases with appropriate cache state management. The use of
index_copy_for updating the delta cache is efficient.
193-256: LGTM!The
FlaDeltaBackendclass correctly implements theAttentionDescriptorinterface. The unusedsource_attn_nodeparameter inget_global_buffer_initializersis required by the interface contract.tensorrt_llm/_torch/auto_deploy/models/custom/modeling_nemotron_flash.py (2)
1088-1090: LGTM!The model registration correctly hooks
NemotronFlashForCausalLMinto the factory system using the expectedconfig_cls_namepattern.
998-1001: I'll verify whetherglobal_attn_idxis always iterable as assumed in line 1000. Let me search the codebase for how this attribute is defined and used.
<function_calls>
#!/bin/bashSearch for global_attn_idx usage and definition
rg -n "global_attn_idx" --type=py -C3
</function_calls>
#!/bin/bashGet more context on config.global_attn_idx
rg -n "config.global_attn_idx|.global_attn_idx\s*=" --type=py -B2 -A2
</function_calls>
#!/bin/bashCheck the specific file around line 925 and 1000
head -n 1005 tensorrt_llm/_torch/auto_deploy/models/custom/modeling_nemotron_flash.py | tail -n 85 | cat -n
</function_calls>tensorrt_llm/_torch/auto_deploy/custom_ops/l2norm.py (1)
32-35: Core Fla-backedfla_l2normbinding looks correctThe
fla_l2normcustom op cleanly delegates tol2norm_fwd(x, eps)and returns its output without side effects beyond tensor math. This matches the intended linkage to the Triton-backed Fla kernel and keeps the signature simple.
tensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/fused_recurrent.py
Show resolved
Hide resolved
tensorrt_llm/_torch/auto_deploy/custom_ops/fla/delta_rule/fused_recurrent.py
Show resolved
Hide resolved
tensorrt_llm/_torch/auto_deploy/models/custom/modeling_nemotron_flash.py
Show resolved
Hide resolved
tensorrt_llm/_torch/auto_deploy/models/custom/modeling_nemotron_flash.py
Show resolved
Hide resolved
tensorrt_llm/_torch/auto_deploy/models/custom/modeling_nemotron_flash.py
Show resolved
Hide resolved
tensorrt_llm/_torch/auto_deploy/models/custom/modeling_nemotron_flash.py
Show resolved
Hide resolved
|
/bot run |
|
PR_Github #25902 [ run ] triggered by Bot. Commit: |
Signed-off-by: Lucas Liebenwein <[email protected]>
|
/bot run |
|
PR_Github #25915 [ run ] triggered by Bot. Commit: |
|
PR_Github #25902 [ run ] completed with state |
|
PR_Github #25915 [ run ] completed with state |
…VIDIA#8779) The performance results of some kernels could be easily affected by the warm/cold L2 cache status. To achieve more precise profiling results, the L2 cache is cleared for every execution by the circular buffer method for better benchmarking during autotuning. Signed-off-by: Yukun He <[email protected]> [None][infra] Waive failed cases for main branch on 11/25 (NVIDIA#9429) Signed-off-by: qqiao <[email protected]> [NVIDIA#8391][chore] test_perf.py to lock clocks read from gpu_configs.yml instead of max freq (NVIDIA#9409) Signed-off-by: Eran Geva <[email protected]> [None][ci] Move more test stages to use OCI machines (NVIDIA#9395) Signed-off-by: Yanchao Lu <[email protected]> Co-authored-by: Matt Lefebvre <[email protected]> [None][feat] Improve TRTLLM MoE in small hidden size throughput cases (NVIDIA#9377) Signed-off-by: Anthony Chang <[email protected]> [https://nvbugs/5537996][fix] Let KV cache manager block initialization be aware whether it is doing a dry run or not (NVIDIA#9093) Before this commit, the kv cache manager does the same regardless, which causes a mis-calculation in free memory available to allocate for the KV cache manager, hence causing a crash. This commit fixes this by letting KV cache manager initialization be aware whether it is doing the dry run or not. If it is a dry run, use the max_tokens setting that is already pre-calculated and filled into kv_cache_config.max_tokens. Signed-off-by: eopXD <[email protected]> [https://nvbugs/5667922][fix] Update long context evaluation config (NVIDIA#9426) Signed-off-by: mni <[email protected]> [None][fix] Mitigate test timeout issues (NVIDIA#9445) Signed-off-by: Shixiaowei02 <[email protected]> [None][chore] Fix trtllm-eval for PyTorchLLM (NVIDIA#9427) Signed-off-by: Fanrong Li <[email protected]> [None][feat] Add a parser to layer-wise benchmarks (NVIDIA#9440) Signed-off-by: Tailing Yuan <[email protected]> [None][feat] Support custom chat template for tool calling (NVIDIA#9297) Signed-off-by: Pengyun Lin <[email protected]> [TRTLLM-8160][feat] Add draft token tree runtime on CDL (NVIDIA#8586) Signed-off-by: Yue Weng <[email protected]> [None][ci] waive a test (NVIDIA#9458) Signed-off-by: Yan Chunwei <[email protected]> [https://nvbugs/5680905][fix] Relax the MMLU accuracy requirement for DS-v3.2 (NVIDIA#9439) Signed-off-by: Fanrong Li <[email protected]> [TRTLLM-8376][feat] top-p optimization (removes redundant softmax) (NVIDIA#9411) Signed-off-by: ixlmar <[email protected]> [TRTLLM-9490][feat] use FlashInfer's top_k_sampling_from_probs (NVIDIA#9457) Signed-off-by: ixlmar <[email protected]> [https://nvbugs/5647400] [fix] Enlarged the AllReduce workspace size to 64MB. Added AllReduce strategy to AD config. (NVIDIA#9145) Signed-off-by: Eran Geva <[email protected]> [TRTLLM-909][feat] Overlap context chunks in pipeline parallel mode (NVIDIA#9308) Signed-off-by: Robin Kobus <[email protected]> [None][chore] AutoDeploy add multi stream moe pass to default.yaml (NVIDIA#9430) Signed-off-by: Suyog Gupta <[email protected]> [https://nvbugs/5685143][fix] avoid cudaFree overlap with cuda graph (NVIDIA#9438) Signed-off-by: Chuang Zhu <[email protected]> [None][chore] Bump version to 1.2.0rc5 (NVIDIA#9455) Signed-off-by: Yiqing Yan <[email protected]> [TRTLLM-8936][test] Add disagg and wideep multi-node multi-gpu test cases (NVIDIA#9356) Signed-off-by: FredricZ-2007 <[email protected]> [None][ci] move some slow test cases of DGX-B200 to post merge (NVIDIA#9467) Signed-off-by: junq <[email protected]> [TRTLLM-9293][feat] Enable partial weight loading to support streaming update weights (NVIDIA#9224) Signed-off-by: shuyix <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [TRTLLM-9264][fix] Add accuracy/unit tests/doc for phi4mm (NVIDIA#9246) Signed-off-by: Wanli Jiang <[email protected]> [https://nvbugs/5580099][fix] Cherry pick IMA issue fix from release/1.1 (NVIDIA#9032) Signed-off-by: Junyi Xu <[email protected]> [None][chore] Upgrade CuteDSL to 4.3.0 (NVIDIA#9444) Signed-off-by: Enwei Zhu <[email protected]> [None][feat] Support MLA chunked prefill for DeepSeek V3.2 model (NVIDIA#9376) Signed-off-by: Chang Liu (Enterprise Products) <[email protected]> [None][feat] Add environment variable to force spec-dec number of accepted tokens (NVIDIA#9371) Signed-off-by: Aurelien Chartier <[email protected]> [None][infra] Update allowed list 2025.11.25 (NVIDIA#9468) Signed-off-by: Yuanjing Xue <[email protected]> [None][infra] Fail the pipeline when slurm ssh dropped (NVIDIA#9157) Signed-off-by: Yuanjing Xue <[email protected]> [None][feat] AutoDeploy: Remove redundant copies in mamba layers (NVIDIA#9461) Signed-off-by: Chenghao Zhang <[email protected]> Co-authored-by: Suyog Gupta <[email protected]> [None][feat] AutoDeploy: Add A_log fusion for Mamba layers (NVIDIA#9422) Signed-off-by: Chenghao Zhang <[email protected]> [None][ci] Waive blackwell test on spec gate. (NVIDIA#9502) Signed-off-by: Zheyu Fu <[email protected]> [https://nvbugs/5608930][fix] Fix a typo (NVIDIA#9487) Signed-off-by: Shixiaowei02 <[email protected]> [NVIDIA#9463][feat] Add revision option to trtllm commands (NVIDIA#9498) Signed-off-by: Aurelien Chartier <[email protected]> [TRTLLM-9085][doc] fix math formula rendering issues (NVIDIA#9481) Signed-off-by: junq <[email protected]> [None][chore] update comments in llm_args.py (NVIDIA#9472) Signed-off-by: junq <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [https://nvbugs/5680310][fix] Fix ctx only timed out test (NVIDIA#9410) Signed-off-by: Patrice Castonguay <[email protected]> [https://nvbugs/5547414][fix] enable case after using local cache model (NVIDIA#9473) Signed-off-by: Hui Gao <[email protected]> [None][fix] Replace PYTORCH_CUDA_ALLOC_CONF with PYTORCH_ALLOC_CONF to fix deprecation warning (NVIDIA#9294) Signed-off-by: Jiagan Cheng <[email protected]> [https://nvbugs/5698581][fix] Init draft tokens for CUDA graph dummy request (NVIDIA#9505) Signed-off-by: ziyixiong-nv <[email protected]> [None][infra] Waive failed case in pre-merge on 11/27 (NVIDIA#9507) Signed-off-by: qqiao <[email protected]> [TRTLLM-9513][docs] Qwen3 deployment guide (NVIDIA#9488) Signed-off-by: Lanyu Liao <[email protected]> Co-authored-by: Lanyu Liao <[email protected]> [None][chore] revert batch_size=1 to prevent timeout and lower accuracy reference by 0.12% as a WAR (NVIDIA#9447) Signed-off-by: Lizhi Zhou <[email protected]> Co-authored-by: Shi Xiaowei <[email protected]> [TRTLLM-9279][infra] Use flexcache for gh200 nodes since they locate in Austin (NVIDIA#9405) Signed-off-by: qqiao <[email protected]> Signed-off-by: Emma Qiao <[email protected]> Co-authored-by: Yanchao Lu <[email protected]> [cherry-pick][https://nvbugs/5670793][fix] Solve trtllm-serve launch_disaggregated issue (NVIDIA#9346) Signed-off-by: xxi <[email protected]> [None][infra] Fix Slurm job script (NVIDIA#9508) Signed-off-by: Yuanjing Xue <[email protected]> [None][fix] change allreduce workspace dtype to torch.int64 to avoid overflow (NVIDIA#9479) Signed-off-by: Zhenhuan Chen <[email protected]> [None][feat] add qwen3-next CI test of accuracy on BF16 and NVFP4 (NVIDIA#9330) Signed-off-by: jiant <[email protected]> [None][fix] fix TP support for DeepSeek-V3.2 on hopper (NVIDIA#9484) Signed-off-by: Fanrong Li <[email protected]> [TRTLLM-9389][chore] Refactor AlltoallMethodType. (NVIDIA#9388) Signed-off-by: Bo Li <[email protected]> [https://nvbugs/5674665][chore] Add test coverage for https://nvbugspro.nvidia.com/bug/5674665 (NVIDIA#9518) Signed-off-by: eopXD <[email protected]> [TRTLLM-7288][infra] Download merged waive list in slurm script (NVIDIA#8999) Signed-off-by: Yiqing Yan <[email protected]> Signed-off-by: Yanchao Lu <[email protected]> Co-authored-by: Yanchao Lu <[email protected]> [https://nvbugs/5687820][fix] Remove self.abort() in DetokenizedGenerationResult (NVIDIA#9449) Signed-off-by: Enwei Zhu <[email protected]> [NVIDIA#9150][feat] AutoDeploy Nemotron-Flash support (NVIDIA#9504) Signed-off-by: Lucas Liebenwein <[email protected]> [None] [chore] Update to cutlass 4.3 (NVIDIA#8637) Signed-off-by: Kaiyu Xie <[email protected]> [https://nvbugs/5637037][chore] Update waive lists. (NVIDIA#9386) Signed-off-by: Bo Li <[email protected]> Signed-off-by: Enwei Zhu <[email protected]> Co-authored-by: Enwei Zhu <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [TRTLLM-8970][infra] Fix generate report when has isolation test result (NVIDIA#8861) Signed-off-by: qqiao <[email protected]> Signed-off-by: Emma Qiao <[email protected]> [https://nvbugs/5685015][fix] Update invalid max_token test (NVIDIA#9435) Signed-off-by: Junyi Xu <[email protected]> [None][fix] Fix on-disk cache and revise logger/statistics for AutoTuner. (NVIDIA#9211) Signed-off-by: Yukun He <[email protected]> [https://nvbugs/5689658][test] Fix gpu lock issue running on cluster (NVIDIA#9441) Signed-off-by: yufeiwu <[email protected]> [None][chore] add spec_decoding configs in perf benchmark scripts and fix typos (NVIDIA#9533) Signed-off-by: Lanyu Liao <[email protected]> Co-authored-by: Lanyu Liao <[email protected]> [None][fix] Remove FP8 K/V buffer from TRTLLM sparse MLA attention kernel (NVIDIA#9529) Signed-off-by: Chang Liu (Enterprise Products) <[email protected]> [None] [chore] Enhancements and clean up to slurm scripts (NVIDIA#9493) Signed-off-by: Kaiyu Xie <[email protected]> [None][chore] Revert "[None][fix] change allreduce workspace dtype to torch.int64 t… (NVIDIA#9538) Signed-off-by: Zhenhuan Chen <[email protected]> [None][infra] Waive failed cases for main branch on 11/28 (NVIDIA#9539) Signed-off-by: qqiao <[email protected]> [None][fix] Pass checkpoint_format to create_input_processor (NVIDIA#9521) Signed-off-by: Robin Kobus <[email protected]> [TRTLLM-9541][infra] Use artifactory mirror for download.pytorch.org (NVIDIA#9477) Signed-off-by: ZhanruiSunCh <[email protected]> Signed-off-by: Zhanrui Sun <[email protected]> Co-authored-by: Yanchao Lu <[email protected]> [TRTLLM-9488][feat] add 'disable_flashinfer_sampling' config option (NVIDIA#9454) Signed-off-by: ixlmar <[email protected]> [None][infra] Waive failed case in pre-merge on 11/28 (NVIDIA#9537) Signed-off-by: Wangshanshan <[email protected]> [None][perf] Helix: improve all-to-all perf for large CP size (NVIDIA#9494) Signed-off-by: Matthias Jouanneaux <[email protected]> Signed-off-by: Zheyu Fu <[email protected]> Co-authored-by: Zheyu Fu <[email protected]> [None][feat] support for more accurate AR calculation (NVIDIA#9323) Signed-off-by: binghanc <[email protected]> [TRTLLM-9488][fix] llmapi references (NVIDIA#9547) Signed-off-by: ixlmar <[email protected]> [NVIDIA#8948][feat] Support custom sharding config (NVIDIA#9143) Signed-off-by: greg-kwasniewski1 <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [None][chore] Weekly mass integration of release/1.1 -- rebase (NVIDIA#9522) Signed-off-by: yunruis <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Wangshanshan <[email protected]> Signed-off-by: qgai <[email protected]> Signed-off-by: Balaram Buddharaju <[email protected]> Signed-off-by: Yan Chunwei <[email protected]> Signed-off-by: Junyi Xu <[email protected]> Signed-off-by: Simeng Liu <[email protected]> Signed-off-by: nv-guomingz <[email protected]> Signed-off-by: Jin Li <[email protected]> Signed-off-by: Ivy Zhang <[email protected]> Signed-off-by: Vincent Zhang <[email protected]> Signed-off-by: peaceh <[email protected]> Signed-off-by: Michal Guzek <[email protected]> Signed-off-by: Michal Guzek <[email protected]> Signed-off-by: Chang Liu (Enterprise Products) <[email protected]> Signed-off-by: leslie-fang25 <[email protected]> Signed-off-by: Shunkang <[email protected]> Signed-off-by: junq <[email protected]> Co-authored-by: yunruis <[email protected]> Co-authored-by: sunnyqgg <[email protected]> Co-authored-by: brb-nv <[email protected]> Co-authored-by: Yan Chunwei <[email protected]> Co-authored-by: JunyiXu-nv <[email protected]> Co-authored-by: Simeng Liu <[email protected]> Co-authored-by: Guoming Zhang <[email protected]> Co-authored-by: Jin Li <[email protected]> Co-authored-by: Ivy Zhang <[email protected]> Co-authored-by: Vincent Zhang <[email protected]> Co-authored-by: peaceh-nv <[email protected]> Co-authored-by: Michal Guzek <[email protected]> Co-authored-by: Chang Liu <[email protected]> Co-authored-by: Leslie Fang <[email protected]> Co-authored-by: Shunkangz <[email protected]> Co-authored-by: Shunkang <[email protected]> Co-authored-by: QI JUN <[email protected]> [TRTLLM-5971][feat] Integrate helix parallelism (NVIDIA#9342) Signed-off-by: Balaram Buddharaju <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [None][infra] - Request idle time exemption for OCI jobs (NVIDIA#9528) Signed-off-by: Yanchao Lu <[email protected]> [None][infra] Wiave failed tests for main branch on 11/30 (NVIDIA#9555) Signed-off-by: qqiao <[email protected]> [None][fix] Fix port conflict in disagg tests (NVIDIA#9474) Signed-off-by: Junyi Xu <[email protected]> [None][ci] Split H100_PCIe-PyTorch-Post-Merge test stage (NVIDIA#9558) Signed-off-by: Yanchao Lu <[email protected]> [None][ci] Split H100_PCIe-PyTorch-Post-Merge test stage (NVIDIA#9559) Signed-off-by: Yanchao Lu <[email protected]> [TRTLLM-8958][feat] and [TRTLLM-8960]: create ConfigurableMoE and support TRTLLMGenFusedMoE as backend (NVIDIA#9486) [None] [feat] Optimize the algorithm part of RocketKV (NVIDIA#9333) Signed-off-by: yuhangh <[email protected]> [https://nvbugs/5690172][fix] Fix Qwen3-235B ATP accuracy issue with PDL (NVIDIA#9530) Signed-off-by: Enwei Zhu <[email protected]> [TRTLLM-6222][feat] Extend cute_dsl_nvfp4_gemm to sm103. (NVIDIA#9543) Signed-off-by: Mindy Li <[email protected]> [None][fix] Correct virtual memory allocation alignment (NVIDIA#9491) Signed-off-by: Yuan Tong <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [https://nvbugs/5684703][fix] Unwaive disagg guided decoding test (NVIDIA#9466) Signed-off-by: Enwei Zhu <[email protected]> [https://nvbugs/5503479][fix] Temporarily lower reference accuracy to stabilize CI (NVIDIA#9398) Signed-off-by: Pengbo Wang <[email protected]> [None][chore] remove qwen3-next accuracy tests (NVIDIA#9534) Signed-off-by: jiant <[email protected]> [None][doc] fix mtp.py typo (NVIDIA#9307) Signed-off-by: liugaoji <[email protected]> [None][feat] add chat template kwargs support to longbench-v2 (NVIDIA#9544) Signed-off-by: Fanrong Li <[email protected]> [NVIDIA#9496][fix] AutoDeploy: remove auto-tuner from nvfp4_gemm forward (NVIDIA#9497) Signed-off-by: Neta Zmora <[email protected]> [None][fix] Replace hash method with unique_id for cutedsl MoE runners. (NVIDIA#9569) Signed-off-by: Yukun He <[email protected]> [None][chore] refactor disaggregated scripts to use named arguments (NVIDIA#9581) Signed-off-by: Zhenhuan Chen <[email protected]> [TRTLLM-6222][feat] Several perf opt for cuteDSL nvf4 gemm (NVIDIA#9428) Signed-off-by: Yuhan Li <[email protected]> [None][chore] reduce the layers of the `devel` docker image (NVIDIA#9077) Signed-off-by: Martin Marciniszyn Mehringer <[email protected]> [https://nvbugs/5651854][infra] Enable perf metrics during accuracy testing (NVIDIA#9140) [None][fix] Skip Allreduce init for Attention DP (NVIDIA#9542) Signed-off-by: Enwei Zhu <[email protected]> [None][test] [None][test] Waive main branch test failures 12/1 (NVIDIA#9566) Signed-off-by: Yanchao Lu <[email protected]> [None][ci] Minor change for Slurm scripts (NVIDIA#9561) Signed-off-by: Yanchao Lu <[email protected]> [TRTLLM-6768][infra] Fix params for not updating github status (NVIDIA#6747) Signed-off-by: Yiqing Yan <[email protected]> [None][infra] Update the pytest options after MI (NVIDIA#9579) Signed-off-by: qqiao <[email protected]> [TRTLLM-6756][feat] Add Beam Search to TorchSampler (NVIDIA#8509) Signed-off-by: Stefan Niebler <[email protected]> [None][chore] Defer exposing context parallel configs (NVIDIA#9552) Signed-off-by: Balaram Buddharaju <[email protected]> [TRTC-1943][feat] Env vars override support in LLM API (NVIDIA#9104) Signed-off-by: Venky Ganesh <[email protected]> [None][feat] AutoDeploy: Use the router gemm op for nemotron MOE (NVIDIA#9500) Signed-off-by: Chenghao Zhang <[email protected]> [NVIDIA#9198][feat] Refactor dist ops in AutoDeploy (NVIDIA#9301) Signed-off-by: Eran Geva <[email protected]> [None][fix] Prevent YAML partial kv_cache_config from incorrectly overriding the complete kv_cache_config (NVIDIA#9262) Signed-off-by: Yuening Li <[email protected]> [TRTLLM-9085][doc] fix math formula rendering issues in github (NVIDIA#9605) Signed-off-by: junq <[email protected]> [None][feat] Unify nvfp4 gemm backend (NVIDIA#8963) Signed-off-by: Shijie Wang <[email protected]> Signed-off-by: Yukun He <[email protected]> Signed-off-by: Shijie <[email protected]> Co-authored-by: Yukun He <[email protected]> [None][feat] Add support for KVCache reuse for DSv32 (NVIDIA#9383) Signed-off-by: Iman Tabrizian <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [None][chroe] Polish qwen3-next modeling code. (NVIDIA#8902) Signed-off-by: nv-guomingz <[email protected]> [https://nvbugs/5703953][fix] Use random port for disagg tests (NVIDIA#9582) Signed-off-by: Junyi Xu <[email protected]> [None][fix] Waive gb200 (NVIDIA#9580) Signed-off-by: Xin He (SW-GPU) <[email protected]> [FMDL-1328][feat] Add support for nano-v3 and super-v3 with pytorch backend (NVIDIA#9261) Signed-off-by: Wanli Jiang <[email protected]> [https://nvbugs/5582091][test] increase warmup times in testing for multi-gpu cases (NVIDIA#9578) Signed-off-by: Ruodi Lu <[email protected]> Co-authored-by: Ruodi Lu <[email protected]> [None][chore] Add failed cases into waives.txt (NVIDIA#9588) Signed-off-by: xinhe-nv <[email protected]> [https://nvbugs/5702793][fix] Fix uncontiguous tensor view (NVIDIA#9576) Signed-off-by: shuyix <[email protected]> [None][infra] Waive failed cases for main branch (NVIDIA#9615) Signed-off-by: qqiao <[email protected]> [TRTLLM-9488][feat] use FlashInfer.sampling by default (NVIDIA#9545) Signed-off-by: ixlmar <[email protected]> [None][infra] Update allowlist 2025/12/01 (NVIDIA#9616) Signed-off-by: Yuanjing Xue <[email protected]> [None][infra] Remove an invalid test name in waives.txt (NVIDIA#9620) Signed-off-by: qqiao <[email protected]> Lock the gpu clocks in L0 perf tests (NVIDIA#9585) Signed-off-by: Eran Geva <[email protected]> [TRTLLM-9466][test] Evaluate helix parallelism with DSV3 Lite (NVIDIA#9597) Signed-off-by: Balaram Buddharaju <[email protected]> [None][fix] Extract GPU count from single-node stage names (NVIDIA#9599) Signed-off-by: Chang Liu (Enterprise Products) <[email protected]> [https://nvbugs/5667774][fix] Refine Piecewise Cuda Graph Condition for DP (NVIDIA#9393) Signed-off-by: Jin Li <[email protected]> [TRTLLM-9144][fix] enhance RPC robustness (NVIDIA#8711) Signed-off-by: Superjomn <[email protected]> Signed-off-by: Erin Ho <[email protected]> Signed-off-by: Yan Chunwei <[email protected]> Co-authored-by: Erin Ho <[email protected]> [https://nvbugs/5627710][fix] Fix synchronization bugs in KvCacheTransferManager that can cause corrupted blocks (NVIDIA#9056) Signed-off-by: thorjohnsen <[email protected]> Signed-off-by: Thor Johnsen <[email protected]> Co-authored-by: Iman Tabrizian <[email protected]> Co-authored-by: Robin Kobus <[email protected]> [TRTLLM-8980][test] Clean up spec dec tests in test_llm_api_pytorch (NVIDIA#8889) Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [NVIDIA#9150][feat] Add code for nano v3 to custom implementation in AD (NVIDIA#9465) * Why? We would like to show an alternative to monkey-patching in AutoDeploy. * What? This commit builds on the existing custom model implementation for NemotronH and adds the bits relevant for MoE layers. Part of NVIDIA#9150. Signed-off-by: William Zhang <[email protected]> [NVIDIA#9150][feat] AutoDeploy: reviewer comments for NVIDIA#9150 (NVIDIA#9527) Signed-off-by: Lucas Liebenwein <[email protected]> [https://nvbugs/5651854][fix] Fix dist-serving perf by clearing CPU affinity (NVIDIA#9549) Signed-off-by: Shixiaowei02 <[email protected]> [NVIDIA#9550][feat] AutoDeploy: Add NVFP4 Cutlass MoE kernels (NVIDIA#9551) Signed-off-by: Neta Zmora <[email protected]> [https://nvbugs/5688388][fix] fix: Reducing num request in disagg test to speed up (NVIDIA#9598) Signed-off-by: Patrice Castonguay <[email protected]> [TRTLLM-8946][feat] Improved heuristics to detect shardable regions (NVIDIA#9200) Signed-off-by: Lucas Liebenwein <[email protected]> Signed-off-by: greg-kwasniewski1 <[email protected]> Co-authored-by: Lucas Liebenwein <[email protected]> [NVIDIA#9632][feat] Support EXTRA_WHEEL_BUILD_ARGS during wheel build (NVIDIA#9633) Signed-off-by: Yu Chi Li <[email protected]> [None][chore] Waive test failing on pre-merge (NVIDIA#9638) Signed-off-by: Balaram Buddharaju <[email protected]> [None][chore] Remove traceback dump for multimodal input processor (NVIDIA#9634) Signed-off-by: Chang Liu (Enterprise Products) <[email protected]> [None][chore] Fix trtllm-eval and move GroupedGemmInputsHelper (NVIDIA#9612) Signed-off-by: Enwei Zhu <[email protected]> [https://nvbugs/5698434][fix] Use separate weight mapper for draft (NVIDIA#9607) Signed-off-by: Anurag Mukkara <[email protected]> [TRTLLM-7101][infra] Reuse passed tests (NVIDIA#6894) Signed-off-by: Yiqing Yan <[email protected]> Co-authored-by: Yanchao Lu <[email protected]> [None][test] Remove duplicate test cases (NVIDIA#9623) Signed-off-by: yufeiwu <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [None][feat] Add RocketKV usage doc and e2e accuracy test on LongBenchV2 (NVIDIA#9572) Signed-off-by: yuhangh <[email protected]> [TRTLLM-9242][doc] Add examples showcasing openai compatible APIs (NVIDIA#9520) Signed-off-by: Junyi Xu <[email protected]> [None][chore] AutoDeploy update cuda stream manager for multi-device (NVIDIA#9575) Signed-off-by: Suyog Gupta <[email protected]> [TRTLLM-9391][chore] Automatically estimate required workspace. (NVIDIA#9535) Signed-off-by: Bo Li <[email protected]> [https://nvbugs/5708475][fix] Fix e2e eval accuracy for helix parallelism (NVIDIA#9647) Signed-off-by: Balaram Buddharaju <[email protected]> [https://nvbugs/5561153][test] Fix log error for perf test (NVIDIA#9622) Signed-off-by: FredricZ-2007 <[email protected]> [TRTLLM-8241][feat] Aliasing to comply to LlmArgs (NVIDIA#9586) Signed-off-by: Pengyun Lin <[email protected]> [None][chore] Add failed cases into waives.txt (NVIDIA#9593) Signed-off-by: Jie Li <[email protected]> Co-authored-by: Jie Li <[email protected]> [TRTLLM-6842][feat] Support Response API for general purpose (NVIDIA#9392) Signed-off-by: Junyi Xu <[email protected]> [None][test] Update Qwen3-next accuracy testing by setting the cuda … (NVIDIA#9613) Signed-off-by: nv-guomingz <[email protected]> [None][feat] update trtllm-gen nvfp4 kernels with better performance (NVIDIA#9510) Signed-off-by: Perkz Zheng <[email protected]> [None][doc] Replace the tensorrt icon with torch icon on overview.md (NVIDIA#9644) Signed-off-by: nv-guomingz <[email protected]> [https://nvbugs/5705197][chore] Unwaive timeout disagg tests (NVIDIA#9637) Signed-off-by: Patrice Castonguay <[email protected]> [https://nvbugs/5552132][fix] Enable LoRa for GPT OSS Torch (NVIDIA#8253) Signed-off-by: Michal Guzek <[email protected]> [None][fix] Fix wide ep MoE error (NVIDIA#9642) Signed-off-by: Iman Tabrizian <[email protected]> [https://nvbugs/5702795][fix] Remove the warning message for aten.log. (NVIDIA#9665) Signed-off-by: nv-guomingz <[email protected]> [https://nvbugs/5693853][fix] Fix error handling when querying machin… (NVIDIA#9483) Signed-off-by: Gal Hubara Agam <[email protected]> [OMNIML-2932] [feat] nvfp4 awq support (NVIDIA#8698) Signed-off-by: weimingc <[email protected]> [NVIDIA#9643][fix] AutoDeploy: fix nano sharding config (NVIDIA#9668) Signed-off-by: Lucas Liebenwein <[email protected]> [NVIDIA#9147][feat] AutoDeploy: Draft Target Speculative Decoding (NVIDIA#9275) Signed-off-by: Govind Ramnarayan <[email protected]> [None][feat] Update Qwen3CodeToolParser to align tool-calling parameters (NVIDIA#9540) Signed-off-by: Wanli Jiang <[email protected]> [TRTLLM-7181][infra] Generate test results when pytest timeout happens (NVIDIA#9396) Signed-off-by: Yiqing Yan <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [TRTLLM-9522][fix] restore `trtllm-serve mm_embedding_serve` (NVIDIA#9669) [TRTLLM-5093][infra] Write env variables to a file in the interactive debug session (NVIDIA#6792) Signed-off-by: Yiqing Yan <[email protected]> [None][fix] fix error when processing batches containing both text and mm data (NVIDIA#8381) Signed-off-by: Nekofish-L <[email protected]> [TRTLLM-7073][feat] Support torch compile for PP for Llama and DeepSeekV3 (NVIDIA#7838) Signed-off-by: Jin Li <[email protected]> [None][feat] Add weights initialization and context phase parser to layer-wise benchmarks (NVIDIA#9667) Signed-off-by: Tailing Yuan <[email protected]> [TRTLLM-8274][feat] Check if executor is shutdown in /health entrypoint (NVIDIA#9057) Signed-off-by: Junyi Xu <[email protected]> [NVIDIA#8733][feat] Add Llama4 MoE handling to AutoDeploy (NVIDIA#9556) Signed-off-by: Tal Cherckez <[email protected]> Signed-off-by: tcherckez-nvidia <[email protected]> Co-authored-by: Neta Zmora <[email protected]> [None][ci] unwaive tests (NVIDIA#9651) Signed-off-by: Yan Chunwei <[email protected]> [None][feat] Add NIXL-LIBFABRIC support (NVIDIA#9225) Signed-off-by: Yoray Zack <[email protected]> Signed-off-by: zackyoray <[email protected]> [None][test] rename wide ep and disagg metric name in perf test (NVIDIA#9704) Signed-off-by: Ruodi Lu <[email protected]> Co-authored-by: Ruodi Lu <[email protected]> [https://nvbugs/5467531][fix] Unwaive fused_moe all to all test with … (NVIDIA#9617) Signed-off-by: Jin Li <[email protected]> [None][fix] Recover TRTLLM MoE Perf for DEP (NVIDIA#9562) Signed-off-by: Anthony Chang <[email protected]> [None][chore] Add failed cases into waives.txt (NVIDIA#9662) Signed-off-by: Xin He (SW-GPU) <[email protected]> Signed-off-by: xinhe-nv <[email protected]> Signed-off-by: Yanchao Lu <[email protected]> Co-authored-by: Yanchao Lu <[email protected]> [None][fix] Fix TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS for MTP/EAGLE (NVIDIA#9608) Signed-off-by: Aurelien Chartier <[email protected]> [None][infra] Add container notices and documentation (NVIDIA#9185) Signed-off-by: Parker Drake <[email protected]> [TRTLLM-5312][infra] Add triton trigger rules (NVIDIA#6440) Signed-off-by: Yiqing Yan <[email protected]> [None][doc] Add feature docs for helix parallelism (NVIDIA#9684) Signed-off-by: Balaram Buddharaju <[email protected]> [TRTLLM-9579][infra] Set mergeWaiveList stage UNSTABLE when there is any issue (NVIDIA#9692) Signed-off-by: Yiqing Yan <[email protected]> [None][doc] Added line about partial reuse (NVIDIA#7846) Signed-off-by: thorjohnsen <[email protected]> [TRTLLM-8920][feat] decouple disagg service from fastapi (NVIDIA#8714) Signed-off-by: Lizhi Zhou <[email protected]> [https://nvbugs/5633340][fix] start disagg workers and servers on free ports (NVIDIA#9694) Signed-off-by: Lizhi Zhou <[email protected]> [TRTLLM-9562] [doc] Add Deployment Guide for Kimi K2 Thinking on TensorRT LLM - Blackwell (NVIDIA#9711) Signed-off-by: Kaiyu Xie <[email protected]> [NVIDIA#9602][feat] AutoDeploy: Support TRTLLM Sampler (NVIDIA#9641) Signed-off-by: Govind Ramnarayan <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [None] [tests] Unwaive EPLB tests (NVIDIA#9625) Signed-off-by: Kaiyu Xie <[email protected]> [https://nvbugs/5518713][test] Refactor core test lists by merging with llm_perf_cluster.yml (NVIDIA#9714) Signed-off-by: yufeiwu <[email protected]> [TRTLLM-7136][feat] Update load_weights method to include mapping parameter in checkpoint loaders (NVIDIA#9583) Signed-off-by: Robin Kobus <[email protected]> [None][refactor] Improve request processing function in sampler (NVIDIA#9671) Signed-off-by: Robin Kobus <[email protected]> [https://nvbugs/5670672][fix] Fix flaky KV connector tests (NVIDIA#9676) Signed-off-by: jthomson04 <[email protected]> [None][infra] Update allowed list 20251204 (NVIDIA#9718) Signed-off-by: Yuanjing Xue <[email protected]> [None][feat] AutoDeploy: Perf optimization for Attention and rmsnorm (NVIDIA#9719) Signed-off-by: Chenghao Zhang <[email protected]> [None][chore] Waive flakey disagg tests (NVIDIA#9749) Signed-off-by: Mike Iovine <[email protected]> [https://nvbugs/5601682][fix] Fix cacheTransceiver hang (NVIDIA#9311) Signed-off-by: Iman Tabrizian <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [TRTLLM-9199][docs] KV Connector Docs (NVIDIA#9325) Signed-off-by: jthomson04 <[email protected]> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [TRTLLM-9160][doc] add doc to llm_runtime.py (NVIDIA#9482) Signed-off-by: Yan Chunwei <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [None][doc] VDR 1.0 trtllm-serve doc enhancement (NVIDIA#9443) Signed-off-by: Pengyun Lin <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [TRTLLM-9086][doc] Clean up TODOs in documentation (NVIDIA#9292) Signed-off-by: junq <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [TRTLLM-9157][doc] Guided decoding doc improvement (NVIDIA#9359) Signed-off-by: Enwei Zhu <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [None][infra] Updated Linux installation guide (NVIDIA#9485) Signed-off-by: Yiqing Yan <[email protected]> Co-authored-by: Yanchao Lu <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [TRTLLM-9075][doc] refine the slurm examples (NVIDIA#9548) Signed-off-by: Yan Chunwei <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [TRTLLM-9093][doc] update hyper links in overview (NVIDIA#9568) Signed-off-by: junq <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [TRTLLM-9092][doc] link to modelopt checkpoints in quick start guide (NVIDIA#9571) Signed-off-by: junq <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [None][fix] Fix triton moe load_weight (NVIDIA#9649) Signed-off-by: shuyix <[email protected]> [None][fix] fix a bug: deepseek_fp8_block_scales in TRTLLMGEN-MoE use 2D x_sf instead of 1D (NVIDIA#9658) Signed-off-by: xxi <[email protected]> [TRTLLM-9372][feat] Enable CuteDSL MoE with Large EP (NVIDIA#9592) Signed-off-by: Enwei Zhu <[email protected]> [TRTLLM-9522][chore] implement default `attach_multimodal_embeddings` (NVIDIA#9664) Signed-off-by: ixlmar <[email protected]> [TRTLLM-9660][feat] Convert cuteDSL GEMM to opt-in feature (NVIDIA#9682) Signed-off-by: Jonas Li <[email protected]> Co-authored-by: Kaiyu Xie <[email protected]> [None][fix] enable hmac in RPC (NVIDIA#9745) Signed-off-by: Superjomn <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [https://nvbugs/5703953][fix] Preserving ip:port for trtllm-serve before initializing llm (NVIDIA#9646) Signed-off-by: Junyi Xu <[email protected]> [None][infra] Waive failed cases for main branch on 12/07 (NVIDIA#9769) Signed-off-by: qqiao <[email protected]> [None][fix] Several minor fixes to CI setting (NVIDIA#9765) Signed-off-by: Yanchao Lu <[email protected]> [OMNIML-3036][doc] Re-branding TensorRT-Model-Optimizer as Nvidia Model-Optimizer (NVIDIA#9679) Signed-off-by: Chenjie Luo <[email protected]> [None][feat] Enable NCCL_SYMMETRIC as default fallback for AllReduce (NVIDIA#9314) Signed-off-by: Ludwig Schneider <[email protected]> [TRTLLM-9000][feat] Add multi-node Perf Tests into CI (NVIDIA#8800) Signed-off-by: Chenfei Zhang <[email protected]> [None][test] add ntp tolerance in time metrics verification (NVIDIA#9741) Signed-off-by: zhengd-nv <[email protected]> [TRTLLM-9603][feat] Enable ConfigurableMoE test in the CI (NVIDIA#9645) [https://nvbugs/5422621][test] Add GB 200 WIDEEP test case for RCCA 5422621 (NVIDIA#9506) Signed-off-by: FredricZ-2007 <[email protected]> [None][fix] Fix two tuning cache miss issues. (NVIDIA#9743) Signed-off-by: Yukun He <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [TRTLLM-9706] [doc] Update wide EP documents (NVIDIA#9724) Signed-off-by: Kaiyu Xie <[email protected]> [https://nvbugs/5666804][test] only adding sampler config for limited models (NVIDIA#9512) Signed-off-by: Ruodi Lu <[email protected]> Co-authored-by: Ruodi Lu <[email protected]> Co-authored-by: yufeiwu-nv <[email protected]> Co-authored-by: Larry Xu <[email protected]> [None][infra] Waive failed cases for main on 12/08 (NVIDIA#9773) Signed-off-by: qqiao <[email protected]> [None][chore] Move the rocketkv e2e test to post-merge (NVIDIA#9768) Signed-off-by: Fanrong Li <[email protected]> [None][chore] Enable tvm_ffi for cute dsl nvfp4_gemm to reduce host overhead. (NVIDIA#9690) Signed-off-by: Mindy Li <[email protected]> [TRTLLM-9431][perf] Enable multistream for Linear Attention in Qwen3-… (NVIDIA#9696) Signed-off-by: nv-guomingz <[email protected]> [None][chore] Remove closed bugs (NVIDIA#9770) Signed-off-by: xinhe-nv <[email protected]> [None][infra] update mooncake in docker images (NVIDIA#9584) Signed-off-by: zhengd-nv <[email protected]> Signed-off-by: Zheng Duan <[email protected]> [None][test] Add Kimi k2 WIDEEP perf and accuracy cases (NVIDIA#9686) Signed-off-by: FredricZ-2007 <[email protected]> Signed-off-by: Kaiyu Xie <[email protected]> Co-authored-by: Kaiyu Xie <[email protected]> [https://nvbugs/5527655][test] Add test case for RCCA 5527655 (NVIDIA#9511) Signed-off-by: FredricZ-2007 <[email protected]> [http://nvbugs/5649010][fix] fix test_auto_scaling.py::test_worker_restart timeout (NVIDIA#9775) Signed-off-by: Lizhi Zhou <[email protected]> [None][fix] Switch AutoDeploy's default allreduce strategy to NCCL (NVIDIA#9666) Signed-off-by: Eran Geva <[email protected]> [TRTLLM-9506][fix] Fix AR for DeepSeek-R1 2 model path (NVIDIA#9661) Signed-off-by: qgai <[email protected]> ray + updatew works trtllm works in async env trtllm works in sync and async env ray + updatew works rebase to the updated verl server mode still cherry pick still cherry pick still cherry pick integrated http interface hang at RyExecutor create workers ray.remote clean code use tensorrt_llm.rlhf_utils Signed-off-by: Liwei Ma <[email protected]> placement, asyncllm, and basic tests Signed-off-by: Erin Ho <[email protected]> connect sleep and wakeup; Add support to pass None to update_weights Signed-off-by: Erin Ho <[email protected]> Batching ctx for IFB scheduler Signed-off-by: Yuan Tong <[email protected]> accuracy WAR for TP>1: always use AllReduceStrategy.NCCL, refactored Signed-off-by: Erin Ho <[email protected]> fix e2e integration Signed-off-by: Superjomn <[email protected]> update asyncllm, other nits Signed-off-by: Erin Ho <[email protected]> fix init setup Signed-off-by: Erin Ho <[email protected]> Fix TRTLLMSampler logprobs perf Signed-off-by: Yuan Tong <[email protected]> fix and cleanup Signed-off-by: Erin Ho <[email protected]> fix server Signed-off-by: Erin Ho <[email protected]> Revert "Batching ctx for IFB scheduler" This reverts commit b51aac0 Signed-off-by: Yuan Tong <[email protected]> update & address comments Signed-off-by: Erin Ho <[email protected]>
Signed-off-by: Lucas Liebenwein <[email protected]>
3 participants
Description
Support for https://huggingface.co/nvidia/Nemotron-Flash-3B-Instruct
fixes #9150
Try it out yourself
Build from source on this branch
Check out docs
Example
Run an example prompt
trtllm-serveSpin up a server
More infos in the docs.
Send a request to the server
More infos in the docs
Summary by CodeRabbit
Release Notes
New Features
Documentation
Performance
✏️ Tip: You can customize this high-level summary in your review settings.
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...Provide a user friendly way for developers to interact with a Jenkins server.
Run
/bot [-h|--help]to print this help message.See details below for each supported subcommand.
Details
run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]Launch build/test pipelines. All previously running jobs will be killed.
--reuse-test (optional)pipeline-id(OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.--disable-reuse-test(OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.--disable-fail-fast(OPTIONAL) : Disable fail fast on build/tests/infra failures.--skip-test(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.--stage-list "A10-PyTorch-1, xxx"(OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.--gpu-type "A30, H100_PCIe"(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.--test-backend "pytorch, cpp"(OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.--only-multi-gpu-test(OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.--disable-multi-gpu-test(OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.--add-multi-gpu-test(OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.--post-merge(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx"(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".--detailed-log(OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.--debug(OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in thestage-listparameter to access the appropriate container environment. Note: Does NOT update GitHub check status.For guidance on mapping tests to stage names, see
docs/source/reference/ci-overview.mdand the
scripts/test_to_stage_mapping.pyhelper.kill
killKill all running builds associated with pull request.
skip
skip --comment COMMENTSkip testing for latest commit on pull request.
--comment "Reason for skipping build/test"is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.reuse-pipeline
reuse-pipelineReuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.