[None][fix] Fix LoRA support for Qwen3 models#12785
Conversation
Signed-off-by: Aurelien Chartier <[email protected]>
|
/bot run --disable-fail-fast |
📝 WalkthroughWalkthroughUpdates to Qwen3 model initialization to pass layer index information to GatedMLP components and forward additional keyword arguments through decoder layers. Adds comprehensive LoRA sanity test coverage for both dense and MoE Qwen3 configurations, including adapter creation, inference execution with and without LoRA, and output validation. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes 🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tests/unittest/_torch/modules/tests_lora_modules/test_qwen3_sanity.py (1)
128-149: Minor: Inner loop break doesn't exit outer loop.When
any_differis set in the inner logprobs comparison loop (line 146), the code breaks out of the inner loop but continues iterating through remaining(lora_out, base_out)pairs unnecessarily. This is functionally correct but slightly inefficient.♻️ Optional: Use labeled loop pattern or helper
def _assert_lora_changes_output(out_lora, out_base): """Assert that LoRA produces at least one different output (tokens or logprobs).""" any_differ = False for lora_out, base_out in zip(out_lora, out_base, strict=True): + if any_differ: + break lora_ids = lora_out.outputs[0].token_ids base_ids = base_out.outputs[0].token_ids if lora_ids != base_ids: any_differ = True break # Even if tokens match, logprobs should differ lp_lora = lora_out.outputs[0].logprobs lp_base = base_out.outputs[0].logprobs if lp_lora and lp_base: for lp_w, lp_wo in zip(lp_lora, lp_base, strict=True): val_w = next(iter(lp_w.values())).logprob val_wo = next(iter(lp_wo.values())).logprob if abs(val_w - val_wo) > 1e-6: any_differ = True break + if any_differ: + break assert any_differ, "LoRA outputs identical to base model (same tokens AND same logprobs)"🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/unittest/_torch/modules/tests_lora_modules/test_qwen3_sanity.py` around lines 128 - 149, The helper _assert_lora_changes_output currently sets any_differ inside the inner logprobs loop then breaks only that inner loop, causing the outer for-loop over (lora_out, base_out) to continue unnecessarily; to fix, make the function return immediately when a difference is detected (or set any_differ and break the outer loop explicitly) so we stop processing further pairs as soon as a differing token or logprob is found—update the inner loop where val_w vs val_wo triggers any_differ to either call return or break out of both loops (e.g., by returning from _assert_lora_changes_output) to short-circuit the check.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@tests/unittest/_torch/modules/tests_lora_modules/test_qwen3_sanity.py`:
- Around line 128-149: The helper _assert_lora_changes_output currently sets
any_differ inside the inner logprobs loop then breaks only that inner loop,
causing the outer for-loop over (lora_out, base_out) to continue unnecessarily;
to fix, make the function return immediately when a difference is detected (or
set any_differ and break the outer loop explicitly) so we stop processing
further pairs as soon as a differing token or logprob is found—update the inner
loop where val_w vs val_wo triggers any_differ to either call return or break
out of both loops (e.g., by returning from _assert_lora_changes_output) to
short-circuit the check.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 94e6104d-c733-44c0-8865-a8ce78fa615c
📒 Files selected for processing (3)
tensorrt_llm/_torch/models/modeling_qwen3.pytensorrt_llm/_torch/models/modeling_qwen3_moe.pytests/unittest/_torch/modules/tests_lora_modules/test_qwen3_sanity.py
|
PR_Github #42005 [ run ] triggered by Bot. Commit: |
|
@CodeRabbit review |
✅ Actions performedReview triggered.
|
|
PR_Github #42005 [ run ] completed with state |
Summary by CodeRabbit
Tests
Improvements
Description
layer_idxtogatedMLPconstructor call**kwargsto decoder layer callTest Coverage
New tests: tests/unittest/_torch/modules/tests_lora_modules/test_qwen3_sanity.py
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.