Skip to content

fix(rtdetr): honor max_det in decoder exports and validation - #25153

Merged
glenn-jocher merged 7 commits into
ultralytics:mainfrom
Hasnaathussain:fix/rtdetr-max-det-redesign
Jul 13, 2026
Merged

glenn-jocher merged 7 commits into
ultralytics:mainfrom
Hasnaathussain:fix/rtdetr-max-det-redesign

Conversation

@Hasnaathussain

@Hasnaathussain Hasnaathussain commented Jul 13, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #3286

Summary

RT-DETR now honors max_det at the layers that own it:

  • Native prediction keeps all decoder queries, then applies confidence/class filtering and max_det in RTDETRPredictor.
  • Validation applies confidence filtering and max_det in RTDETRValidator.
  • Export limits decoder top-k to max_det, clamped by RT-DETR query capacity rather than YOLO anchor count.

The earlier proposal marked RT-DETR as a standard YOLO end-to-end head. That was too broad: training resume uses that flag to enter the one-to-one criterion update lifecycle, which RT-DETR does not implement.

Deleted: the proposed agnostic_nms and end2end RTDETRDecoder flags and their incorrect standard-lifecycle integration.

The source change is net-positive by 6 lines, plus 4 lines in an existing export test. Deletion alone cannot make exported decoder top-k configurable or apply validation post-filter limits; the remaining additions live at those two ownership boundaries.

Validation

  • Ruff format and lint pass on all changed files.
  • Decoder probes verify native custom-query models retain every candidate while export honors max_det.
  • Validator probes verify confidence filtering precedes max_det truncation.
  • The existing RT-DETR CoreML export test now asserts its 300-detection output dimension.
  • Fresh adversarial review rounds found and resolved the training-lifecycle, anchor-clamp, and native pre-filter truncation regressions; the final round returned LGTM.

@Hasnaathussain

Copy link
Copy Markdown
Contributor Author

I have read the CLA Document and I sign the CLA

@UltralyticsAssistant UltralyticsAssistant added bug Something isn't working as intended in the official Ultralytics package. detect Object Detection issues, PR's python Pull requests that update python code labels Jul 13, 2026
@UltralyticsAssistant

Copy link
Copy Markdown
Member

👋 Hello @Hasnaathussain, thank you for submitting a ultralytics/ultralytics 🚀 PR! To ensure a seamless integration of your work, please review the following checklist:

  • ✅ Define a Purpose: Clearly explain the purpose of your fix or feature in your PR description, and link to any relevant issues. Ensure your commit messages are clear, concise, and adhere to the project's conventions.
  • ✅ Synchronize with Source: Confirm your PR is synchronized with the ultralytics/ultralytics main branch. If it's behind, update it by clicking the 'Update branch' button or by running git pull and git merge main locally.
  • ✅ Ensure CI Checks Pass: Verify all Ultralytics Continuous Integration (CI) checks are passing. If any checks fail, please address the issues.
  • ✅ Update Documentation: Update the relevant documentation for any new or modified features.
  • ✅ Add Tests: If applicable, include or update tests to cover your changes, and confirm that all tests are passing.
  • ✅ Sign the CLA: Please ensure you have signed our Contributor License Agreement if this is your first Ultralytics PR by writing "I have read the CLA Document and I sign the CLA" in a new message.
  • ✅ Minimize Changes: Limit your changes to the minimum necessary for your bug fix or feature addition. "It is not daily increase but daily decrease, hack away the unessential. The closer to the source, the less wastage there is." — Bruce Lee

For more guidance, please refer to our Contributing Guide. This is an automated welcome message, and an Ultralytics engineer will assist with the review. Don't hesitate to leave a comment if you have any questions. Thank you for contributing to Ultralytics! 🚀

@UltralyticsAssistant UltralyticsAssistant left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 PR Review

Made with ❤️ by Ultralytics Actions

Clean PR. The added head attributes are consumed by the standard model lifecycle, and the top-k change safely caps RT-DETR decoder outputs at max_det while preserving the existing query/class selection behavior.

@codecov

codecov Bot commented Jul 13, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.33333% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
ultralytics/models/rtdetr/val.py 0.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@glenn-jocher glenn-jocher changed the title fix(rtdetr): integrate RTDETRDecoder into standard end2end model lifecycle fix(rtdetr): honor max_det in decoder exports and validation Jul 13, 2026
@glenn-jocher
glenn-jocher merged commit cdccf05 into ultralytics:main Jul 13, 2026
19 checks passed
@glenn-jocher

glenn-jocher commented Jul 13, 2026 •

Copy link
Copy Markdown
Member

@Hasnaathussain Thanks for the contribution! We scoped max_det to the RT-DETR paths that own it: export uses decoder query capacity, while native prediction and validation limit detections after filtering. We removed the broader end2end lifecycle flags because RT-DETR does not use the standard YOLO criterion-update lifecycle. This is now merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working as intended in the official Ultralytics package. detect Object Detection issues, PR's python Pull requests that update python code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

max_det parameter not working in rt-detr

3 participants