Add D-FINE Model into Transformers#35400
Conversation
f98051f to
980c54f
Compare
1ab773f to
94f615e
Compare
94f615e to
ca14545
Compare
ca14545 to
d0e9212
Compare
d0e9212 to
af3103d
Compare
aaeb37c to
3168616
Compare
|
Model implementation is done. Unit tests are still missing. I am working on it |
a8b54d8 to
c588b0d
Compare
There was a problem hiding this comment.
Hi @VladOS95-cyber! Thanks a lot for working on the model, super excited to see it merged! It’s a state-of-the-art real-time object detection model at the moment 🚀
Before diving into the implementation details, here are some general comments we need to address in the PR:
-
Integration Tests
Let's set up integration tests to ensure output consistency is maintained while refactoring the modeling code. Please usetorch.testing.assert_closewhen comparing tensors (not useself.assertTrue(torch.allclose(...)) -
Modeling Code & RT-DETRv2
Great job keeping using modular! I see it's based on RT-DETR, but RT-DETRv2 was just merged yesterday. It might be helpful, especially since I noticed thev2attention is used. -
Model Organization
Each model should have its own folder. We should separate the backbone model and the D-Fine model, following the same structure as RT-DETR.Checked RT-DETR, it's actually 2 models in one folder, so we can leave it as is (in case you didn't change yet) -
Consistent Naming for Classes & Modules
- Model names and types:
DFineandd_fine,DFineResnetandd_fine_resnet. - All module names should be prefixed with the model name. For example, instead of
ResidualBlock, useDFineResnetResidualBlock.
- Model names and types:
-
Code Paths & Cleanup
Remove unused conditional statements if no existing configuration requires them. Also, remove the unnecessary config parameters.# Instead of: if config.use_last_layer_norm: self.norm = nn.LayerNorm() else: self.norm = nn.Identity()
# If `use_last_layer_norm = True` in all configs, simplify it to: self.norm = nn.LayerNorm()
-
Code Style
- Use clear, descriptive variable names: avoid single-letter (
x) or placeholder (tmp) variables. - Add comments for non-obvious code.
- Use type hints for modules (e.g. in
__init__andforward).
- Use clear, descriptive variable names: avoid single-letter (
-
Conversion Script Standard
Please follow themllamamodel format for the conversion script. Define a single key mapping using regex (seert_detr_v2,ijepa,mllama,supergluemodels for reference). This is our new standard for all newly added models.
Thanks again! Looking forward to the updates 🚀
c588b0d to
b10e399
Compare
|
Hi @qubvel! Thank you for your comment! So, I added some integration tests, refactored modules and changed conversion script according to the latest standards. |
0680723 to
f92cc4a
Compare
|
Hi @qubvel! I fixed all failing tests except repository consistency, because during resnet modular conversion, the script add additional resnet postfix into configuration import path in modeling_d_fine_resnet file like from .configuration_d_fine_resnet_resnet import DFineResNetConfig. I manually fixed that, but honestly, I have no idea why modular conversion script behave like that.. |
|
Hey @VladOS95-cyber, thanks a lot for the update! I have a sprint, but I will try to review it as soon as have any bandwidth! cc @Cyrilvallez re |
a85d38d to
e980d0b
Compare
qubvel
left a comment
There was a problem hiding this comment.
Hey @VladOS95-cyber, I made an initial review for modular_d_fine. Looks much better, thanks for the update! See the comment below, and all these also related to backbone file as well
e980d0b to
ab9306b
Compare
|
Hi @qubvel! I resolved your comments and did some refactoring and cleanup |
What does this PR do?
This PR add D-FINE into the Transformers library. There is a new thing in transformers called modular, which adds new models by creating a modeling_modelname.py file. Since D-FINE updates several RT-DETR arch parts while keeping the rest of the model unchanged, it serves as an ideal use case for this modular approach.
Before submitting
Pull Request section?
to it if that's the case. Link: Request to add D-FINE #35283
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@qubvel @Rocketknight1 @ArthurZucker