Skip to content

Add D-FINE Model into Transformers#35400

Closed
VladOS95-cyber wants to merge 124 commits into
huggingface:mainfrom
VladOS95-cyber:add-d-fine-model
Closed

Add D-FINE Model into Transformers#35400
VladOS95-cyber wants to merge 124 commits into
huggingface:mainfrom
VladOS95-cyber:add-d-fine-model

Conversation

@VladOS95-cyber

@VladOS95-cyber VladOS95-cyber commented Dec 23, 2024

Copy link
Copy Markdown
Contributor

What does this PR do?

This PR add D-FINE into the Transformers library. There is a new thing in transformers called modular, which adds new models by creating a modeling_modelname.py file. Since D-FINE updates several RT-DETR arch parts while keeping the rest of the model unchanged, it serves as an ideal use case for this modular approach.

Before submitting

Who can review?

@qubvel @Rocketknight1 @ArthurZucker

@Rocketknight1

Copy link
Copy Markdown
Member

cc @qubvel @NielsRogge

@VladOS95-cyber
VladOS95-cyber force-pushed the add-d-fine-model branch 2 times, most recently from aaeb37c to 3168616 Compare February 6, 2025 15:49
@VladOS95-cyber
VladOS95-cyber marked this pull request as ready for review February 6, 2025 15:49
@VladOS95-cyber

Copy link
Copy Markdown
Contributor Author

Model implementation is done. Unit tests are still missing. I am working on it

@qubvel qubvel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @VladOS95-cyber! Thanks a lot for working on the model, super excited to see it merged! It’s a state-of-the-art real-time object detection model at the moment 🚀

Before diving into the implementation details, here are some general comments we need to address in the PR:

  • Integration Tests
    Let's set up integration tests to ensure output consistency is maintained while refactoring the modeling code. Please use torch.testing.assert_close when comparing tensors (not use self.assertTrue(torch.allclose(...))

  • Modeling Code & RT-DETRv2
    Great job keeping using modular! I see it's based on RT-DETR, but RT-DETRv2 was just merged yesterday. It might be helpful, especially since I noticed the v2 attention is used.

  • Model Organization
    Each model should have its own folder. We should separate the backbone model and the D-Fine model, following the same structure as RT-DETR. Checked RT-DETR, it's actually 2 models in one folder, so we can leave it as is (in case you didn't change yet)

  • Consistent Naming for Classes & Modules

    • Model names and types: DFine and d_fine, DFineResnet and d_fine_resnet.
    • All module names should be prefixed with the model name. For example, instead of ResidualBlock, use DFineResnetResidualBlock.
  • Code Paths & Cleanup
    Remove unused conditional statements if no existing configuration requires them. Also, remove the unnecessary config parameters.

    # Instead of:
    if config.use_last_layer_norm:
        self.norm = nn.LayerNorm()
    else:
        self.norm = nn.Identity()
    # If `use_last_layer_norm = True` in all configs, simplify it to:  
    self.norm = nn.LayerNorm()
  • Code Style

    • Use clear, descriptive variable names: avoid single-letter (x) or placeholder (tmp) variables.
    • Add comments for non-obvious code.
    • Use type hints for modules (e.g. in __init__ and forward).
  • Conversion Script Standard
    Please follow the mllama model format for the conversion script. Define a single key mapping using regex (see rt_detr_v2, ijepa, mllama, superglue models for reference). This is our new standard for all newly added models.

Thanks again! Looking forward to the updates 🚀

@VladOS95-cyber

Copy link
Copy Markdown
Contributor Author

Hi @qubvel! Thank you for your comment! So, I added some integration tests, refactored modules and changed conversion script according to the latest standards.

@VladOS95-cyber

Copy link
Copy Markdown
Contributor Author

Hi @qubvel! I fixed all failing tests except repository consistency, because during resnet modular conversion, the script add additional resnet postfix into configuration import path in modeling_d_fine_resnet file like from .configuration_d_fine_resnet_resnet import DFineResNetConfig. I manually fixed that, but honestly, I have no idea why modular conversion script behave like that..

@qubvel

qubvel commented Feb 12, 2025

Copy link
Copy Markdown
Contributor

Hey @VladOS95-cyber, thanks a lot for the update! I have a sprint, but I will try to review it as soon as have any bandwidth!
Regarding modular conversion, it might have some issues, it's still in active development.

cc @Cyrilvallez re modular issues

@qubvel qubvel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @VladOS95-cyber, I made an initial review for modular_d_fine. Looks much better, thanks for the update! See the comment below, and all these also related to backbone file as well

Comment thread src/transformers/models/d_fine/modular_d_fine.py Outdated
Comment thread src/transformers/models/d_fine/modular_d_fine.py Outdated
Comment thread src/transformers/models/d_fine/modular_d_fine.py Outdated
Comment thread src/transformers/models/d_fine/modular_d_fine.py Outdated
Comment thread src/transformers/models/d_fine/modular_d_fine.py Outdated
Comment thread src/transformers/models/d_fine/modular_d_fine.py Outdated
Comment thread src/transformers/models/d_fine/modular_d_fine.py Outdated
Comment thread src/transformers/models/d_fine/modular_d_fine.py Outdated
Comment thread src/transformers/models/d_fine/modular_d_fine.py Outdated
Comment thread src/transformers/models/d_fine/modular_d_fine.py Outdated
@VladOS95-cyber

Copy link
Copy Markdown
Contributor Author

Hi @qubvel! I resolved your comments and did some refactoring and cleanup

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants