Skip to content

New Top-Down Pose Estimation Model - RTMPose #2795

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
n-poulsen merged 53 commits into from
Dec 20, 2024
Merged

New Top-Down Pose Estimation Model - RTMPose #2795

n-poulsen merged 53 commits into from
Dec 20, 2024

Conversation

@n-poulsen
Copy link
Contributor

@n-poulsen n-poulsen commented Nov 25, 2024 •
edited by MMathisLab
Loading

This pull requests adds the implementation for RTMPose: Real-Time Multi-Person Pose Estimation.

Full changes:

  • New mixin for backbones with pretrained weights on HuggingFace : HuggingFaceWeightsMixin
  • New backbone: CSPNeXt
  • New head: RTMCCHead
  • New target generator: SimCCGenerator
  • New predictor: SimCCPredictor
  • New weight initializer: Rtmpose
  • New RandomBBoxTransform augmentation for top-down models (jitters ground truth bounding boxes)
  • Configs for CSPNeXt backbones: cspnext_s, cspnext_m, cspnext_x
  • Configs for RTMPose models: rtmpose_s, rtmpose_m, rtmpose_x
  • Updated way to scheduler configurations, to allow for the use of SequentialLR (to match the RTMPose training schedulers)
  • Added documentation: using a SequentialLR scheduler for warm-up epochs
  • Updated the top-down crop to function with no-square crops (and re-implemented with numpy)
    • Added top_down_crop_size: (crop_w, crop_h) and top_down_crop_margin in the pytorch_config.yaml file for an easy way to control the size of the crops for top-down models.
  • Bug fix: bbox_from_keypoints when there's more than 1 individual (bug had no impact as method was always called in a way it wouldn't fail)

DEMO

ski-dlc.mov

@n-poulsen
added rtmpose head
3678c57
@n-poulsen
added CSPNext backbone
094ba6f
@n-poulsen
add code to load pretrained CSPNext weights
ebebb3b
@n-poulsen
update number of backbone channels
ed0cbee
@n-poulsen
update weight init
6a66683
@n-poulsen
added possibility for SequentialLR, tests, docs, default for RTMPose
34f1baf
@n-poulsen
fix scheduler/optimizer config update
f737e53
@n-poulsen
fix tests
62e288f
@n-poulsen
made top down crop work on non-square images, in numpy to make faster :)
0b49323
@n-poulsen
added example to run top-down video analysis without the detector
6ed1ab3
@n-poulsen
improved inits
8aab808
@n-poulsen
edit rtmpose_x for default square bbox
f13f88e
@n-poulsen
add random bbox transform; fix top_down_crop
5add563
@n-poulsen
add missing transform method
a8470d2
@n-poulsen
fix transform - label must be int
27679eb
@n-poulsen
only apply top-down crop on non-empty image
5ae8bea
@n-poulsen
bug fix: collating scales
57da127
@n-poulsen
set image dtype when no bounding box
49c6a4a
@n-poulsen
fix simcc keypoint weights when -1 or 2
eebbfe5
@n-poulsen
update scheduler settings for rtmpose
083dcba
@n-poulsen
fix rtmpose-m config
5d2dc6c
@n-poulsen
bug fix: bbox from keypoints
91e40eb
@n-poulsen
update lr and switch to linear warmup
1697e12
@n-poulsen
bug fix - input size
8e6ad27
@n-poulsen
added documentation
3635698
@n-poulsen
added code to download backbone weights from huggingface
76b1414
@n-poulsen
bug fix; added s config
1bd0564
@n-poulsen
fix LR to 5e-05 after cosine annealing LR
5f9dd9c
@n-poulsen
init learning rate to 5e-3
26d9227
@n-poulsen
Merge branch 'pytorch_dlc' into niels/rtmpose
fe88d4f
@MMathisLab MMathisLab requested a review from AlexEMG December 5, 2024 10:58
maximpavliv
maximpavliv suggested changes Dec 10, 2024
View reviewed changes
maximpavliv
maximpavliv approved these changes Dec 18, 2024
View reviewed changes
Copy link
Contributor

@maximpavliv maximpavliv left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great!!

@n-poulsen n-poulsen merged commit 1d26325 into pytorch_dlc Dec 20, 2024
1 check passed
@n-poulsen n-poulsen deleted the niels/rtmpose branch December 20, 2024 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AlexEMG AlexEMG Awaiting requested review from AlexEMG

1 more reviewer

@maximpavliv maximpavliv maximpavliv approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

DLC3.0🔥

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@n-poulsen @maximpavliv @MMathisLab