🔥 New main backend Deep Learning Library (Engine) for DeepLabCut: PyTorch #2613
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* improved augmenter selection * improved pytorch config docs * added documentation for missing parameters * remove unused param from DLCParams * bug fix: add a shuffle
- makes usage of TQDM in evaluate_assembly optional (so it doesn't log during training) - only computes rmse_pcutoff if there are keypoints above the cutoff (otherwise a warning was printed) - force the logger updates when starting to train, otherwise it could be buggy - don't use `verbose`for the LR scheduler as it's deprecated - log a single time will all metrics instead of multiple - logs metrics and losses to different prefixes so they are put in the correct panels on wandb - log input images, target and output heatmaps to WandB
Merge public changes
* fixed superanimal video inference * added demo notebook for transfer learning
updated code to get scorer for pytorch models, fixed bugs for video analysis
* Fix NaN centroids with 0 likelihood * Minor compatibility fixes * Improved error message * Fix config dict key setting * Delete redundant neighbor search implementations
…n data augmentation, add tests (#190) * bug fix: incorrectly reading snapshot index from config in analyze videos * added files for backwards compatibility (e.g. when loading pickles) * update default top down augmentation * fixed tokenpose cfg * added tests for topdown config creation, fixed bug * add coarse dropout test, pin albu * switch default resnet50 to GN model * changed default resnet output stride to 16 * freeze BN stats but not weights as default * fixed tests * fix topdown aug
* Loading SuperAnimal weights for model training * various bug fixes
* bug fix when creating train dataset, updated code to use model stride correctly * stride can be float
* prepare for 3.0.0rc1 (bump version) * Update version.py * Update reinstall.sh * Update NOTICE.yml (pytorch license)
Co-authored-by: Niels Poulsen <[email protected]>
* added CSV logger * fixed imports
* Fix matching w/ single body parts * Minor fixes
niels: GUI bug fix, engine selector
* Tmp fix create_pretrained_project: create TF trainingset metadata * Type hints * Split create_pretrained_project() into tf and torch * Add individuals param to create_new_project() * Fix list_videos_in_folder() In original/tf methods (f.ex. analyze_videos), default value of videotype was "", this default value was not treated properly in list_videos_in_folder() * Refactor _create_and_save_fake_metadata * Refactor: _process_videos * Implement create_pretrained_project for torch * Update create_pretrained_project usages * Refactoring Move read_config_as_dict, write_config and pretty_print to core * Fix get_super_animal_project_cfg When calling read_config, the superanimal yaml config in the repo was modified * Write integration script * Fix other Superanimal tests * Run black
* improve testing * fix ID-based tracklet creation * fix ID metric computation when NaNs are present * fix id assignment during ID-only tracking
* Updates to the documentation - Maxim (#2869) * Updates to the docs (#2871) * add more docs in single animal userguide * unify train_network docs * unify train_network docs * improved augmentation code * remove link to 3mice TF Colab on homepage * Update readme and conda env * Update readme * docs/pytorch/user_guide Installation: point to docs/installation * Standardize subtitiles accross single and multi user guides * Add missing docstring * improved installation docs * added quick commands * improved docs * install torch and torchvision with pip in the DeepLabCut Conda file * updated labelling in userguides * Update DEEPLABCUT.yaml * install deeplabcut through the pre-release --------- Co-authored-by: Niels <[email protected]> Co-authored-by: Niels Poulsen <[email protected]> Co-authored-by: Mackenzie Mathis <[email protected]>
…ne (#2868) * update openfield COLAB * updated COLAB_YOURDATA_TrainNetwork_VideoAnalysis and COLAB_maDLC_TrainNetwork_VideoAnalysis * fix ReID notebook * fix ReID notebook * save predictions to destfolder * update superanimal notebook * fixed demo_data.py, Demo_labeledexample_MouseReaching.ipynb * set default batch size to 8, fix notebooks * updated notebooks * fix edited config * fix edited config * fix edited config * update 3mice demo * install deeplabcut through prerelease --------- Co-authored-by: Mackenzie Mathis <[email protected]>
* bug fix for bbox dict building * add option to export top-down models without detectors * add option to export top-down models without detectors * Update examples/openfield-Pranav-2018-10-30/config.yaml * Update examples/openfield-Pranav-2018-10-30/config.yaml --------- Co-authored-by: Mackenzie Mathis <[email protected]>
MMathisLab
changed the title
AlexEMG
approved these changes
Feb 28, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New main backend Deep Learning Library (Engine) for DeepLabCut: PyTorch 🔥
This post serves to provide the source code of DeepLabCut 3.0, which marks the first major release change since 2 (in late 2018). With this change, we move to PyTorch.
PyTorch combines the GPU-accelerated backend libraries from Torch with an intuitive Python frontend that focuses on rapid prototyping, readable code, and support for the widest possible variety of deep learning models.
It is high-performance, easier to use, easier to install, and developer-friendly.
Image reference https://pytorch.org/features/
Why:
We wanted to modernize our stack and also make it more user-friendly for all our stakeholders. Namely, the stakeholders in DeepLabCut are:
What is new:
We have dedicated documentation that walks you through the new options
(note, these links point to a PR, but will be in main docs upon merger)
User Guide & RoadMap
PyTorch configuration file
Neural Networks
What to expect:
Importantly, the high-level API stays the same 🪄.
The dev team worked very hard to keep both DeepLabCut PyTorch and (for the time) DeepLabCut TensorFlow for you and keep the workflow identical. Truly, you can use the same 12 steps you know and love! ❤️
In short:
You have the GUI, with the same launch code:
And at the command line interface you have the same 12 core steps!
Get started by testing this PR:
First, this is not yet on
pypi, so you will need to test this branch 💜. This is straightforward to do:(1) We recommend building a new
condaenvironment. In your anaconda terminal, please type:You will be prompted (y/n) to install, and then wait for the magic to happen. At the end, check the terminal, it should prompt you to then type:
Now, we are going to install the core dependencies. The way this works is that there are "package managers" such as
condaitself and python'spip. We are going to deploy a mix based on what we know works across ooperating systems.Install PyTorch
To select the right version, head to the “Install PyTorch” instructions in the official PyTorch Docs. Select your desired PyTorch build, operating system, and Python as the language. Select your compute platform (either a CUDA version or CPU only). Then, use the command to install the PyTorch package. Below are a few possible examples:
Install DeepLabCut PyTorch
Alright! Next, we will install
Tables(also called pytables), which is a package to read the HDF5 files that make up the backbone of data management in DeepLabCut, then we will install all thedeeplabcutsource code 🔥. Please type:Then launch DeepLabCut! for example, to use this in the GUI:
You will see version 3.0.0rc1 loading:
Loading DLC 3.0.0rc1...Please also see the related PR with the Documentation Updates: #2605
Benchmarking:
This will be continually updated, but in short, here is the report for the classic DLC-openfield dataset:
Benchmarking Results on DeepLabCut-Openfield data:
Please note the exact data splits are not identical to the TF 1.0 splits, but we provide metrics for ease of understanding the TL;DR, which is, we have better performance with the PyTorch Engine than the TensorFlow engine. The 0.5 data ratio below means training with 506 randomly drawn frames.
*Values from Mathis et al. 2018: https://www.nature.com/articles/s41593-018-0209-y
ResNet-50, 3.09 ± 0.04; ResNet-101, 2.90 ± 0.09; ResNet-101 with intermediate supervision, 2.88 ± 0.06; pixel mean ± s.e.m
Here are more benchmarking results on DLC-openfield considering various data ratios. The Data Ratio is the % of frames used for training. 0.01 is 10, 0.10 is 101, 0.5 is 506, 1 is 1012 frames. To note, for more information on these various SuperAnimal methods, please see Ye et al. 2023/2024 (arxiv/Nature Communications).
Benchmarking Results tri-mouse multi-animal:
Contributors:
This has been a huge team effort across several years. We will write more about this process in a forthcoming blog post, but in the meantime, we greatly thank the community, users, and these core authors:
DeepLabCut 3.0 Toolbox
Mackenzie Mathis, [email protected] | https://github.com/MMathisLab
Alexander Mathis, [email protected] | https://github.com/AlexEMG
Niels Poulsen, [email protected] | https://github.com/n-poulsen
Shaokai Ye, [email protected] | https://github.com/yeshaokai
Anastasiia Filippova, [email protected] | https://github.com/nastya236
Quentin Macé | https://github.com/QuentinJGMace
Jessy Lauer, [email protected] | https://github.com/jeylau
Lucas Stoffl, [email protected] | https://github.com/LucZot
We also greatly thank the 2023 DeepLabCut AI Residents who contributed:
Anna Teruel-Sanchis | https://github.com/anna-teruel
Riza Rae Pineda | https://github.com/rizarae-p
Konrad Danielewski | https://github.com/KonradDanielewski
Products:
PyTorch backend for DeepLabCut
Expanded SuperAnimal capabilities
New model architectures (WIP: stay tuned, but includes BUCTD)