Cannot use individual video crops for refinement? #2036
Description
Is there an existing issue for this?
- I have searched the existing issues
Bug description
Hi all -- thank you for maintaining this code base so well! I have some videos where the mouse occupies about 1/4 of the frame, so I cropped them manually during initial frame extraction. I uploaded my videos to a computing cluster to run the training, and it worked quite well with the cropping. The relevant part of the config looks something like:
video_sets:
? /laptop/path/to/basler_2022-10-10T19_24_54.mp4
: crop: 268, 827, 269, 823
? /laptop/path/to/basler_2022-10-10T20_46_35.mp4
: crop: 207, 794, 280, 829
[etc]
[and then the same again but with paths on the cluster]
? /cluster/path/to/basler_2022-10-10T19_24_54.mp4
: crop: 268, 827, 269, 823
? /cluster/path/to/basler_2022-10-10T20_46_35.mp4
: crop: 207, 794, 280, 829
[etc]
I used displaycropped=True to make labeled videos, and they look almost perfect! But I had some outliers in the videos, so I decided to do some refinement with dlc.extract_outlier_frames. It seems like the refinement step has a separate cropping parameter in the config file, which only allows for one set of crop params to be set at a time. Naively I expected that if the video_sets had video-specific crops, it would just use those instead, but no luck. The extracted outlier frames look like the frames from #1885 (ie the labels are all clearly offset by the amount of the crop). Oddly, the config file also gets overwritten, such that the cluster-side paths now show no cropping at all:
? /cluster/path/to/basler_2022-10-10T19_24_54.mp4
: crop: 0, 1280, 0, 1024
[etc]
Is there any way to send video-specific crop parameters into the refinement? If no, I may end up writing a script to make a config copy for each video and pass the correct crop to the refinement-specific crop parameters, which I can share here -- but seems like since cropping is part of the upstream pipeline, there should be a feature to use it in refinement as well? Apologies if I've just missed it or am doing something wrong. Thank you!
(And ultimately, the lesson here may be to crop my videos during acquisition! But thought I'd open an issue anyways in case others have this issue.)
Operating System
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
3.10.0-1160.76.1.el7.x86_64
DeepLabCut version
2.2.3
DeepLabCut mode
single animal
Device type
This was done with a CPU.
Steps To Reproduce
dlc.extract_outlier_frames(config, vid, outlieralgorithm='jump', epsilon=200, extractionalgorithm='uniform', automatic=True)
Relevant log output
Loading DLC 2.2.3...
DLC loaded in light mode; you cannot use any GUI (labeling, relabeling and standalone GUI)
/home/jop9552/miniconda3/envs/DEEPLABCUT/lib/python3.8/site-packages/deeplabcut/__init__.py:81: UserWarning:
As PyTorch is not installed, unsupervised identity learning will not be available.
Please run `pip install torch`, or ignore this warning.
warnings.warn(
/home/jop9552/miniconda3/envs/DEEPLABCUT/lib/python3.8/site-packages/deeplabcut/refine_training_dataset/outlier_frames.py:401: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().
sum_ = temp_dt.sum(axis=1, level=1)
Method jump found 252 putative outlier frames.
Do you want to proceed with extracting 20 of those?
If this list is very large, perhaps consider changing the parameters (start, stop, p_bound, comparisonbodyparts) or use a different method.
Frames from video basler_2022-10-13T20_25_58 already extracted (more will be added)!
Loading video...
Duration of video [s]: 4136.416666666667 , recorded @ 120.0 fps!
Overall # of frames: 496370 with (cropped) frame dimensions:
Uniformly extracting of frames from 0.0 seconds to 4136.42 seconds.
Let's select frames indices: [246953, 148180, 377139, 148419, 246894, 377334, 148598, 377718, 104635, 246755, 246847, 377785, 148588, 377253, 246897, 104788, 377427, 377142, 246927, 148379]
Attempting to create a symbolic link of the video ...
/bin/sh: mklink: command not found
Symlink creation impossible (exFat architecture?): cutting/pasting the video instead.
/n/groups/datta/Jonah/Turbulator/dlc_projects/turbulator_conc_100_1000_gmou85_gmou86-Jonah-2022-10-21/videos/basler_2022-10-13T20_25_58.mp4 moved to /n/groups/datta/Jonah/Turbulator/dlc_projects/turbulator_conc_100_1000_gmou85_gmou86-Jonah-2022-10-21/videos/basler_2022-10-13T20_25_58.mp4
New videos were added to the project! Use the function 'extract_frames' to select frames for labeling.
The outlier frames are extracted. They are stored in the subdirectory labeled-data\basler_2022-10-13T20_25_58.
Once you extracted frames for all videos, use 'refine_labels' to manually correct the labels.Anything else?
No response
Code of Conduct
- I agree to follow this project's Code of Conduct