IndexError: list index out of range when using analyze_videos #2449
Description
Is there an existing issue for this?
- I have searched the existing issues
Bug description
Hi!
This is my first time using python and DeepLabCut, so any suggestions or comments are greatly appreciated! When I was trying to analyze the video file, it was running fine until it failed halfway. Then I tried the suggestion in #1527 but the same error occured again. Due to the capacity of the gpu, I am running on batchsize=1.
Operating System
Ubuntu 20.04.6 LTS
DeepLabCut version
2.3.8
DeepLabCut mode
multi animal
Device type
gpu
Steps To Reproduce
- Enter conda environment by conda activate DEEPLABCUTAUD
- Switch to ipython
- import deeplabcut
- Analyze videos
- Error appears after analyzing two videos
Relevant log output
In [28]: deeplabcut.analyze_videos('/home/fishlab4/Fish_audrey-Audrey-2023-10-31/config.yaml',['/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos'],videotype='.MP4',auto_track=True,allow_growth=True)
Using snapshot-500000 for model /home/fishlab4/Fish_audrey-Audrey-2023-10-31/dlc-models/iteration-0/Fish_audreyOct31-trainset95shuffle1
/home/fishlab4/anaconda3/envs/DEEPLABCUTAUD/lib/python3.9/site-packages/tensorflow/python/keras/engine/base_layer_v1.py:1694: UserWarning: `layer.apply` is deprecated and will be removed in a future version. Please use `layer.__call__` method instead.
warnings.warn('`layer.apply` is deprecated and '
Activating extracting of PAFs
2023-11-15 11:33:07.319245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1371 MB memory: -> device: 0, name: Quadro P620, pci bus id: 0000:18:00.0, compute capability: 6.1
Analyzing all the videos in the directory...
Starting to analyze % /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Non-Resident1.MP4
Loading /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Non-Resident1.MP4
Duration of video [s]: 712.71 , recorded with 59.94 fps!
Overall # of frames: 42720 found with (before cropping) frame dimensions: 2704 1520
Starting to extract posture from the video(s) with batchsize: 1
2023-11-15 11:33:09.072472: W tensorflow/core/common_runtime/bfc_allocator.cc:290] Allocator (GPU_0_bfc) ran out of memory trying to allocate 592.19MiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available.
0%| | 0/42720 [4:56:08<?, ?it/s]
34%|████████████████████████████████████████████████████▉ | 14493/42720 [4:50:29<9:24:09, 1.20s/it 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 42720/42720 [14:15:16<00:00, 1.20s/it]
Video Analyzed. Saving results in /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos...██████████████████████████████████████████████████████████████████████████| 42720/42720 [14:15:16<00:00, 1.20s/it]
Using snapshot-500000 for model /home/fishlab4/Fish_audrey-Audrey-2023-10-31/dlc-models/iteration-0/Fish_audreyOct31-trainset95shuffle1
Processing... /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Non-Resident1.MP4
Analyzing /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Non-Resident1DLC_dlcrnetms5_Fish_audreyOct31shuffle1_500000.h5
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 42720/42720 [00:22<00:00, 1871.29it/s]
42720it [00:51, 826.14it/s]
The tracklets were created (i.e., under the hood deeplabcut.convert_detections2tracklets was run). Now you can 'refine_tracklets' in the GUI, or run 'deeplabcut.stitch_tracklets'.
Processing... /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Non-Resident1.MP4
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2595/2595 [00:22<00:00, 115.13it/s]
Starting to analyze % /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Non-Resident1.MP4
Loading /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Non-Resident1.MP4
Duration of video [s]: 711.71 , recorded with 59.94 fps!
Overall # of frames: 42660 found with (before cropping) frame dimensions: 2704 1520
Starting to extract posture from the video(s) with batchsize: 1
63%|█████████████████████████████████████████████████████████████████████████████████████████████████▊ | 26746/42660 [8:55:39<5:17:41, 1.20s/it]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 42660/42660 [14:16:36<00:00, 1.20s/it]
Video Analyzed. Saving results in /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos...
Using snapshot-500000 for model /home/fishlab4/Fish_audrey-Audrey-2023-10-31/dlc-models/iteration-0/Fish_audreyOct31-trainset95shuffle1
Processing... /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Non-Resident1.MP4
Analyzing /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Non-Resident1DLC_dlcrnetms5_Fish_audreyOct31shuffle1_500000.h5
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 42660/42660 [00:20<00:00, 2042.56it/s]
42660it [00:41, 1019.74it/s]
The tracklets were created (i.e., under the hood deeplabcut.convert_detections2tracklets was run). Now you can 'refine_tracklets' in the GUI, or run 'deeplabcut.stitch_tracklets'.
Processing... /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Non-Resident1.MP4
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1490/1490 [00:14<00:00, 100.95it/s]
Starting to analyze % /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Non-Resident1.MP4
Loading /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Non-Resident1.MP4
Duration of video [s]: 712.71 , recorded with 59.94 fps!
Overall # of frames: 42720 found with (before cropping) frame dimensions: 2704 1520
Starting to extract posture from the video(s) with batchsize: 1
4%|██████▍ | 1728/42720 [34:31<13:42:12, 1.20s/it]---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
Cell In[28], line 1
----> 1 deeplabcut.analyze_videos('/home/fishlab4/Fish_audrey-Audrey-2023-10-31/config.yaml',['/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos'],videotype='.MP4',auto_track=True,allow_growth=True)
File ~/anaconda3/envs/DEEPLABCUTAUD/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/predict_videos.py:628, in analyze_videos(config, videos, videotype, shuffle, trainingsetindex, gputouse, save_as_csv, in_random_order, destfolder, batchsize, cropping, TFGPUinference, dynamic, modelprefix, robust_nframes, allow_growth, use_shelve, auto_track, n_tracks, calibrate, identity_only, use_openvino)
623 from deeplabcut.pose_estimation_tensorflow.predict_multianimal import (
624 AnalyzeMultiAnimalVideo,
625 )
627 for video in Videos:
--> 628 AnalyzeMultiAnimalVideo(
629 video,
630 DLCscorer,
631 trainFraction,
632 cfg,
633 dlc_cfg,
634 sess,
635 inputs,
636 outputs,
637 destfolder,
638 robust_nframes=robust_nframes,
639 use_shelve=use_shelve,
640 )
641 if auto_track: # tracker type is taken from default in cfg
642 convert_detections2tracklets(
643 config,
644 [video],
(...)
651 identity_only=identity_only,
652 )
File ~/anaconda3/envs/DEEPLABCUTAUD/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/predict_multianimal.py:184, in AnalyzeMultiAnimalVideo(video, DLCscorer, trainFraction, cfg, dlc_cfg, sess, inputs, outputs, destfolder, robust_nframes, use_shelve)
172 PredicteData, nframes = GetPoseandCostsF(
173 cfg,
174 dlc_cfg,
(...)
181 shelf_path,
182 )
183 else:
--> 184 PredicteData, nframes = GetPoseandCostsS(
185 cfg,
186 dlc_cfg,
187 sess,
188 inputs,
189 outputs,
190 vid,
191 nframes,
192 shelf_path,
193 )
195 stop = time.time()
197 if cfg["cropping"] == True:
File ~/anaconda3/envs/DEEPLABCUTAUD/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/predict_multianimal.py:511, in GetPoseandCostsS(cfg, dlc_cfg, sess, inputs, outputs, cap, nframes, shelf_path)
503 frame = rgba2rgb(frame)
504 dets = predict.predict_batched_peaks_and_costs(
505 dlc_cfg,
506 np.expand_dims(frame, axis=0),
(...)
509 outputs,
510 )
--> 511 db[key] = dets[0]
512 del dets
513 elif counter >= nframes:
IndexError: list index out of range
In[30] deeplabcut.analyze_videos('/home/fishlab4/Fish_audrey-Audrey-2023-10-31/config.yaml',['/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Non-Resident1.MP4','/home/fishlab4/Fish_audr
...: ey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Predator1.MP4','/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Resident1.MP4','/home/fishlab4/Fish_audrey-Audrey-2023-10-
...: 31/videos/LD1_(Tank51)_Non-Resident2.MP4'],videotype='.MP4',auto_track=True)
Using snapshot-500000 for model /home/fishlab4/Fish_audrey-Audrey-2023-10-31/dlc-models/iteration-0/Fish_audreyOct31-trainset95shuffle1
/home/fishlab4/anaconda3/envs/DEEPLABCUTAUD/lib/python3.9/site-packages/tensorflow/python/keras/engine/base_layer_v1.py:1694: UserWarning: `layer.apply` is deprecated and will be removed in a future version. Please use `layer.__call__` method instead.
warnings.warn('`layer.apply` is deprecated and '
Activating extracting of PAFs
2023-11-17 12:16:38.828896: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1371 MB memory: -> device: 0, name: Quadro P620, pci bus id: 0000:18:00.0, compute capability: 6.1
Starting to analyze % /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Non-Resident1.MP4
Loading /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Non-Resident1.MP4
Duration of video [s]: 712.71 , recorded with 59.94 fps!
Overall # of frames: 42720 found with (before cropping) frame dimensions: 2704 1520
Starting to extract posture from the video(s) with batchsize: 1
--------------------------------------------------------------------------- | 1728/42720 [34:33<13:36:07, 1.19s/it]
IndexError Traceback (most recent call last)
Cell In[30], line 1
----> 1 deeplabcut.analyze_videos('/home/fishlab4/Fish_audrey-Audrey-2023-10-31/config.yaml',['/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Non-Resident1.MP4','/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Predator1.MP4','/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Resident1.MP4','/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Non-Resident2.MP4'],videotype='.MP4',auto_track=True)
File ~/anaconda3/envs/DEEPLABCUTAUD/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/predict_videos.py:628, in analyze_videos(config, videos, videotype, shuffle, trainingsetindex, gputouse, save_as_csv, in_random_order, destfolder, batchsize, cropping, TFGPUinference, dynamic, modelprefix, robust_nframes, allow_growth, use_shelve, auto_track, n_tracks, calibrate, identity_only, use_openvino)
623 from deeplabcut.pose_estimation_tensorflow.predict_multianimal import (
624 AnalyzeMultiAnimalVideo,
625 )
627 for video in Videos:
--> 628 AnalyzeMultiAnimalVideo(
629 video,
630 DLCscorer,
631 trainFraction,
632 cfg,
633 dlc_cfg,
634 sess,
635 inputs,
636 outputs,
637 destfolder,
638 robust_nframes=robust_nframes,
639 use_shelve=use_shelve,
640 )
641 if auto_track: # tracker type is taken from default in cfg
642 convert_detections2tracklets(
643 config,
644 [video],
(...)
651 identity_only=identity_only,
652 )
File ~/anaconda3/envs/DEEPLABCUTAUD/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/predict_multianimal.py:184, in AnalyzeMultiAnimalVideo(video, DLCscorer, trainFraction, cfg, dlc_cfg, sess, inputs, outputs, destfolder, robust_nframes, use_shelve)
172 PredicteData, nframes = GetPoseandCostsF(
173 cfg,
174 dlc_cfg,
(...)
181 shelf_path,
182 )
183 else:
--> 184 PredicteData, nframes = GetPoseandCostsS(
185 cfg,
186 dlc_cfg,
187 sess,
188 inputs,
189 outputs,
190 vid,
191 nframes,
192 shelf_path,
193 )
195 stop = time.time()
197 if cfg["cropping"] == True:
File ~/anaconda3/envs/DEEPLABCUTAUD/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/predict_multianimal.py:511, in GetPoseandCostsS(cfg, dlc_cfg, sess, inputs, outputs, cap, nframes, shelf_path)
503 frame = rgba2rgb(frame)
504 dets = predict.predict_batched_peaks_and_costs(
505 dlc_cfg,
506 np.expand_dims(frame, axis=0),
(...)
509 outputs,
510 )
--> 511 db[key] = dets[0]
512 del dets
513 elif counter >= nframes:
IndexError: list index out of rangeAnything else?
Below is my config file
# Project definitions (do not edit)
Task: Fish_audrey
scorer: Audrey
date: Oct31
multianimalproject: true
identity: true
# Project path (change when moving around)
project_path: /home/fishlab4/Fish_audrey-Audrey-2023-10-31
# Annotation data set configuration (and individual video cropping parameters)
video_sets:
/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Non-Resident1.MP4:
crop: 0, 2704, 0, 1520
/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Predator1.MP4:
crop: 0, 2704, 0, 1520
/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Resident1.MP4:
crop: 0, 2704, 0, 1520
/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Non-Resident1.MP4:
crop: 0, 2704, 0, 1520
/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Non-Resident2.MP4:
crop: 0, 2704, 0, 1520
/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Predator1.MP4:
crop: 0, 2704, 0, 1520
/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Resident1.MP4:
crop: 0, 2704, 0, 1520
/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Non-Resident1.MP4:
crop: 0, 2704, 0, 1520
/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Predator1.MP4:
crop: 0, 2704, 0, 1520
/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Resident1.MP4:
crop: 0, 2704, 0, 1520
individuals:
- Cleaner
- ClientPredator
- ClientResident
- ClientNonResident
uniquebodyparts: []
multianimalbodyparts:
- HeadTerminalMouth
- HeadBasisUp
- HeadBasisDown
- TailBottomCorner
- TailUpperCorner
- TailJunctionBody
- BodyMidUp
- BodyMidDown
bodyparts: MULTI!
# Fraction of video to start/stop when extracting frames for labeling/refinement
start: 0
stop: 1
numframes2pick: 20
# Plotting configuration
skeleton:
- bodypart1
- bodypart2
- bodypart2
- bodypart3
- bodypart1
- bodypart3
skeleton_color: black
pcutoff: 0.6
dotsize: 12
alphavalue: 0.7
colormap: rainbow
# Training,Evaluation and Analysis configuration
TrainingFraction:
- 0.95
iteration: 0
default_net_type: dlcrnet_ms5
default_augmenter: multi-animal-imgaug
default_track_method: ellipse
snapshotindex: -1
batch_size: 1
# Cropping Parameters (for analysis and outlier frame detection)
cropping: false
#if cropping is true for analysis, then set the values here:
x1: 0
x2: 640
y1: 277
y2: 624
# Refinement configuration (parameters from annotation dataset configuration also relevant in this stage)
corner2move2:
- 50
- 50
move2corner: trueCode of Conduct
- I agree to follow this project's Code of Conduct