-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Closed
Labels
Description
Is there an existing issue for this?
- I have searched the existing issues
Bug description
I use DLC with PyTorch engine and when I set the batch size for training in pytorch_config.yaml to anything except of 1 I get the following error message:
Loading DLC 3.0.0rc5...
DLC loaded in light mode; you cannot use any GUI (labeling, relabeling and standalone GUI)
Traceback (most recent call last):
File "/user/vladyslav.ivanov01/u11170/dlc/ExR_18cam-smg-2024-10-25/scripts/train.py", line 13, in <module>
deeplabcut.train_network(config_path, max_snapshots_to_keep=20, shuffle=shuffle)
File "/usr/local/lib/python3.10/dist-packages/deeplabcut/compat.py", line 245, in train_network
return train_network(
File "/usr/local/lib/python3.10/dist-packages/deeplabcut/pose_estimation_pytorch/apis/train.py", line 338, in train_network
train(
File "/usr/local/lib/python3.10/dist-packages/deeplabcut/pose_estimation_pytorch/apis/train.py", line 186, in train
runner.fit(
File "/usr/local/lib/python3.10/dist-packages/deeplabcut/pose_estimation_pytorch/runners/train.py", line 166, in fit
train_loss = self._epoch(
File "/usr/local/lib/python3.10/dist-packages/deeplabcut/pose_estimation_pytorch/runners/train.py", line 216, in _epoch
for i, batch in enumerate(loader):
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 701, in __next__
data = self._next_data()
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 757, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 52, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/usr/local/lib/python3.10/dist-packages/deeplabcut/pose_estimation_pytorch/data/dataset.py", line 156, in __getitem__
transformed = self.apply_transform_all_keypoints(
File "/usr/local/lib/python3.10/dist-packages/deeplabcut/pose_estimation_pytorch/data/dataset.py", line 334, in apply_transform_all_keypoints
transformed = apply_transform(
File "/usr/local/lib/python3.10/dist-packages/deeplabcut/pose_estimation_pytorch/data/utils.py", line 492, in apply_transform
transformed = _apply_transform(
File "/usr/local/lib/python3.10/dist-packages/deeplabcut/pose_estimation_pytorch/data/utils.py", line 546, in _apply_transform
transformed = transform(
File "/usr/local/lib/python3.10/dist-packages/albumentations/core/composition.py", line 222, in __call__
data = t(**data)
File "/usr/local/lib/python3.10/dist-packages/albumentations/core/transforms_interface.py", line 114, in __call__
params_dependent_on_targets = self.get_params_dependent_on_targets(targets_as_params)
File "/usr/local/lib/python3.10/dist-packages/deeplabcut/pose_estimation_pytorch/data/transforms.py", line 289, in get_params_dependent_on_targets
kpts = kpts[~np.isnan(kpts).all(axis=1)]
File "/usr/local/lib/python3.10/dist-packages/numpy/core/_methods.py", line 64, in _all
return umr_all(a, axis, dtype, out, keepdims)
numpy.exceptions.AxisError: axis 1 is out of bounds for array of dimension 1
Operating System
Rocky Linux 8.8
DeepLabCut version
3.0.0rc5
DeepLabCut mode
single animal
Device type
gpu
Steps To Reproduce
No response
Relevant log output
No response
Anything else?
No response
Code of Conduct
- I agree to follow this project's Code of Conduct