Skip to content

Changing batch size causes an error: KeypointAwareCrop #2769

@vonaviv

Description

@vonaviv

Is there an existing issue for this?

  • I have searched the existing issues

Bug description

I use DLC with PyTorch engine and when I set the batch size for training in pytorch_config.yaml to anything except of 1 I get the following error message:

Loading DLC 3.0.0rc5...
DLC loaded in light mode; you cannot use any GUI (labeling, relabeling and standalone GUI)
Traceback (most recent call last):
  File "/user/vladyslav.ivanov01/u11170/dlc/ExR_18cam-smg-2024-10-25/scripts/train.py", line 13, in <module>
    deeplabcut.train_network(config_path, max_snapshots_to_keep=20, shuffle=shuffle)
  File "/usr/local/lib/python3.10/dist-packages/deeplabcut/compat.py", line 245, in train_network
    return train_network(
  File "/usr/local/lib/python3.10/dist-packages/deeplabcut/pose_estimation_pytorch/apis/train.py", line 338, in train_network
    train(
  File "/usr/local/lib/python3.10/dist-packages/deeplabcut/pose_estimation_pytorch/apis/train.py", line 186, in train
    runner.fit(
  File "/usr/local/lib/python3.10/dist-packages/deeplabcut/pose_estimation_pytorch/runners/train.py", line 166, in fit
    train_loss = self._epoch(
  File "/usr/local/lib/python3.10/dist-packages/deeplabcut/pose_estimation_pytorch/runners/train.py", line 216, in _epoch
    for i, batch in enumerate(loader):
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 701, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 757, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 52, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.10/dist-packages/deeplabcut/pose_estimation_pytorch/data/dataset.py", line 156, in __getitem__
    transformed = self.apply_transform_all_keypoints(
  File "/usr/local/lib/python3.10/dist-packages/deeplabcut/pose_estimation_pytorch/data/dataset.py", line 334, in apply_transform_all_keypoints
    transformed = apply_transform(
  File "/usr/local/lib/python3.10/dist-packages/deeplabcut/pose_estimation_pytorch/data/utils.py", line 492, in apply_transform
    transformed = _apply_transform(
  File "/usr/local/lib/python3.10/dist-packages/deeplabcut/pose_estimation_pytorch/data/utils.py", line 546, in _apply_transform
    transformed = transform(
  File "/usr/local/lib/python3.10/dist-packages/albumentations/core/composition.py", line 222, in __call__
    data = t(**data)
  File "/usr/local/lib/python3.10/dist-packages/albumentations/core/transforms_interface.py", line 114, in __call__
    params_dependent_on_targets = self.get_params_dependent_on_targets(targets_as_params)
  File "/usr/local/lib/python3.10/dist-packages/deeplabcut/pose_estimation_pytorch/data/transforms.py", line 289, in get_params_dependent_on_targets
    kpts = kpts[~np.isnan(kpts).all(axis=1)]
  File "/usr/local/lib/python3.10/dist-packages/numpy/core/_methods.py", line 64, in _all
    return umr_all(a, axis, dtype, out, keepdims)
numpy.exceptions.AxisError: axis 1 is out of bounds for array of dimension 1

Operating System

Rocky Linux 8.8

DeepLabCut version

3.0.0rc5

DeepLabCut mode

single animal

Device type

gpu

Steps To Reproduce

No response

Relevant log output

No response

Anything else?

No response

Code of Conduct

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions