fix auxiliary loss training in DetrSegmentation by SangbumChoi · Pull Request #28354 · huggingface/transformers

SangbumChoi · 2024-01-05T07:54:47Z

What does this PR do?

Fixes # (issue)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[v] Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Hi, @amyeroberts. I fixed the training error when setting auxiliary loss to set 'True'
outputs_class = self.detr.class_embed(hs)
Since original DETR does not have their own class of self.class_labels_classifier, we have to import from self.detr

see https://github.com/facebookresearch/detr/blob/3af9fa878e73b6894ce3596450a8d9b89d918ca9/models/segmentation.py#L49

amyeroberts

Thanks for fixing!

Could you add a test which makes sure this code path is now reached?

SangbumChoi · 2024-01-07T10:44:42Z

@amyeroberts Hi back again with the test!
Mainly I added function called test_forward_auxiliary_loss. This will check if the two major components when the config.auxiliary_loss is set to True

whether output of auxiliary loss is not None
check the output len is equivalent to the num_hidden_layers - 1 (-1 is due to last layers is not 'aux')

BTW, this test will also check in not only segmentation but also object_detection_model!

following result is the pytest of detr

root@0a2b4fe54761:/mnt/nas2/users/sbchoi/transformers# RUN_SLOW=1 pytest tests/models/detr
================================================================================ test session starts =================================================================================
platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.0.0
rootdir: /mnt/nas2/users/sbchoi/transformers
configfile: pyproject.toml
plugins: hypothesis-6.92.0, hydra-core-1.3.2
collected 154 items

tests/models/detr/test_image_processing_detr.py ................                                                                                                               [ 10%]
tests/models/detr/test_modeling_detr.py .......................ssssss..sssssssss......s...............s......s........s....ssssssss.ss.sssssssssssssss.s..s.......s........... [ 97%]
....                                                                                                                                                                           [100%]

================================================================================== warnings summary ==================================================================================
../../../../../opt/conda/lib/python3.10/site-packages/_pytest/config/__init__.py:1373
  /opt/conda/lib/python3.10/site-packages/_pytest/config/__init__.py:1373: PytestConfigWarning: Unknown config option: doctest_glob

    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

src/transformers/deepspeed.py:23
  /mnt/nas2/users/sbchoi/transformers/src/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
    warnings.warn(

../../../../../opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28
  /opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import packaging  # type: ignore[attr-defined]

../../../../../opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py:2871
../../../../../opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py:2871
  /opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

tests/models/detr/test_modeling_detr.py::DetrModelTest::test_disk_offload_bin
  /opt/conda/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
    return self.fget.__get__(instance, owner)()

tests/models/detr/test_modeling_detr.py::DetrModelTest::test_fast_init_context_manager
  /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:460: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(init_instance.linear.bias, expected_bias, rtol=1e-3, atol=1e-4)

tests/models/detr/test_modeling_detr.py::DetrModelTest::test_fast_init_context_manager
  /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:463: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(

tests/models/detr/test_modeling_detr.py::DetrModelTest::test_pipeline_image_segmentation
tests/models/detr/test_modeling_detr.py::DetrModelTest::test_pipeline_object_detection
  /opt/conda/lib/python3.10/site-packages/huggingface_hub/repocard.py:105: UserWarning: Repo card metadata block was not found. Setting CardData to empty.
    warnings.warn("Repo card metadata block was not found. Setting CardData to empty.")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================== 107 passed, 47 skipped, 10 warnings in 110.95s (0:01:50) ==============================================================
root@0a2b4fe54761:/mnt/nas2/users/sbchoi/transformers#

amyeroberts

@SangbumChoi Thanks for adding and running the tests - looks great!

Could you add these tests to the other DETR-related modes too (if not already covered)?:

conditional DETR
deformable DETR
DETA
Table Transformer
Yolos
Maskformer

SangbumChoi · 2024-01-08T12:48:12Z

@amyeroberts No problem! However, should I make another PR or just modify in this PR?

amyeroberts · 2024-01-08T14:44:36Z

@SangbumChoi Up to you. I'd suggest making a follow-up PR to prevent holding this one up, but happy to go for whatever you'd prefer

SangbumChoi · 2024-01-09T00:01:28Z

@amyeroberts Yeah, I will make follow-up PR and link this PR also. If there's nothing else to add for this topic, let's MERGE this first :)

amyeroberts · 2024-01-09T10:17:37Z

@SangbumChoi Merged! Thanks again for this contribution 💪

fix auxiliary loss training in detrSegmentation

cf32660

amyeroberts approved these changes Jan 5, 2024

View reviewed changes

add auxiliary_loss testing

d807b10

amyeroberts approved these changes Jan 8, 2024

View reviewed changes

SangbumChoi mentioned this pull request Jan 9, 2024

Fix auxiliary loss related code in transformers #28406

Merged

5 tasks

amyeroberts merged commit 357971e into huggingface:main Jan 9, 2024

SangbumChoi deleted the fix_detr branch March 5, 2024 05:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix auxiliary loss training in DetrSegmentation#28354

fix auxiliary loss training in DetrSegmentation#28354
amyeroberts merged 2 commits into
huggingface:mainfrom
SangbumChoi:fix_detr

SangbumChoi commented Jan 5, 2024

Uh oh!

amyeroberts left a comment

Uh oh!

SangbumChoi commented Jan 7, 2024 •

edited

Loading

Uh oh!

amyeroberts left a comment

Uh oh!

SangbumChoi commented Jan 8, 2024

Uh oh!

amyeroberts commented Jan 8, 2024

Uh oh!

SangbumChoi commented Jan 9, 2024

Uh oh!

amyeroberts commented Jan 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

SangbumChoi commented Jan 5, 2024

What does this PR do?

Before submitting

Who can review?

Uh oh!

amyeroberts left a comment

Choose a reason for hiding this comment

Uh oh!

SangbumChoi commented Jan 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amyeroberts left a comment

Choose a reason for hiding this comment

Uh oh!

SangbumChoi commented Jan 8, 2024

Uh oh!

amyeroberts commented Jan 8, 2024

Uh oh!

SangbumChoi commented Jan 9, 2024

Uh oh!

amyeroberts commented Jan 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SangbumChoi commented Jan 7, 2024 •

edited

Loading