[huggingface_pytorch] Training - update for Transformers to 4.28.1 PyTorch 2.0 by JingyaHuang · Pull Request #2993 · aws/deep-learning-containers

JingyaHuang · 2023-05-12T15:02:35Z

GitHub Issue #2986 :

Description

This PR updates Hugginface's PyTorch DLC for inference. Here are the corresponding updated dependencies versions:

transformers: 4.28.1
datasets: 2.12.0
evaluate: 0.4.0
accelerate: 0.19.0
torch: 2.0.0
diffusers: 0.16.1

And extra tests:

PyTorch 2.0 compile test
Diffusers test

Note:

If merging this PR should also close the associated Issue, please also add that Issue # to the Linked Issues section on the right.
All PR's are checked weekly for staleness. This PR will be closed if not updated in 30 days.

Description

Tests run

NOTE: By default, docker builds are disabled. In order to build your container, please update dlc_developer_config.toml and specify the framework to build in "build_frameworks"

I have run builds/tests on commit for my changes.

NOTE: If you are creating a PR for a new framework version, please ensure success of the standard, rc, and efa sagemaker remote tests by updating the dlc_developer_config.toml file:

Revision A: sagemaker_remote_tests = "standard"
Revision B: sagemaker_remote_tests = "rc"
Revision C: sagemaker_remote_tests = "efa"

Additionally, please run the sagemaker local tests in at least one revision:

sagemaker_local_tests = true

Formatting

I have run black -l 100 on my code (formatting tool: https://black.readthedocs.io/en/stable/getting_started.html)

DLC image/dockerfile

Additional context

PR Checklist

I've prepended PR tag with frameworks/job this applies to : [mxnet, tensorflow, pytorch] | [ei/neuron/graviton] | [build] | [test] | [benchmark] | [ec2, ecs, eks, sagemaker]
If the PR changes affects SM test, I've modified dlc_developer_config.toml in my PR branch by setting sagemaker_tests = true and efa_tests = true
If this PR changes existing code, the change fully backward compatible with pre-existing code. (Non backward-compatible changes need special approval.)
(If applicable) I've documented below the DLC image/dockerfile this relates to
(If applicable) I've documented below the tests I've run on the DLC image
(If applicable) I've reviewed the licenses of updated and new binaries and their dependencies to make sure all licenses are on the Apache Software Foundation Third Party License Policy Category A or Category B license list. See https://www.apache.org/legal/resolved.html.
(If applicable) I've scanned the updated and new binaries to make sure they do not have vulnerabilities associated with them.

Pytest Marker Checklist

(If applicable) I have added the marker @pytest.mark.model("<model-type>") to the new tests which I have added, to specify the Deep Learning model that is used in the test (use "N/A" if the test doesn't use a model)
(If applicable) I have added the marker @pytest.mark.integration("<feature-being-tested>") to the new tests which I have added, to specify the feature that will be tested
(If applicable) I have added the marker @pytest.mark.multinode(<integer-num-nodes>) to the new tests which I have added, to specify the number of nodes used on a multi-node test
(If applicable) I have added the marker @pytest.mark.processor(<"cpu"/"gpu"/"eia"/"neuron">) to the new tests which I have added, if a test is specifically applicable to only one processor type

EIA/NEURON/GRAVITON Testing Checklist

When creating a PR:

I've modified dlc_developer_config.toml in my PR branch by setting ei_mode = true, neuron_mode = true or graviton_mode = true

Benchmark Testing Checklist

When creating a PR:

I've modified dlc_developer_config.toml in my PR branch by setting benchmark_mode = true

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

saimidu

Should there be another test added for the evaluate feature?

JingyaHuang · 2023-05-12T22:18:44Z

Should there be another test added for the evaluate feature?

Yep, I replaced previous sklean metrics by those in evaluate. Thanks for the heads up.

saimidu · 2023-05-13T02:29:03Z

It looks like the diffuser test never ran because of:

TypeError: _test_diffusers_model() missing 1 required positional argument: 'instance_type'

All other sagemaker integration tests for the image failed due to:

AttributeError: 'TrainingArguments' object has no attribute 'distributed_state'

JingyaHuang · 2023-05-14T10:06:30Z

It looks like the diffuser test never ran because of:
TypeError: _test_diffusers_model() missing 1 required positional argument: 'instance_type'
All other sagemaker integration tests for the image failed due to:
AttributeError: 'TrainingArguments' object has no attribute 'distributed_state'

Hi @saimidu,

The first issue is fixed. For the second issue, sagemaker distributed training features seem broken since the transformers 4.29.* released this week. Let's continue with 4.28.* . I will discuss this break internally with the Transformers team.

Could you help me with the sanity test? Thanks!

saimidu · 2023-05-15T10:01:44Z

@JingyaHuang Thanks for looking into the issues with SM distributed training when using transformers 4.29.*.

One of the failures on the sanity tests is the pip check test that fails because of the installation of tensorflow==2.11.1, which seems to require protobuf<3.20, while the smdebug already installed on the PT 2.0 image requires protobuf>=3.20.0,<=3.20.3. If the version of tensorflow installed is updated to tensorflow==2.12.0, the problem seems to resolve itself. Not sure why the version installed was 2.11.1 instead of 2.12.0, when the latest version is already available.

W.r.t. the second failure, could you add a file named Dockerfile.gpu.os_scan_allowlist.json similar to https://github.com/aws/deep-learning-containers/blob/master/huggingface/tensorflow/training/docker/2.11/py3/cu112/Dockerfile.gpu.os_scan_allowlist.json, which contains the exact dictionary entry for the tf.keras.utils.get_file vulnerability, as printed out in the CI job logs?

JingyaHuang · 2023-05-15T15:53:01Z

Hi @saimidu,

I just pinged the tensorflow to 2.12.0, TF is needed only as a dependency for torch.utils.tensorboard, so we can update it as far as it works with the tensorboard feature, let's see if the CI is green.

For the os_scan_allowlist, it is usually DLC team who takes care of it, maybe @arjkesh could help?

[Update] TensorFlow 2.12.0 is not compatible transformers 4.28.1, maybe we should add TensorFlow lower version the the allow list? As TF is on board just for tensorboard and lower version of TF works for it.

ERROR: Cannot install tensorflow==2.12.0 and transformers[audio,sentencepiece,sklearn,vision]==4.28.1 because these package versions have conflicting dependencies.


The conflict is caused by:
    transformers[audio,sentencepiece,sklearn,vision] 4.28.1 depends on protobuf<=3.20.2; extra == "sentencepiece"
    tensorflow 2.12.0 depends on protobuf!=4.21.0, !=4.21.1, !=4.21.2, !=4.21.3, !=4.21.4, !=4.21.5, <5.0.0dev and >=3.20.3

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

… update-hf-pt2.0-train

saimidu · 2023-05-17T19:59:24Z

@JingyaHuang All tests seem to pass with tensorflow removed, as per CI on commit 123450d.

Removing tensorflow from ecr-enhanced-scan allowlist and enabling other SM integration tests for the image to move forward.

saimidu

Approved, unless any further changes are necessary. @JingyaHuang Shall we merge this PR?

saimidu

PR approved for merge.

JingyaHuang added 3 commits May 12, 2023 14:22

Update HF packages

70f33b9

add git compile test

7ff0872

add script path

83e9844

aws-deep-learning-containers-ci Bot added build Reflects file change in build folder huggingface Reflects file change in huggingface folder sagemaker_tests Size:S Determines the size of the PR test Reflects file change in test folder labels May 12, 2023

JingyaHuang changed the title ~~Update hf pt2.0 train~~ [huggingface_pytorch] Training - update for Transformers to 4.29.1 PyTorch 2.0 May 12, 2023

JingyaHuang added 2 commits May 12, 2023 15:05

change dlc_developer_confi

1406828

add diffusers test

d0be697

saimidu reviewed May 12, 2023

View reviewed changes

JingyaHuang and others added 5 commits May 12, 2023 21:28

fix test

7271e11

Merge branch 'master' into update-hf-pt2.0-train

43a7b2c

use evaluate & update ddifffusers version fetching

2098bb9

update also metric for trcomp

f1ff8c3

update also metric for trcomp

ff0ab19

JingyaHuang marked this pull request as ready for review May 12, 2023 22:27

JingyaHuang requested a review from a team as a code owner May 12, 2023 22:27

JingyaHuang added 2 commits May 12, 2023 22:29

enable sm remote test

c201f9e

style with blacak and ruff

46aedec

JingyaHuang added 3 commits May 13, 2023 17:23

fix diffusers test

58b4266

what if trfrs 4.28

e389ca8

cleaning

0de0d9e

JingyaHuang changed the title ~~[huggingface_pytorch] Training - update for Transformers to 4.29.1 PyTorch 2.0~~ [huggingface_pytorch] Training - update for Transformers to 4.28.1 PyTorch 2.0 May 13, 2023

ping tensorflow

3b887db

revert TF version ping

c8d814f

JingyaHuang mentioned this pull request May 16, 2023

[Sagemaker] sagemaker distributed features in Trainer broken since Transformers 4.29 huggingface/transformers#23390

Closed

4 tasks

saimidu and others added 4 commits May 16, 2023 15:15

Add tensorflow to allowlist for enhanced scan

76e1817

Merge branch 'master' of github.com:aws/deep-learning-containers into…

d351119

… update-hf-pt2.0-train

what if remove tensorflow

123450d

Disable builds, enable rc tests, remove tf from enhanced scan allowlist

874cd72

Run sagemaker efa tests

abe5974

saimidu previously approved these changes May 18, 2023

View reviewed changes

remove dist_gpu_backend

0a588df

JingyaHuang dismissed saimidu’s stale review via 0a588df May 19, 2023 12:21

JingyaHuang and others added 2 commits May 19, 2023 13:44

trigger sm remote test

df45822

Merge branch 'master' into update-hf-pt2.0-train

bc41995

saimidu previously approved these changes May 19, 2023

View reviewed changes

Revert dev config toml changes

714a8a7

saimidu dismissed their stale review via 714a8a7 May 19, 2023 17:26

saimidu approved these changes May 19, 2023

View reviewed changes

Merge branch 'master' into update-hf-pt2.0-train

4302e0a

saimidu merged commit 54368ae into aws:master May 20, 2023

JingyaHuang deleted the update-hf-pt2.0-train branch May 22, 2023 10:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[huggingface_pytorch] Training - update for Transformers to 4.28.1 PyTorch 2.0#2993

[huggingface_pytorch] Training - update for Transformers to 4.28.1 PyTorch 2.0#2993
saimidu merged 27 commits into
aws:masterfrom
JingyaHuang:update-hf-pt2.0-train

JingyaHuang commented May 12, 2023 •

edited

Loading

Uh oh!

saimidu left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JingyaHuang commented May 12, 2023 •

edited

Loading

Uh oh!

saimidu commented May 13, 2023

Uh oh!

JingyaHuang commented May 14, 2023 •

edited

Loading

Uh oh!

saimidu commented May 15, 2023

Uh oh!

JingyaHuang commented May 15, 2023 •

edited

Loading

Uh oh!

saimidu commented May 17, 2023

Uh oh!

saimidu left a comment

Uh oh!

saimidu left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

JingyaHuang commented May 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Description

Tests run

Formatting

DLC image/dockerfile

Additional context

PR Checklist

Pytest Marker Checklist

EIA/NEURON/GRAVITON Testing Checklist

Benchmark Testing Checklist

Uh oh!

saimidu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JingyaHuang commented May 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

saimidu commented May 13, 2023

Uh oh!

JingyaHuang commented May 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

saimidu commented May 15, 2023

Uh oh!

JingyaHuang commented May 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

saimidu commented May 17, 2023

Uh oh!

saimidu left a comment

Choose a reason for hiding this comment

Uh oh!

saimidu left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

JingyaHuang commented May 12, 2023 •

edited

Loading

JingyaHuang commented May 12, 2023 •

edited

Loading

JingyaHuang commented May 14, 2023 •

edited

Loading

JingyaHuang commented May 15, 2023 •

edited

Loading