CI upgradeapalooza `bionic`->`focal`, `gcc7`->`gcc9`, `clang7`->`clang10` #105260

malfet · 2023-07-15T01:28:31Z

Bionic support was finished back in April 2023, see https://ubuntu.com/blog/ubuntu-18-04-eol-for-devices

And neither gcc-7 nor clang7 are fully compatible with c++17, update minimal tested gcc to gcc9 and clang to clang-10

Note: OpenMP support is broken in Focal's clang9, so move up to a clang10

Suppress -Wuninitialized in complex_test as gcc-11 fires a seemingly false-positive warning:

In file included from /home/malfet/git/pytorch/pytorch/c10/test/util/complex_test.cpp:1:
/home/malfet/git/pytorch/pytorch/c10/test/util/complex_test_common.h: In member function ‘virtual void memory::TestMemory_ReinterpretCast_Test::TestBody()’:
/home/malfet/git/pytorch/pytorch/c10/test/util/complex_test_common.h:38:25: warning: ‘z’ is used uninitialized [-Wuninitialized]
   38 |     c10::complex<float> zz = *reinterpret_cast<c10::complex<float>*>(&z);
      |                         ^~
/home/malfet/git/pytorch/pytorch/c10/test/util/complex_test_common.h:37:25: note: ‘z’ declared here
   37 |     std::complex<float> z(1, 2);
      |                         ^

Downgrade ucc to 2.15, as 2.16 brings an incompatible libnccl, that causes crash during the initialization
Install pango from condo environment for doctr torch bench tests to pass, as one available in the system is too new for conda
Suppress some functorch tests when used with python-3.8+dynamo, see Dynamo test_vmap failures on Python-3.8 #107173

pytorch-bot · 2023-07-15T01:28:34Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/105260

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 1 Unrelated Failure

As of commit 78addb1:

NEW FAILURES - The following jobs have failed:

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

cuda12.1-py3.10-gcc9-sm86 / test (inductor, 1, 1, linux.g5.4xlarge.nvidia.gpu, unstable) (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

malfet · 2023-07-29T01:14:36Z

clang9+OpenMP is broken on Ubuntu-20.04, but clang10 works fine, so update to clang10

To `slow.yml` and `mac-mps.yaml`, based on the results of the following grep: ``` % grep "sync-tag: " .github/workflows/*.yml .github/workflows/mac-mps.yml: sync-tag: macos-12-py3-arm64-build .github/workflows/mac-mps.yml: sync-tag: macos-12-py3-arm64-mps-test .github/workflows/pull.yml: sync-tag: asan-build .github/workflows/pull.yml: sync-tag: asan-test .github/workflows/pull.yml: sync-tag: win-cpu-build .github/workflows/pull.yml: sync-tag: rocm-build .github/workflows/slow.yml: sync-tag: asan-build .github/workflows/slow.yml: sync-tag: asan-test .github/workflows/trunk.yml: sync-tag: macos-12-py3-arm64-build .github/workflows/trunk.yml: sync-tag: macos-12-py3-arm64-mps-test .github/workflows/trunk.yml: sync-tag: win-cpu-build .github/workflows/trunk.yml: sync-tag: win-cuda-build .github/workflows/trunk.yml: sync-tag: rocm-build ``` Allow synced workflows to diverge with regards to `test-matrix`, to allow for both `mac-mps` and slow part of ASAN tests. Discovered while working on #105260 that slow sync-tag is not checked. Pull Request resolved: #106331 Approved by: https://github.com/huydhn, https://github.com/atalman, https://github.com/seemethere

Modify `install_conda` to remove libstdc++ from libstdcxx-ng to use one from OS Modify `install_torchvision` to workaround weird glibc bug, where malloc interposers (such as ASAN) are causing a hang in internationalization library, see https://sourceware.org/bugzilla/show_bug.cgi?id=27653 and https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90589 Extracted from #105260 [ghstack-poisoned]

…my)" Modify `install_conda` to remove libstdc++ from libstdcxx-ng to use one from OS Modify `install_torchvision` to workaround weird glibc bug, where malloc interposers (such as ASAN) are causing a hang in internationalization library, see https://sourceware.org/bugzilla/show_bug.cgi?id=27653 and https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90589 Extracted from #105260 [ghstack-poisoned]

Modify `install_conda` to remove libstdc++ from libstdcxx-ng to use one from OS Modify `install_torchvision` to workaround weird glibc bug, where malloc interposers (such as ASAN) are causing a hang in internationalization library, see https://sourceware.org/bugzilla/show_bug.cgi?id=27653 and https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90589 Extracted from #105260 [ghstack-poisoned]

…my)" Modify `install_conda` to remove libstdc++ from libstdcxx-ng to use one from OS Modify `install_torchvision` to workaround weird glibc bug, where malloc interposers (such as ASAN) are causing a hang in internationalization library, see https://sourceware.org/bugzilla/show_bug.cgi?id=27653 and https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90589 Extracted from #105260 [ghstack-poisoned]

Modify `install_conda` to remove libstdc++ from libstdcxx-ng to use one from OS Modify `install_torchvision` to workaround weird glibc bug, where malloc interposers (such as ASAN) are causing a hang in internationalization library, see https://sourceware.org/bugzilla/show_bug.cgi?id=27653 and https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90589 Extracted from #105260 [ghstack-poisoned]

…my)" Modify `install_conda` to remove libstdc++ from libstdcxx-ng to use one from OS Modify `install_torchvision` to workaround weird glibc bug, where malloc interposers (such as ASAN) are causing a hang in internationalization library, see https://sourceware.org/bugzilla/show_bug.cgi?id=27653 and https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90589 Extracted from #105260 [ghstack-poisoned]

Modify `install_conda` to remove libstdc++ from libstdcxx-ng to use one from OS Modify `install_torchvision` to workaround weird glibc bug, where malloc interposers (such as ASAN) are causing a hang in internationalization library, see https://sourceware.org/bugzilla/show_bug.cgi?id=27653 and https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90589 Extracted from #105260 [ghstack-poisoned]

malfet · 2023-08-15T03:05:10Z

@pytorchbot merge -i

pytorchmergebot · 2023-08-15T03:06:53Z

Merge started

Your change will be merged while ignoring the following 3 checks: periodic / macos-12-py3-x86-64 / test (default, 2, 4, macos-12), inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor, 1, 1, linux.g5.4xlarge.nvidia.gpu, unstable), trunk / linux-focal-rocm5.6-py3.8 / test (default, 3, 3, linux.rocm.gpu)

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Followup after pytorch/pytorch#105260 as Bionic reached end of life

Regression introduced during migration of `bionic` to `focal` by #105260

Regression introduced during migration of `bionic` to `focal` by #105260 Pull Request resolved: #110177 Approved by: https://github.com/clee2000

malfet requested review from a team and jeffdaily as code owners July 15, 2023 01:28

pytorch-bot bot added the topic: not user facing topic category label Jul 15, 2023

huydhn approved these changes Jul 15, 2023

View reviewed changes

malfet changed the title ~~[CI] Upgrade all bionic to focal builds~~ CI deprecatopalooza bionic->focal, gcc7->gcc9, clang7->clang9 Jul 15, 2023

Skylion007 approved these changes Jul 16, 2023

View reviewed changes

ZainRizvi approved these changes Jul 25, 2023

View reviewed changes

malfet force-pushed the malfet/deprecate-bionic branch from c65683f to 05817a0 Compare July 26, 2023 14:00

seemethere approved these changes Jul 26, 2023

View reviewed changes

malfet changed the title ~~CI deprecatopalooza bionic->focal, gcc7->gcc9, clang7->clang9~~ CI deprecatopalooza bionic->focal, gcc7->gcc9, clang7->clang10 Jul 29, 2023

malfet force-pushed the malfet/deprecate-bionic branch 3 times, most recently from f98c1de to 7dd7e08 Compare July 31, 2023 16:31

malfet mentioned this pull request Jul 31, 2023

Extend workflow sync to more workflow #106331

Closed

malfet force-pushed the malfet/deprecate-bionic branch from 7dcf601 to ecbe376 Compare August 1, 2023 00:09

malfet mentioned this pull request Aug 1, 2023

Move ASAN to clang12 and Ubuntu-22.04 (Jammy) #106355

Closed

malfet added 11 commits August 14, 2023 10:33

Move Caffe2 to jammy

0e8e9cc

Pin nccl version

01d0207

s/bionic/focal/ for inductor workflows

d9c7120

And those

c6ed903

:(

88348dd

And that's it?

06feebc

Allow downgrades

56e66c2

And install pango

2958ea1

And held packages

2044669

And vulkan on clang10

e4066b5

clang9->clang10 here

526267f

malfet mentioned this pull request Aug 14, 2023

Dynamo test_vmap failures on Python-3.8 #107173

Closed

And skip tests

78addb1

malfet force-pushed the malfet/deprecate-bionic branch from 1ced1a1 to 78addb1 Compare August 14, 2023 20:51

malfet requested review from Chillee, kshitij12345 and zou3519 as code owners August 14, 2023 20:51

pytorchmergebot added the merging label Aug 15, 2023

pytorchmergebot added Merged and removed merging labels Aug 15, 2023

pytorchmergebot closed this in 574442b Aug 15, 2023

malfet added a commit to pytorch/tutorials that referenced this pull request Aug 15, 2023

Update docker image base

f7271f3

Followup after pytorch/pytorch#105260 as Bionic reached end of life

malfet mentioned this pull request Aug 15, 2023

Update docker image base pytorch/tutorials#2528

Closed

malfet added a commit that referenced this pull request Sep 27, 2023

Fix XLA update rules

1096b1f

Regression introduced during migration of `bionic` to `focal` by #105260

malfet mentioned this pull request Sep 27, 2023

Fix XLA update rules #110177

Closed

malfet deleted the malfet/deprecate-bionic branch October 18, 2023 20:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CI upgradeapalooza `bionic`->`focal`, `gcc7`->`gcc9`, `clang7`->`clang10` #105260

CI upgradeapalooza `bionic`->`focal`, `gcc7`->`gcc9`, `clang7`->`clang10` #105260

Uh oh!

malfet commented Jul 15, 2023 •

edited

Loading

Uh oh!

pytorch-bot bot commented Jul 15, 2023 •

edited

Loading

Uh oh!

malfet commented Jul 29, 2023

Uh oh!

malfet commented Aug 15, 2023

Uh oh!

pytorchmergebot commented Aug 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

CI upgradeapalooza bionic->focal, gcc7->gcc9, clang7->clang10 #105260

CI upgradeapalooza bionic->focal, gcc7->gcc9, clang7->clang10 #105260

Uh oh!

Conversation

malfet commented Jul 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jul 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/105260

❌ 2 New Failures, 1 Unrelated Failure

Uh oh!

malfet commented Jul 29, 2023

Uh oh!

malfet commented Aug 15, 2023

Uh oh!

pytorchmergebot commented Aug 15, 2023

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

CI upgradeapalooza `bionic`->`focal`, `gcc7`->`gcc9`, `clang7`->`clang10` #105260

CI upgradeapalooza `bionic`->`focal`, `gcc7`->`gcc9`, `clang7`->`clang10` #105260

malfet commented Jul 15, 2023 •

edited

Loading

pytorch-bot bot commented Jul 15, 2023 •

edited

Loading