Pin auditwheel to 6.2.0 #149471

atalman · 2025-03-19T02:10:07Z

Observing aarch64 failure in nightly:
https://github.com/pytorch/pytorch/actions/runs/13917778961/job/38943911228

Similar to: pytorch/vision#8982

2025-03-18T08:44:58.4128744Z Repairing Wheel with AuditWheel
2025-03-18T08:44:58.5440988Z INFO:auditwheel.main_repair:Repairing torch-2.8.0.dev20250318+cpu-cp39-cp39-linux_aarch64.whl
2025-03-18T08:45:20.3393288Z Traceback (most recent call last):
2025-03-18T08:45:20.3393732Z   File "/opt/python/cp39-cp39/bin/auditwheel", line 8, in <module>
2025-03-18T08:45:20.3394115Z     sys.exit(main())
2025-03-18T08:45:20.3394559Z   File "/opt/_internal/cpython-3.9.21/lib/python3.9/site-packages/auditwheel/main.py", line 53, in main
2025-03-18T08:45:20.3395064Z     result: int | None = args.func(args, p)
2025-03-18T08:45:20.3395626Z   File "/opt/_internal/cpython-3.9.21/lib/python3.9/site-packages/auditwheel/main_repair.py", line 203, in execute
2025-03-18T08:45:20.3396163Z     out_wheel = repair_wheel(
2025-03-18T08:45:20.3396657Z   File "/opt/_internal/cpython-3.9.21/lib/python3.9/site-packages/auditwheel/repair.py", line 84, in repair_wheel
2025-03-18T08:45:20.3397184Z     raise ValueError(msg)
2025-03-18T08:45:20.3397620Z ValueError: Cannot repair wheel, because required library "libarm_compute.so" could not be located
2025-03-18T08:45:20.3678843Z Traceback (most recent call last):
2025-03-18T08:45:20.3679267Z   File "/pytorch/.ci/aarch64_linux/aarch64_wheel_ci_build.py", line 236, in <module>
2025-03-18T08:45:20.3680988Z     pytorch_wheel_name = complete_wheel("/pytorch/")
2025-03-18T08:45:20.3681449Z   File "/pytorch/.ci/aarch64_linux/aarch64_wheel_ci_build.py", line 141, in complete_wheel
2025-03-18T08:45:20.3681976Z     check_call(["auditwheel", "repair", f"dist/{wheel_name}"], cwd=folder)
2025-03-18T08:45:20.3682860Z   File "/opt/python/cp39-cp39/lib/python3.9/subprocess.py", line 373, in check_call
2025-03-18T08:45:20.3683308Z     raise CalledProcessError(retcode, cmd)
2025-03-18T08:45:20.3684034Z subprocess.CalledProcessError: Command '['auditwheel', 'repair', 'dist/torch-2.8.0.dev20250318+cpu-cp39-cp39-linux_aarch64.whl']' returned non-zero exit status 1.
2025-03-18T08:45:20.3790063Z ##[error]Process completed with exit code 1.
2025-03-18T08:45:20.3862012Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2025-03-18T08:45:20.3862448Z with:

Please note aarch64 CUDA failures are related to: #149351

pytorch-bot · 2025-03-19T02:10:12Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/149471

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 18 New Failures, 10 Unrelated Failures

As of commit 2d3befd with merge base b06b5c3 ():

NEW FAILURES - The following jobs have failed:

linux-aarch64-binary-manywheel / manywheel-py3_10-cuda-aarch64-12_8-build / build (gh)
bootstrap.cc:543:53: error: cannot convert ‘char*’ to ‘void**’ in argument passing
linux-aarch64-binary-manywheel / manywheel-py3_11-cuda-aarch64-12_8-build / build (gh)
bootstrap.cc:543:53: error: cannot convert ‘char*’ to ‘void**’ in argument passing
linux-aarch64-binary-manywheel / manywheel-py3_12-cuda-aarch64-12_8-build / build (gh)
bootstrap.cc:543:53: error: cannot convert ‘char*’ to ‘void**’ in argument passing
linux-aarch64-binary-manywheel / manywheel-py3_13-cuda-aarch64-12_8-build / build (gh)
bootstrap.cc:543:53: error: cannot convert ‘char*’ to ‘void**’ in argument passing
linux-aarch64-binary-manywheel / manywheel-py3_13t-cuda-aarch64-12_8-build / build (gh)
bootstrap.cc:543:53: error: cannot convert ‘char*’ to ‘void**’ in argument passing
linux-aarch64-binary-manywheel / manywheel-py3_9-cuda-aarch64-12_8-build / build (gh)
bootstrap.cc:543:53: error: cannot convert ‘char*’ to ‘void**’ in argument passing
linux-binary-manywheel / manywheel-py3_10-xpu-build / build (gh)
Process completed with exit code 1.
linux-binary-manywheel / manywheel-py3_11-cuda12_6-full-test / test (gh)
Process completed with exit code 1.
linux-binary-manywheel / manywheel-py3_11-xpu-build / build (gh)
Process completed with exit code 1.
linux-binary-manywheel / manywheel-py3_12-xpu-build / build (gh)
Process completed with exit code 1.
linux-binary-manywheel / manywheel-py3_13-xpu-build / build (gh)
Process completed with exit code 1.
linux-binary-manywheel / manywheel-py3_13t-xpu-build / build (gh)
Process completed with exit code 1.
linux-binary-manywheel / manywheel-py3_9-xpu-build / build (gh)
Process completed with exit code 1.
linux-s390x-binary-manywheel / manywheel-py3_10-cpu-s390x-test / test (gh)
Process completed with exit code 1.
linux-s390x-binary-manywheel / manywheel-py3_11-cpu-s390x-test / test (gh)
Process completed with exit code 1.
linux-s390x-binary-manywheel / manywheel-py3_12-cpu-s390x-test / test (gh)
Process completed with exit code 1.
linux-s390x-binary-manywheel / manywheel-py3_13-cpu-s390x-test / test (gh)
Process completed with exit code 1.
linux-s390x-binary-manywheel / manywheel-py3_9-cpu-s390x-test / test (gh)
Process completed with exit code 1.

FLAKY - The following job failed but was likely due to flakiness present on trunk:

linux-binary-manywheel / manywheel-py3_11-cpu-cxx11-abi-test / test (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128

BROKEN TRUNK - The following jobs failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / linux-focal-cuda12.6-py3.10-gcc11 / test (default, 2, 5, linux.4xlarge.nvidia.gpu) (gh) (trunk failure)
inductor/test_provenance_tracing.py::TestProvenanceTracingArtifact::test_triton_kernel_to_post_grad_tracing_cpu
pull / linux-focal-cuda12.6-py3.10-gcc11-sm89 / test (default, 1, 5, linux.g6.4xlarge.experimental.nvidia.gpu) (gh) (trunk failure)
inductor/test_aot_inductor_arrayref.py::AOTInductorTestABICompatibleCpuWithStackAllocation::test_update_inactive_constant_buffer_cpu_with_stack_allocation
pull / linux-focal-cuda12.6-py3.10-gcc11-sm89 / test (default, 2, 5, linux.g6.4xlarge.experimental.nvidia.gpu) (gh) (trunk failure)
inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_update_inactive_constant_buffer_cpu
pull / linux-focal-cuda12.6-py3.10-gcc11-sm89 / test (default, 5, 5, linux.g6.4xlarge.experimental.nvidia.gpu) (gh) (trunk failure)
inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_free_inactive_buffer_cuda
pull / linux-focal-py3.13-clang10 / test (default, 2, 5, linux.4xlarge) (gh) (trunk failure)
inductor/test_provenance_tracing.py::TestProvenanceTracingArtifact::test_triton_kernel_to_post_grad_tracing_cpu
pull / linux-focal-py3.9-clang10 / test (default, 2, 5, linux.4xlarge) (gh) (trunk failure)
inductor/test_provenance_tracing.py::TestProvenanceTracingArtifact::test_triton_kernel_to_post_grad_tracing_cpu
pull / linux-jammy-py3.10-clang15-asan / test (default, 2, 6, linux.4xlarge) (gh) (trunk failure)
inductor/test_provenance_tracing.py::TestProvenanceTracingArtifact::test_triton_kernel_to_post_grad_tracing_cpu
pull / linux-jammy-py3.9-gcc11 / test (default, 2, 5, linux.2xlarge) (gh) (trunk failure)
inductor/test_provenance_tracing.py::TestProvenanceTracingArtifact::test_triton_kernel_to_post_grad_tracing_cpu

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

pull / linux-jammy-py3-clang12-executorch / test (executorch, 1, 1, linux.2xlarge) (gh) (#144480)
Process completed with exit code 1.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

.ci/aarch64_linux/aarch64_ci_build.sh

malfet

LGTM, but aren't we using docker (and shouldn't it be installed there)?

atalman · 2025-03-19T15:53:09Z

@pytorchmergebot merge -f "lint and required workflows are green"

pytorchmergebot · 2025-03-19T15:54:45Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

atalman · 2025-03-19T16:04:43Z

@pytorchbot cherry-pick --onto release/2.7 --fixes "aarch64 cpu builds" -c critical

Observing aarch64 failure in nightly: https://github.com/pytorch/pytorch/actions/runs/13917778961/job/38943911228 Similar to: pytorch/vision#8982 ``` 2025-03-18T08:44:58.4128744Z Repairing Wheel with AuditWheel 2025-03-18T08:44:58.5440988Z INFO:auditwheel.main_repair:Repairing torch-2.8.0.dev20250318+cpu-cp39-cp39-linux_aarch64.whl 2025-03-18T08:45:20.3393288Z Traceback (most recent call last): 2025-03-18T08:45:20.3393732Z File "/opt/python/cp39-cp39/bin/auditwheel", line 8, in <module> 2025-03-18T08:45:20.3394115Z sys.exit(main()) 2025-03-18T08:45:20.3394559Z File "/opt/_internal/cpython-3.9.21/lib/python3.9/site-packages/auditwheel/main.py", line 53, in main 2025-03-18T08:45:20.3395064Z result: int | None = args.func(args, p) 2025-03-18T08:45:20.3395626Z File "/opt/_internal/cpython-3.9.21/lib/python3.9/site-packages/auditwheel/main_repair.py", line 203, in execute 2025-03-18T08:45:20.3396163Z out_wheel = repair_wheel( 2025-03-18T08:45:20.3396657Z File "/opt/_internal/cpython-3.9.21/lib/python3.9/site-packages/auditwheel/repair.py", line 84, in repair_wheel 2025-03-18T08:45:20.3397184Z raise ValueError(msg) 2025-03-18T08:45:20.3397620Z ValueError: Cannot repair wheel, because required library "libarm_compute.so" could not be located 2025-03-18T08:45:20.3678843Z Traceback (most recent call last): 2025-03-18T08:45:20.3679267Z File "/pytorch/.ci/aarch64_linux/aarch64_wheel_ci_build.py", line 236, in <module> 2025-03-18T08:45:20.3680988Z pytorch_wheel_name = complete_wheel("/pytorch/") 2025-03-18T08:45:20.3681449Z File "/pytorch/.ci/aarch64_linux/aarch64_wheel_ci_build.py", line 141, in complete_wheel 2025-03-18T08:45:20.3681976Z check_call(["auditwheel", "repair", f"dist/{wheel_name}"], cwd=folder) 2025-03-18T08:45:20.3682860Z File "/opt/python/cp39-cp39/lib/python3.9/subprocess.py", line 373, in check_call 2025-03-18T08:45:20.3683308Z raise CalledProcessError(retcode, cmd) 2025-03-18T08:45:20.3684034Z subprocess.CalledProcessError: Command '['auditwheel', 'repair', 'dist/torch-2.8.0.dev20250318+cpu-cp39-cp39-linux_aarch64.whl']' returned non-zero exit status 1. 2025-03-18T08:45:20.3790063Z ##[error]Process completed with exit code 1. 2025-03-18T08:45:20.3862012Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2025-03-18T08:45:20.3862448Z with: ``` Please note aarch64 CUDA failures are related to: #149351 Pull Request resolved: #149471 Approved by: https://github.com/malfet (cherry picked from commit 4df66e0)

pytorchbot · 2025-03-19T16:09:58Z

Cherry picking #149471

The cherry pick PR is at #149525 and it is linked with issue aarch64 cpu builds. The following tracker issues are updated:

[v.2.7.0] Release Tracker #149044 (comment)

Details for Dev Infra team

Raised by workflow job

by applying commits from pytorch/pytorch#149471

Pin auditwheel to < 6.3.0

6554a48

atalman requested a review from a team as a code owner March 19, 2025 02:10

atalman added topic: not user facing topic category ciflow/binaries Trigger all binary build and upload jobs on the PR labels Mar 19, 2025

atalman commented Mar 19, 2025

View reviewed changes

.ci/aarch64_linux/aarch64_ci_build.sh Outdated Show resolved Hide resolved

Update .ci/aarch64_linux/aarch64_ci_build.sh

2d3befd

atalman changed the title ~~Pin auditwheel to < 6.3.0~~ Pin auditwheel to 6.2.0 Mar 19, 2025

malfet approved these changes Mar 19, 2025

View reviewed changes

pytorchmergebot added the merging label Mar 19, 2025

pytorchmergebot closed this in 4df66e0 Mar 19, 2025

pytorchmergebot added Merged and removed merging labels Mar 19, 2025

pytorchbot mentioned this pull request Mar 19, 2025

[v.2.7.0] Release Tracker #149044

Closed

oraluben mentioned this pull request Mar 23, 2025

track pytorch issue https://github.com/pytorch/pytorch/pull/149471 pypa/auditwheel#561

Open

jondea added a commit to ARM-software/Tool-Solutions that referenced this pull request Mar 24, 2025

Fix build error due to auditwheel version change

04aeced

by applying commits from pytorch/pytorch#149471

jondea added a commit to ARM-software/Tool-Solutions that referenced this pull request Mar 24, 2025

Fix build error due to auditwheel version change

906c857

by applying commits from pytorch/pytorch#149471

jondea mentioned this pull request Mar 24, 2025

Fix build error due to auditwheel version change ARM-software/Tool-Solutions#308

Merged

oraluben mentioned this pull request May 11, 2025

Update libraries later if possible pypa/auditwheel#582

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Pin auditwheel to 6.2.0 #149471

Pin auditwheel to 6.2.0 #149471

Uh oh!

atalman commented Mar 19, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Mar 19, 2025 •

edited

Loading

Uh oh!

Uh oh!

malfet left a comment

Uh oh!

atalman commented Mar 19, 2025

Uh oh!

pytorchmergebot commented Mar 19, 2025

Uh oh!

atalman commented Mar 19, 2025

Uh oh!

pytorchbot commented Mar 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Pin auditwheel to 6.2.0 #149471

Pin auditwheel to 6.2.0 #149471

Uh oh!

Conversation

atalman commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/149471

❌ 18 New Failures, 10 Unrelated Failures

Uh oh!

Uh oh!

malfet left a comment

Choose a reason for hiding this comment

Uh oh!

atalman commented Mar 19, 2025

Uh oh!

pytorchmergebot commented Mar 19, 2025

Merge started

Uh oh!

atalman commented Mar 19, 2025

Uh oh!

pytorchbot commented Mar 19, 2025

Cherry picking #149471

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

atalman commented Mar 19, 2025 •

edited

Loading

pytorch-bot bot commented Mar 19, 2025 •

edited

Loading