Add guard to use AMX for x86_64 only #129479

atalman · 2024-06-25T15:48:02Z

Trying to mitigate aarch64 and s390 nightly failures as per this comment:
#127195 (comment)

Fixes #129443

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10

pytorch-bot · 2024-06-25T15:48:05Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/129479

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 14 Unrelated Failures

As of commit 6833f63 with merge base 0e6bb7f ():

NEW FAILURES - The following jobs have failed:

pull / linux-focal-cuda12.1-py3.10-gcc9-bazel-test / build-and-test (default, 1, 1, lf.linux.4xlarge.nvidia.gpu) (gh)
Unable to load the kernel module 'nvidia.ko'. This happens most frequently when this kernel module was built against the wrong or improperly configured kernel sources, with a version of gcc that differs from the one used to build the target kernel, or if another driver, such as nouveau, is present and prevents the NVIDIA kernel module from obtaining ownership of the NVIDIA device(s), or no NVIDIA device installed in this system is supported by this NVIDIA Linux graphics driver release.
pull / linux-focal-cuda12.4-py3.10-gcc9-bazel-test / build-and-test (default, 1, 1, lf.linux.4xlarge.nvidia.gpu) (gh)
Unable to load the kernel module 'nvidia.ko'. This happens most frequently when this kernel module was built against the wrong or improperly configured kernel sources, with a version of gcc that differs from the one used to build the target kernel, or if another driver, such as nouveau, is present and prevents the NVIDIA kernel module from obtaining ownership of the NVIDIA device(s), or no NVIDIA device installed in this system is supported by this NVIDIA Linux graphics driver release.
windows-binary-wheel / wheel-py3_11-cpu-test (gh)
Process completed with exit code 1.
windows-binary-wheel / wheel-py3_12-cpu-test (gh)
Process completed with exit code 1.

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

linux-aarch64-binary-manywheel / manywheel-py3_9-cpu-aarch64-test / test (gh) (detected as infra flaky with no log or failing log classifier)
pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.8xlarge.nvidia.gpu) (gh) (matched linux rule in flaky-rules.json)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, lf.linux.8xlarge.nvidia.gpu) (gh) (matched linux rule in flaky-rules.json)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, lf.linux.8xlarge.nvidia.gpu) (gh) (matched linux rule in flaky-rules.json)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 1, 5, lf.linux.4xlarge.nvidia.gpu) (gh) (matched linux rule in flaky-rules.json)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 2, 5, lf.linux.4xlarge.nvidia.gpu) (gh) (matched linux rule in flaky-rules.json)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 3, 5, lf.linux.4xlarge.nvidia.gpu) (gh) (matched linux rule in flaky-rules.json)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 4, 5, lf.linux.4xlarge.nvidia.gpu) (gh) (matched linux rule in flaky-rules.json)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 5, 5, lf.linux.4xlarge.nvidia.gpu) (gh) (matched linux rule in flaky-rules.json)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

UNSTABLE - The following jobs failed but were likely due to flakiness present on trunk and has been marked as unstable:

pull / linux-focal-cuda12.1-py3.10-gcc9-experimental-split-build / test (default, 1, 5, lf.linux.4xlarge.nvidia.gpu, unstable) (gh) (#129080)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
pull / linux-focal-cuda12.1-py3.10-gcc9-experimental-split-build / test (default, 2, 5, lf.linux.4xlarge.nvidia.gpu, unstable) (gh) (#129080)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
pull / linux-focal-cuda12.1-py3.10-gcc9-experimental-split-build / test (default, 3, 5, lf.linux.4xlarge.nvidia.gpu, unstable) (gh) (#129080)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
pull / linux-focal-cuda12.1-py3.10-gcc9-experimental-split-build / test (default, 4, 5, lf.linux.4xlarge.nvidia.gpu, unstable) (gh) (#129080)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
pull / linux-focal-cuda12.1-py3.10-gcc9-experimental-split-build / test (default, 5, 5, lf.linux.4xlarge.nvidia.gpu, unstable) (gh) (#129080)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

kiszk · 2024-06-25T16:01:26Z

Looks good. I did not know aarch64 CI builds are also failing.

I am curious whether && !defined(__ANDROID__) is still necessary.

nWEIdia

Thanks for the fix!

swolchok · 2024-06-25T17:23:35Z

build is still broken with this for me on mac:

pytorch/aten/src/ATen/cpu/Utils.cpp:37:34: error: use of undeclared identifier 'cpuinfo_has_x86_amx_tile'; did you mean 'cpuinfo_has_x86_mmx_plus'?
  return cpuinfo_initialize() && cpuinfo_has_x86_amx_tile();

abhishek-iitmadras · 2024-06-25T17:25:18Z

Looks good. I did not know aarch64 CI builds are also failing.

I am curious whether && !defined(__ANDROID__) is still necessary.

Necessary to prevent errors on Android devices, which typically don't support AMX-specific system calls.

malfet

LGTM

malfet · 2024-06-25T17:29:25Z

build is still broken with this for me on mac:

pytorch/aten/src/ATen/cpu/Utils.cpp:37:34: error: use of undeclared identifier 'cpuinfo_has_x86_amx_tile'; did you mean 'cpuinfo_has_x86_mmx_plus'?
  return cpuinfo_initialize() && cpuinfo_has_x86_amx_tile();

@swolchok you just need to update the submodule, this call is available for all arches, but is dummy false on everything but x86_64

malfet · 2024-06-25T17:29:36Z

@pytorchbot merge -f "Lint is green"

kiszk · 2024-06-25T17:29:49Z

Necessary to prevent errors on Android devices, which typically don't support AMX-specific system calls.

Thanks. I did not know Android on x86.

pytorchmergebot · 2024-06-25T17:31:17Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Add guard to use AMX for x86_64 only

6833f63

pytorch-bot bot added the module: cpu CPU specific problem (e.g., perf, algorithm) label Jun 25, 2024

atalman added the ciflow/binaries_wheel Trigger binary build and upload jobs for wheel on the PR label Jun 25, 2024

nWEIdia approved these changes Jun 25, 2024

View reviewed changes

malfet approved these changes Jun 25, 2024

View reviewed changes

pytorchmergebot added the merging label Jun 25, 2024

pytorchmergebot closed this in e317a8b Jun 25, 2024

pytorchmergebot added Merged and removed merging labels Jun 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add guard to use AMX for x86_64 only #129479

Add guard to use AMX for x86_64 only #129479

Uh oh!

atalman commented Jun 25, 2024 •

edited by malfet

Loading

Uh oh!

pytorch-bot bot commented Jun 25, 2024 •

edited

Loading

Uh oh!

kiszk commented Jun 25, 2024

Uh oh!

nWEIdia left a comment

Uh oh!

swolchok commented Jun 25, 2024

Uh oh!

abhishek-iitmadras commented Jun 25, 2024

Uh oh!

malfet left a comment

Uh oh!

malfet commented Jun 25, 2024

Uh oh!

malfet commented Jun 25, 2024

Uh oh!

kiszk commented Jun 25, 2024

Uh oh!

pytorchmergebot commented Jun 25, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Add guard to use AMX for x86_64 only #129479

Add guard to use AMX for x86_64 only #129479

Uh oh!

Conversation

atalman commented Jun 25, 2024 • edited by malfet Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jun 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/129479

❌ 4 New Failures, 14 Unrelated Failures

Uh oh!

kiszk commented Jun 25, 2024

Uh oh!

nWEIdia left a comment

Choose a reason for hiding this comment

Uh oh!

swolchok commented Jun 25, 2024

Uh oh!

abhishek-iitmadras commented Jun 25, 2024

Uh oh!

malfet left a comment

Choose a reason for hiding this comment

Uh oh!

malfet commented Jun 25, 2024

Uh oh!

malfet commented Jun 25, 2024

Uh oh!

kiszk commented Jun 25, 2024

Uh oh!

pytorchmergebot commented Jun 25, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

atalman commented Jun 25, 2024 •

edited by malfet

Loading

pytorch-bot bot commented Jun 25, 2024 •

edited

Loading