[Inductor] don't set XBLOCK larger than xnumel #138730

shunting314 · 2024-10-23T19:03:11Z

Stack from ghstack (oldest at bottom):

When fp8 dtype is involved, Inductor may set min_elem_per_thread to be a positive value. This will force increasing XBLOCK even for a small xnumel (e.g. 1). Inductor will report an error later when sanity check the triton config.

The simple fix here is to just not let XBLOCK to be larger than xnumel.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang @aakhundov

[ghstack-poisoned]

pytorch-bot · 2024-10-23T19:03:15Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/138730

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (15 Unrelated Failures)

As of commit 7df49a6 with merge base c6609ec ():

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
stable_diffusion_unet
inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
stable_diffusion_unet
inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
stable_diffusion_unet

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

inductor / cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (trunk failure)
stable_diffusion_unet
inductor / cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (trunk failure)
stable_diffusion_unet
inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (trunk failure)
stable_diffusion_unet
inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_aot_inductor_amp_freezing_torchbench, 2, 2, linux.12xlarge) (gh) (trunk failure)
stable_diffusion_unet
inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_aot_inductor_freezing_torchbench, 2, 2, linux.12xlarge) (gh) (trunk failure)
stable_diffusion_unet
inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_amp_freezing_torchbench, 2, 2, linux.16xlarge.spr) (gh) (trunk failure)
stable_diffusion_unet
inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_freezing_avx2_torchbench, 2, 2, linux.10xlarge.avx2) (gh) (trunk failure)
stable_diffusion_unet
inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_freezing_torchbench, 2, 2, linux.12xlarge) (gh) (trunk failure)
stable_diffusion_unet
inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_torchbench, 2, 2, linux.12xlarge) (gh) (trunk failure)
stable_diffusion_unet
inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_aot_inductor_amp_freezing_torchbench, 2, 2, linux.12xlarge) (gh) (trunk failure)
stable_diffusion_unet
inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_aot_inductor_freezing_torchbench, 2, 2, linux.12xlarge) (gh) (trunk failure)
stable_diffusion_unet
inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_torchbench, 2, 2, linux.12xlarge) (gh) (trunk failure)
stable_diffusion_unet

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: 32b61c2 Pull Request resolved: #138730

Chillee

Honestly I don't really understand all of our scaling up/down code, and I've run into a couple bugs with it before.

Is this the right way to solve it or just a patch?

shunting314 · 2024-10-23T19:09:02Z

Is this the right way to solve it or just a patch?

The stacked PR fails some internal tests: https://www.internalfb.com/diff/D64796587 . Those tests exists in OSS but we don't run it in OSS CI since we don't have h100...

This PR is a patch for quick fix those test failures.

A fundamental fix may need us to auditing all the scaling rules for BLOCK size and all kind of overriding (like min_elem_per_thread).

Chillee

Can you open an issue to track refactoring our scaling logic here? I think it's pretty messy.

shunting314 · 2024-10-23T20:45:01Z

Sure, here is the issue: #138743 . @Chillee

shunting314 · 2024-10-23T20:46:00Z

@pytorchbot merge

pytorchmergebot · 2024-10-23T20:47:35Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-10-23T21:25:20Z

Merge failed

Reason: 12 jobs have failed, first few of them are: inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_torchbench, 2, 2, linux.12xlarge), inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_freezing_torchbench, 2, 2, linux.12xlarge), inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_amp_freezing_torchbench, 2, 2, linux.16xlarge.spr), inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_torchbench, 2, 2, linux.12xlarge), inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_aot_inductor_freezing_torchbench, 2, 2, linux.12xlarge)

Details for Dev Infra team

Raised by workflow job

shunting314 · 2024-10-23T23:40:46Z

@pytorchbot merge -i

pytorchmergebot · 2024-10-23T23:42:36Z

Merge started

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-10-23T23:43:02Z

Merge failed

Reason: 15 jobs have failed, first few of them are: inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_torchbench, 2, 2, linux.12xlarge), inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_freezing_torchbench, 2, 2, linux.12xlarge), inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_amp_freezing_torchbench, 2, 2, linux.16xlarge.spr), inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_torchbench, 2, 2, linux.12xlarge), inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_aot_inductor_freezing_torchbench, 2, 2, linux.12xlarge)

Details for Dev Infra team

Raised by workflow job

shunting314 · 2024-10-24T18:20:25Z

@pytorchbot merge

pytorchmergebot · 2024-10-24T18:22:12Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-10-24T18:22:49Z

Merge failed

Reason: 15 jobs have failed, first few of them are: inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_torchbench, 2, 2, linux.12xlarge), inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_freezing_torchbench, 2, 2, linux.12xlarge), inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_amp_freezing_torchbench, 2, 2, linux.16xlarge.spr), inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_torchbench, 2, 2, linux.12xlarge), inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_aot_inductor_freezing_torchbench, 2, 2, linux.12xlarge)

Details for Dev Infra team

Raised by workflow job

shunting314 · 2024-10-24T18:24:03Z

@pytorchbot merge -i

pytorchmergebot · 2024-10-24T18:25:55Z

Merge started

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-10-24T18:26:26Z

Merge failed

Reason: 15 jobs have failed, first few of them are: inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_torchbench, 2, 2, linux.12xlarge), inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_freezing_torchbench, 2, 2, linux.12xlarge), inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_amp_freezing_torchbench, 2, 2, linux.16xlarge.spr), inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_torchbench, 2, 2, linux.12xlarge), inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_aot_inductor_freezing_torchbench, 2, 2, linux.12xlarge)

Details for Dev Infra team

Raised by workflow job

shunting314 · 2024-10-24T18:28:49Z

@pytorchbot merge -f '..'

pytorch-bot · 2024-10-24T18:28:53Z

You need to provide a reason for using force merge, in the format @pytorchbot merge -f 'Explanation'.
The explanation needs to be clear on why this is needed. Here are some good examples:

Bypass checks due to unrelated upstream failures from ...
This is a minor fix to ..., which shouldn't break anything
This is pre-tested in a previous CI run
Bypass flaky ... check

shunting314 · 2024-10-24T18:29:21Z

@pytorchbot merge -f "merge -i does not work"

pytorchmergebot · 2024-10-24T18:30:45Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

[Inductor] don't set XBLOCK larger than xnumel

7df49a6

[ghstack-poisoned]

shunting314 added a commit that referenced this pull request Oct 23, 2024

[Inductor] don't set XBLOCK larger than xnumel

81ed827

ghstack-source-id: 32b61c2 Pull Request resolved: #138730

pytorch-bot bot added ciflow/inductor module: inductor labels Oct 23, 2024

shunting314 requested review from Chillee, eellison and jansel October 23, 2024 19:03

Chillee reviewed Oct 23, 2024

View reviewed changes

Chillee approved these changes Oct 23, 2024

View reviewed changes

shunting314 mentioned this pull request Oct 23, 2024

Cleanup the scaling logic in runtime.triton_heuristics.triton_config #138743

Open

shunting314 added the topic: not user facing topic category label Oct 23, 2024

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 23, 2024

pytorchmergebot added the merging label Oct 23, 2024

pytorchmergebot removed the merging label Oct 23, 2024

This was referenced Oct 23, 2024

[inductor] don't fuse two nodes if likely increase peak memory #138756

Closed

[inductor] collect memory snapshort in the wrapper #138429

Closed

[Inductor] auto-chunker #136702

Open

pytorchmergebot added the merging label Oct 23, 2024

pytorchmergebot removed the merging label Oct 23, 2024

pytorchmergebot added the merging label Oct 24, 2024

pytorchmergebot removed the merging label Oct 24, 2024

pytorchmergebot added the merging label Oct 24, 2024

pytorchmergebot removed the merging label Oct 24, 2024

pytorchmergebot added the merging label Oct 24, 2024

pytorchmergebot added the Merged label Oct 24, 2024

pytorchmergebot closed this in 239a21f Oct 24, 2024

pytorchmergebot removed the merging label Oct 24, 2024

[Inductor] don't set XBLOCK larger than xnumel #138730

[Inductor] don't set XBLOCK larger than xnumel #138730

Uh oh!

Conversation

shunting314 commented Oct 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/138730

✅ You can merge normally! (15 Unrelated Failures)

Uh oh!

Chillee left a comment

Choose a reason for hiding this comment

Uh oh!

shunting314 commented Oct 23, 2024

Uh oh!

Chillee left a comment

Choose a reason for hiding this comment

Uh oh!

shunting314 commented Oct 23, 2024

Uh oh!

shunting314 commented Oct 23, 2024

Uh oh!

pytorchmergebot commented Oct 23, 2024

Merge started

Uh oh!

pytorchmergebot commented Oct 23, 2024

Merge failed

Uh oh!

shunting314 commented Oct 23, 2024

Uh oh!

pytorchmergebot commented Oct 23, 2024

Merge started

Uh oh!

pytorchmergebot commented Oct 23, 2024

Merge failed

Uh oh!

shunting314 commented Oct 24, 2024

Uh oh!

pytorchmergebot commented Oct 24, 2024

Merge started

Uh oh!

pytorchmergebot commented Oct 24, 2024

Merge failed

Uh oh!

shunting314 commented Oct 24, 2024

Uh oh!

pytorchmergebot commented Oct 24, 2024

Merge started

Uh oh!

pytorchmergebot commented Oct 24, 2024

Merge failed

Uh oh!

shunting314 commented Oct 24, 2024

Uh oh!

pytorch-bot bot commented Oct 24, 2024

Uh oh!

shunting314 commented Oct 24, 2024

Uh oh!

pytorchmergebot commented Oct 24, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shunting314 commented Oct 23, 2024 •

edited

Loading

pytorch-bot bot commented Oct 23, 2024 •

edited

Loading