[MPS] Add support for autocast in MPS #99272

kulinseth · 2023-04-16T19:40:46Z

Fixes #88415

Need to run inductor/test_cpu_select_algorithm

cc @mcarilli @ptrblck @leslie-fang-intel @jgong5

pytorch-bot · 2023-04-16T19:40:50Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/99272

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit c5aec31 with merge base 71383dd ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

albanD

Sounds pretty good. Mostly questions about testing.

albanD · 2023-04-26T15:19:51Z

aten/src/ATen/autocast_mode.cpp

+  m.fallback(torch::CppFunction::makeFallthrough());
+}
+
+TORCH_LIBRARY_IMPL(aten, AutocastMPS, m) {


Not really a blocker for the PR, are these actually different from the fp16 rules we already have for autograd?

torch/csrc/autograd/init.cpp

torch/utils/checkpoint.py

test/test_mps.py

test/test_autocast.py

johnrichardrinehart · 2023-06-08T11:13:30Z

@kulinseth It seems there is still some work to be done, here. What's your plan to address some of these comments?

albanD · 2023-06-22T02:14:27Z

Any update on this @kulinseth ?

kulinseth · 2023-06-23T01:02:13Z

Any update on this @kulinseth ?

Yes, I need to rebase and update here. I will do it this week.

johnrichardrinehart · 2023-06-29T13:20:10Z

@kulinseth Still in progress? Looks so based on #104191 .

djollieb · 2023-08-10T12:26:22Z

@kulinseth Is there any update on this please? This change would fix so many issues. Thank you

kulinseth · 2023-08-10T14:18:29Z

@kulinseth Is there any update on this please? This change would fix so many issues. Thank you

Yes , I will cleanup and rebase again . Will have something by early next week to try

johnrichardrinehart · 2023-08-19T18:19:04Z

@kulinseth Is there any update on this please? This change would fix so many issues. Thank you

Yes , I will cleanup and rebase again . Will have something by early next week to try

Any updates @kulinseth ?

diffractometer · 2023-09-04T15:07:03Z

@kulinseth I'll buy you a coffee!

john1089 · 2023-09-09T03:49:07Z

Any updates @kulinseth ?

PhilippeFerreiraDeSousa · 2023-09-18T04:02:42Z

@kulinseth Can we do something to help?

malfet · 2024-09-05T23:17:22Z

can bf16 remain as an option, please?

@bghira can you please elaborate why/when you need something like that? But if this is the case, we definitely should extend it in follow-up PR, but perhaps rules should be different that for fp16, aren't they?

pytorchmergebot · 2024-09-05T23:17:36Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Fixes pytorch#88415 Need to run inductor/test_cpu_select_algorithm Pull Request resolved: pytorch#99272 Approved by: https://github.com/malfet Co-authored-by: Siddharth Kotapati <[email protected]> Co-authored-by: Nikita Shulga <[email protected]> Co-authored-by: Roy Hvaara <[email protected]>

bastienjalbert · 2024-10-12T12:34:55Z

@malfet, @skotapati why the PR has been closed ? And if the code has been effectively merged, on which release would it be available ?

Note: May be unnecessary after PyTorch PR (ref: pytorch/pytorch#99272) is released.

hvaara · 2024-10-15T10:36:30Z

@malfet, @skotapati why the PR has been closed ?

@bastienjalbert The PR is closed because merging code in PyTorch is more complicated than GitHub's default workflow and is handled by bot commands. If you're interested in learning more about what goes on in the process you can take a look at the workflow file. When the workflow completes successfully the bot closes the PR

The changes proposed in this PR was added in main in 144fde4.

And if the code has been effectively merged, on which release would it be available ?

It is in the release branch for 2.5 and I expect it'll be available in 2.5.0.

awmartin · 2024-10-28T04:36:59Z

I get the following warning for autocast on MPS using pytorch 2.6.0 nightly:

sam2/.env/lib/python3.11/site-packages/torch/amp/autocast_mode.py:332: UserWarning: In MPS autocast, but the target dtype is not supported. Disabling autocast.
MPS Autocast only supports dtype of torch.bfloat16 currently.

My offending code (taken from Segment Anything 2.1 sample code) appears to reference the proper dtype:

with torch.inference_mode(), torch.autocast("mps", dtype=torch.bfloat16):

However, it seems the actual supported type is torch.float16 as per this line. Is this expected? Or is the warning message on line 330 incorrect?

hvaara · 2024-10-29T13:59:54Z

I created #139190 to add some context and track a fix. tl;dr: The error message is incorrect and should be updated. Thanks for reporting the issue, @awmartin!

hvaara · 2024-10-31T13:05:50Z

I added a proposal to extend MPS autocast support to bf16 in #139390.

cc @awmartin

awmartin · 2024-10-31T16:57:08Z

@hvaara Much appreciated!

* Activate torch.cuda.amp.autocast() for roformer inference * Use `autocast()` for all inference. Update torch (2.3.1 -> 2.4.1) to use `is_autocast_available()`. * Fix autocast error on Apple Silicon Macs Note: May be unnecessary after PyTorch PR (ref: pytorch/pytorch#99272) is released. * Small refactor * A new flag `use_autocast` is added to Separator class and CLI. - Default value is False - Autocast is used only if flag is True and device supports it * Update README * Small refactor

This PR adds support for bf16 autocast. Most of the code and ideas are copied from #99272. Most of the heavy lifting was done by AI. Fixes #139386 Pull Request resolved: #139390 Approved by: https://github.com/malfet Co-authored-by: Kulin Seth <[email protected]> Co-authored-by: Nikita Shulga <[email protected]>

This PR adds support for bf16 autocast. Most of the code and ideas are copied from pytorch#99272. Most of the heavy lifting was done by AI. Fixes pytorch#139386 Pull Request resolved: pytorch#139390 Approved by: https://github.com/malfet Co-authored-by: Kulin Seth <[email protected]> Co-authored-by: Nikita Shulga <[email protected]>

kulinseth requested review from albanD and soulitzer as code owners April 16, 2023 19:40

pytorch-bot bot added ciflow/mps Run MPS tests (subset of trunk) release notes: jit release notes category labels Apr 16, 2023

github-actions bot added the module: amp (automated mixed precision) autocast label Apr 16, 2023

pytorchbot added the open source label Apr 16, 2023

kulinseth added the ciflow/trunk Trigger trunk jobs on your pull request label Apr 21, 2023

mikaylagawarecki added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Apr 25, 2023

kulinseth force-pushed the autocast branch 2 times, most recently from 786f3f1 to 424cf41 Compare April 26, 2023 06:25

albanD reviewed Apr 26, 2023

View reviewed changes

faraday mentioned this pull request Jun 13, 2023

Feature Request: Support Apple Silicon facebookresearch/audiocraft#31

Open

kulinseth mentioned this pull request Jun 27, 2023

torch.embedding: Trying to convert BFloat16 to the MPS backend but it does not have support for that dtype. #104191

Closed

drscotthawley mentioned this pull request Jun 30, 2023

Apple MPS Compatibility fastai/diffusion-nbs#17

Closed

tmm1 mentioned this pull request Jun 30, 2023

AttributeError: module 'torch.mps' has no attribute 'is_autocast_enabled' #104478

Closed

derekelewis mentioned this pull request Jul 3, 2023

MPS 16Bit Not Working correctly #78168

Closed

Mathanraj-Sharma mentioned this pull request Jul 19, 2023

MPT-7B, 30B RuntimeError: Placeholder storage has not been allocated on MPS device! h2oai/h2ogpt#463

Closed

erip mentioned this pull request Sep 15, 2023

add MPS support with a few tiny hacks. facebookresearch/nougat#78

Merged

slashdottir mentioned this pull request Sep 19, 2023

Can it run on Mac M1 (Apple silicon?) pharmapsychotic/clip-interrogator#83

Open

malfet removed the release notes: jit release notes category label Sep 5, 2024

pytorchmergebot added the merging label Sep 5, 2024

pytorchmergebot closed this in 144fde4 Sep 5, 2024

pytorchmergebot removed the merging label Sep 5, 2024

ntamotsu added a commit to ntamotsu/fork_python-audio-separator that referenced this pull request Oct 12, 2024

Fix autocast error on Apple Silicon Macs

08aecb1

Note: May be unnecessary after PyTorch PR (ref: pytorch/pytorch#99272) is released.

hvaara mentioned this pull request Oct 29, 2024

[MPS] Typo in error message for supported autocast type #139190

Closed

This was referenced Oct 31, 2024

[MPS] Extend autocast support to bf16 #139386

Closed

[MPS] Add support for bf16 autocast #139390

Closed

hvaara mentioned this pull request Nov 28, 2024

[MPS] Autocast fails for F.scaled_dot_product_attention #141774

Closed

kurzdev mentioned this pull request Dec 9, 2024

Enable GradScaler for MPS devices #142397

Closed

laclouis5 mentioned this pull request Dec 13, 2024

MPS Mixed-precision Autocast Lightning-AI/pytorch-lightning#20497

Open

laclouis5 mentioned this pull request Jan 6, 2025

Support for MPS device mixed-precision Lightning-AI/pytorch-lightning#20531

Closed

7 tasks

jacobsela mentioned this pull request Jan 19, 2025

Fix open-clip-torch model inference voxel51/fiftyone#5395

Merged

SunMarc mentioned this pull request Jan 29, 2025

Add bf16/fp16 support for amp with mps device huggingface/accelerate#3373

Merged

fmfeng mentioned this pull request Apr 24, 2025

Does it support single-machine multi-GPU running? wooyeolbaek/attention-map-diffusers#20

Open

[MPS] Add support for autocast in MPS #99272

[MPS] Add support for autocast in MPS #99272

Uh oh!

Conversation

kulinseth commented Apr 16, 2023 • edited by clee2000 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Apr 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/99272

✅ No Failures

Uh oh!

albanD left a comment

Choose a reason for hiding this comment

Uh oh!

albanD Apr 26, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

johnrichardrinehart commented Jun 8, 2023

Uh oh!

albanD commented Jun 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kulinseth commented Jun 23, 2023

Uh oh!

johnrichardrinehart commented Jun 29, 2023

Uh oh!

djollieb commented Aug 10, 2023

Uh oh!

kulinseth commented Aug 10, 2023

Uh oh!

johnrichardrinehart commented Aug 19, 2023

Uh oh!

diffractometer commented Sep 4, 2023

Uh oh!

john1089 commented Sep 9, 2023

Uh oh!

PhilippeFerreiraDeSousa commented Sep 18, 2023

Uh oh!

malfet commented Sep 5, 2024

Uh oh!

pytorchmergebot commented Sep 5, 2024

Merge started

Uh oh!

bastienjalbert commented Oct 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hvaara commented Oct 15, 2024

Uh oh!

awmartin commented Oct 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hvaara commented Oct 29, 2024

Uh oh!

hvaara commented Oct 31, 2024

Uh oh!

awmartin commented Oct 31, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

kulinseth commented Apr 16, 2023 •

edited by clee2000

Loading

pytorch-bot bot commented Apr 16, 2023 •

edited

Loading

albanD commented Jun 22, 2023 •

edited

Loading

bastienjalbert commented Oct 12, 2024 •

edited

Loading

awmartin commented Oct 28, 2024 •

edited

Loading