Initial Python 3.12 build fixes #106083

albanD · 2023-07-26T22:41:07Z

This compiles with python 3.12
You can get numpy from https://anaconda.org/scientific-python-nightly-wheels/numpy/files so that you don't need to remove numpy from test files.

Basic core tests work but obviously dynamo and first class dims don't work.

cc @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @aakhundov

pytorch-bot · 2023-07-26T22:41:10Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/106083

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 9a50db8 with merge base 79b3a9f ():

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

linux-focal-rocm5.6-py3.8 / test (default, 1, 3, linux.rocm.gpu, unstable) (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Same treatment as many other objects such as https://github.com/pytorch/pytorch/blob/main/torch/csrc/autograd/python_hook.cpp#L99 This one can outlive the python runtime due to structs like: https://github.com/pytorch/pytorch/blob/2f35715f0d230f8d5527bdf01b9793f6613f3a2e/torch/csrc/autograd/python_cpp_function.cpp#L232 With the pybind patch and this one, the 3.12 build at #106083 stops segfaulting and runs test_autograd.py just fine. Pull Request resolved: #106334 Approved by: https://github.com/ezyang

ezyang · 2023-08-24T14:03:34Z

requirements.txt

 jinja2
 fsspec
+# setuptools was removed from default python install
+setuptools ; python_version >= "3.12"


Are we going to get people to stop using setup.py directly 🤔

I don't think we have a good alternative at the moment. The setuptools package doesn't seem to be going anywhere anytime soon: https://pypi.org/project/setuptools/

ezyang · 2023-08-24T14:05:13Z

this is clearly not ready, how would you like to proceed with review?

albanD · 2023-08-24T14:24:40Z

Actually I would argue this is ready for review.
Just like we did for 3.11 I think it is best to just land this so that each team can start fixing their own tests in parallel.
I can add more strict errors in Dynamo if you want? But the probability of anyone building from source with 3.12 is quite low I would say.

ezyang · 2023-08-24T14:53:40Z

torch/csrc/dynamo/cpython_defs.c

 #if PY_VERSION_HEX >= 0x030C0000 // 3.12
-#error "Please ensure that the functions below still match the CPython implementation for 3.12"
+// Spoiler alert: They don't! This will be done in a follow up.
+// #error "Please ensure that the functions below still match the CPython implementation for 3.12"


Are you planning to do this?

I am planning on doing some of the C++ cleanup to build a case to share with the CPython team yes.
I am not planning on making Dynamo fully work though. There will need to be some staffing from the Dynamo side to fix this (like we did for 3.11).

ezyang · 2023-08-24T14:54:27Z

torch/csrc/dynamo/eval_frame.c

+  #if IS_PYTHON_3_12_PLUS
+  // Most of these don't exist in 3.12 anymore.
+  // _PyFunction_CopyWithNewCode and _PyFrame_InitializeSpecials in particular
+  PyFunctionObject* func;


This seems bad. At minimum you should error in this branch right?

I can make that an error yes.

ezyang

sure I guess

albanD · 2023-08-24T21:30:18Z

@pytorchbot merge

pytorchmergebot · 2023-08-24T21:32:05Z

Merge failed

Reason: This PR needs a release notes: label
If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Details for Dev Infra team

Raised by workflow job

albanD · 2023-08-25T13:21:52Z

@pytorchbot merge

pytorchmergebot · 2023-08-25T13:23:43Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

This compiles with python 3.12 You can get numpy from https://anaconda.org/scientific-python-nightly-wheels/numpy/files so that you don't need to remove numpy from test files. Basic core tests work but obviously dynamo and first class dims don't work. Pull Request resolved: #106083 Approved by: https://github.com/ezyang

github-actions bot added the module: dynamo label Jul 26, 2023

This was referenced Jul 31, 2023

Do not check the GIL/decref object if the Python runtime is dead pybind/pybind11#4769

Closed

Do not call decref if python runtime is already dead #106334

Closed

albanD force-pushed the build_3.12 branch from 177db12 to 59e4a69 Compare August 16, 2023 01:09

albanD added 4 commits August 23, 2023 15:12

It compiles!

5cd7da7

botched 3.11

2b91de8

Fix build without 3.12 cherry pick

86b0226

Cleanup temporary comments/fix

81d7b26

albanD force-pushed the build_3.12 branch from 59e4a69 to 81d7b26 Compare August 23, 2023 19:26

Fix negative if statements

668d873

albanD changed the title ~~Quick and dirty Python 3.12 build fixes~~ Initial Python 3.12 build fixes Aug 23, 2023

albanD requested review from ezyang, williamwen42 and zdevito August 24, 2023 13:32

ezyang reviewed Aug 24, 2023

View reviewed changes

albanD marked this pull request as ready for review August 24, 2023 14:23

ezyang reviewed Aug 24, 2023

View reviewed changes

ezyang approved these changes Aug 24, 2023

View reviewed changes

Make dynamo error out more loudly

9a50db8

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Aug 24, 2023

pytorchmergebot added the merging label Aug 24, 2023

pytorchmergebot removed the merging label Aug 24, 2023

albanD added the topic: not user facing topic category label Aug 25, 2023

pytorchmergebot added the merging label Aug 25, 2023

pytorchmergebot added Merged and removed merging labels Aug 25, 2023

pytorchmergebot closed this in b9472de Aug 25, 2023

malfet mentioned this pull request Dec 14, 2024

Python 3.12 "from functorch.einops import rearrange" fails with "RuntimeError: First class dim doesn't work with python 3.12" #142032

Open

Initial Python 3.12 build fixes #106083

Initial Python 3.12 build fixes #106083

Uh oh!

Conversation

albanD commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/106083

✅ You can merge normally! (1 Unrelated Failure)

Uh oh!

ezyang Aug 24, 2023

Choose a reason for hiding this comment

Uh oh!

albanD Aug 24, 2023

Choose a reason for hiding this comment

Uh oh!

ezyang commented Aug 24, 2023

Uh oh!

albanD commented Aug 24, 2023

Uh oh!

ezyang Aug 24, 2023

Choose a reason for hiding this comment

Uh oh!

albanD Aug 24, 2023

Choose a reason for hiding this comment

Uh oh!

ezyang Aug 24, 2023

Choose a reason for hiding this comment

Uh oh!

albanD Aug 24, 2023

Choose a reason for hiding this comment

Uh oh!

ezyang left a comment

Choose a reason for hiding this comment

Uh oh!

albanD commented Aug 24, 2023

Uh oh!

pytorchmergebot commented Aug 24, 2023

Merge failed

Uh oh!

albanD commented Aug 25, 2023

Uh oh!

pytorchmergebot commented Aug 25, 2023

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

albanD commented Jul 26, 2023 •

edited

Loading

pytorch-bot bot commented Jul 26, 2023 •

edited

Loading