Add Python 3.10 drop 3.5 by lidizheng · Pull Request #26074 · grpc/grpc

lidizheng · 2021-04-23T19:33:01Z

This PR includes:

Adding Python 3.10 binary wheels to macOS/Linux/Windows
Install 3.10 in Windows and macOS, and use a newer "manylinux2014" (I didn't noticed a tag was added, causing the script content hash mismatch with my local build, and failing artifacts build job)
Dropping Python 3.5 binary wheel compilation, and updating READMEs
Updating Python build script to avoid using the wrong pip
Remove ubuntu1604 from our distribtests since its default Python 3 is 3.5
Fix the invalid command: bdist_wheel error on macOS with 3.10
Use "manylinux_2_17" instead of "manylinux2014" since it has higher priority in latest auditwheel (one is just an alias of another)
In latest wheel, "manylinux2014" has higher priority than "manylinux_2_17". This PR let auditwheel decide the wheel name, and let wheel be a tool to pack/unpack wheels.
Upgraded the Debian image from stretch to buster for binary wheel installation test (the compile-from-source test is still jessie)

Given the complex details generated around changing supported Python versions, we might not in a stage of automating this once-per-year task.

This change is

stale · 2021-07-23T04:34:37Z

This issue/PR has been automatically marked as stale because it has not had any update (including commits, comments, labels, milestones, etc) for 30 days. It will be closed automatically if no further update occurs in 7 day. Thank you for your contributions!

lidizheng · 2021-08-27T19:37:08Z

Compiling with a manylinux2014 image but the binary wheel demands manylinux_2_17 or above?

+ auditwheel show dist/grpcio-1.41.0.dev0-cp36-cp36m-linux_x86_64.whl
+ tee /dev/stderr
+ grep -E -w manylinux2014_x86_64

grpcio-1.41.0.dev0-cp36-cp36m-linux_x86_64.whl is consistent with the
following platform tag: "manylinux_2_17_x86_64".

The wheel references external versioned symbols in these
system-provided shared libraries: librt.so.1 with versions
{'GLIBC_2.2.5'}, libm.so.6 with versions {'GLIBC_2.2.5'},
libpthread.so.0 with versions {'GLIBC_2.2.5', 'GLIBC_2.12'},
libstdc++.so.6 with versions {'GLIBCXX_3.4.18', 'GLIBCXX_3.4.11',
'GLIBCXX_3.4.9', 'GLIBCXX_3.4.15', 'GLIBCXX_3.4.14', 'CXXABI_1.3.5',
'GLIBCXX_3.4', 'CXXABI_1.3', 'GLIBCXX_3.4.17'}, libc.so.6 with
versions {'GLIBC_2.9', 'GLIBC_2.3.2', 'GLIBC_2.2.5', 'GLIBC_2.3',
'GLIBC_2.6', 'GLIBC_2.10', 'GLIBC_2.16', 'GLIBC_2.7', 'GLIBC_2.4'}

This constrains the platform tag to "manylinux_2_17_x86_64". In order
to achieve a more compatible tag, you would need to recompile a new
wheel from source on a system with earlier versions of these
libraries, such as a recent manylinux image.

tools/dockerfile/grpc_artifact_python_manylinux2014_x64/Dockerfile

gnossen

Nice work!

tools/dockerfile/grpc_artifact_python_manylinux2014_x64/Dockerfile

jtattermusch · 2021-09-01T11:02:29Z

tools/dockerfile/grpc_artifact_python_manylinux_2_24_aarch64/Dockerfile

 # Thanks to that, wheel build with this image aren't actually
 # compliant with manylinux2014, but only with manylinux_2_24
-FROM dockcross/manylinux2014-aarch64:20200929-608e6ac
+FROM dockcross/manylinux2014-aarch64:20210826-19322ba


It definitely sounds like updating the the image tag here would require updating the above comment as well.
Based on my knowledge, if you used a newer version of image than 20200929-608e6ac, you won't be able to build (precisely because of what's decribed in the comment).

Currently, all artifacts building jobs are green. The Linux Python artifacts build includes the aarch64 platform. ~~Does this mean by using this tag (20210826-19322ba) we are free from the commented issue?~~

We might still need the AUDITWHEEL_PLAT override. I don't have enough context around this issue. What do you think I should update in the comment section?

Ok, so the background is a bit complicated.

in the past, we had to use a version of dockcross/manylinux2014-aarch64 before Update GCC to 4.8.5 and GLIBC to 2.17 in manylinux2014-aarch64 dockcross/dockcross#449, because newer versions didn't had a too old version of gcc and it was impossible to build grpc with them (and that had a side effect of non being able to generate manylinux 2_17 images, only manylinux 2_24). This is what the comment is about.

just recently, in Add manylinux2014-aarch64 with gcc=9.3 dockcross/dockcross#562, the dockcross/manylinux2014-aarch64 image was improved to contain gcc9.3, which made the limitation from previous bulletpoint go away.

So, as a result:

what you currently have works, just the comment "We use an older version of dockcross image..." is now out of date.

the wheels generated by this image now are actually manylinux_2_17 compliant (not just manylinux_2_24). So in a followup PR, we could actually update the docker image name and some other details. But let's not do this here as what you currently have seems to be working just fine.

veblush · 2021-09-01T15:39:27Z

tools/dockerfile/grpc_artifact_python_manylinux2014_x64/Dockerfile

-# Use a tagged version to use Python 3.5
-FROM quay.io/pypa/manylinux2014_x86_64:2021-05-01-28d233a
+# Use a tagged version to use Python 3.10
+FROM quay.io/pypa/manylinux2014_x86_64:2021-08-26-12e5da0


Lidi, you don't need to use a tag version here. Just quay.io/pypa/manylinux2014_x86_64 would be better because we can get the latest image once actual python 3.10 is released.

Doesn't this open us up to non-hermetic breakage?

@veblush I think when a newer upstream Docker image is released, it would be better if we can write a PR for it. We still need to manually build and push the images.

As Richard mentioned, if a certain image version introduce a breakage, it could be hard to debug.

+1 to tagging the version and making tests more hermetic that way.

veblush

Thank you for taking care of this!

jtattermusch

Added some clarifications, the aarch64 wheel seems to be working fine, but we might want a small followup cleanup PR after this gets merged.

jtattermusch · 2021-09-02T06:22:40Z

tools/dockerfile/grpc_artifact_python_manylinux2014_x64/Dockerfile

-# Use a tagged version to use Python 3.5
-FROM quay.io/pypa/manylinux2014_x86_64:2021-05-01-28d233a
+# Use a tagged version to use Python 3.10
+FROM quay.io/pypa/manylinux2014_x86_64:2021-08-26-12e5da0


+1 to tagging the version and making tests more hermetic that way.

jtattermusch · 2021-09-02T06:56:58Z

tools/dockerfile/grpc_artifact_python_manylinux_2_24_aarch64/Dockerfile

 # Thanks to that, wheel build with this image aren't actually
 # compliant with manylinux2014, but only with manylinux_2_24
-FROM dockcross/manylinux2014-aarch64:20200929-608e6ac
+FROM dockcross/manylinux2014-aarch64:20210826-19322ba


Ok, so the background is a bit complicated.

in the past, we had to use a version of dockcross/manylinux2014-aarch64 before Update GCC to 4.8.5 and GLIBC to 2.17 in manylinux2014-aarch64 dockcross/dockcross#449, because newer versions didn't had a too old version of gcc and it was impossible to build grpc with them (and that had a side effect of non being able to generate manylinux 2_17 images, only manylinux 2_24). This is what the comment is about.

just recently, in Add manylinux2014-aarch64 with gcc=9.3 dockcross/dockcross#562, the dockcross/manylinux2014-aarch64 image was improved to contain gcc9.3, which made the limitation from previous bulletpoint go away.

So, as a result:

what you currently have works, just the comment "We use an older version of dockcross image..." is now out of date.

the wheels generated by this image now are actually manylinux_2_17 compliant (not just manylinux_2_24). So in a followup PR, we could actually update the docker image name and some other details. But let's not do this here as what you currently have seems to be working just fine.

* Add Python 3.10.0rc1 binary wheels * Drop Python 3.5 artifacts * Document the drop of 3.5 * Fix the wrong pip pointer * Update manylinux2014 to a newer version, remove 3.5 distribtest * Update manylinux aarch64 to see if the absl error go away * Use the preferred alias * Allow different wheel library to produce different tag order * Remove unused shell var and log produced wheels * Use copy instead of move * Make bash happy about the wildcard * Upgrade the debian image to use 3.5+ Python * Polish the comments for the Dockerfiles

lidizheng added lang/Python release notes: yes Indicates if PR needs to be in release notes labels Apr 23, 2021

stale bot added the disposition/stale label Jul 23, 2021

lidizheng added disposition/never stale and removed disposition/stale labels Jul 23, 2021

lidizheng force-pushed the python-3.10.0 branch 3 times, most recently from 3dadded to c2a0bfa Compare August 24, 2021 21:04

lidizheng added the kokoro:force-run label Aug 25, 2021

grpc-kokoro removed the kokoro:force-run label Aug 25, 2021

lidizheng changed the title ~~Add Python 3.10 artifacts~~ Add Python 3.10 drop 3.5 Aug 26, 2021

haberman mentioned this pull request Aug 26, 2021

Added release notes indicating that 3.17.3 is the last Python 2.x release. protocolbuffers/protobuf#8901

Merged

lidizheng added 5 commits August 27, 2021 10:23

Add Python 3.10.0rc1 binary wheels

c8f83c0

Drop Python 3.5 artifacts

9e9c249

Document the drop of 3.5

4f5dc4b

Fix the wrong pip pointer

afff9b7

Update manylinux2014 to a newer version, remove 3.5 distribtest

d336a35

lidizheng force-pushed the python-3.10.0 branch from 05293b3 to d336a35 Compare August 27, 2021 17:37

Update manylinux aarch64 to see if the absl error go away

833b03d

Use the preferred alias

45892ce

lidizheng added the kokoro:force-run label Aug 27, 2021

grpc-kokoro removed the kokoro:force-run label Aug 27, 2021

lidizheng added 5 commits August 30, 2021 00:02

Allow different wheel library to produce different tag order

c6d0574

Remove unused shell var and log produced wheels

6fbf790

Use copy instead of move

8cf7558

Make bash happy about the wildcard

cdb9d8a

Upgrade the debian image to use 3.5+ Python

fcfff69

lidizheng marked this pull request as ready for review August 31, 2021 23:45

lidizheng requested review from apolcyn, jtattermusch and nicolasnoble as code owners August 31, 2021 23:45

lidizheng requested review from gnossen and veblush August 31, 2021 23:46

lidizheng assigned gnossen and veblush Aug 31, 2021

gnossen reviewed Aug 31, 2021

View reviewed changes

tools/dockerfile/grpc_artifact_python_manylinux2014_x64/Dockerfile Show resolved Hide resolved

gnossen approved these changes Aug 31, 2021

View reviewed changes

lidizheng assigned jtattermusch Aug 31, 2021

jtattermusch reviewed Sep 1, 2021

View reviewed changes

tools/dockerfile/grpc_artifact_python_manylinux2014_x64/Dockerfile Outdated Show resolved Hide resolved

jtattermusch reviewed Sep 1, 2021

View reviewed changes

veblush reviewed Sep 1, 2021

View reviewed changes

veblush approved these changes Sep 1, 2021

View reviewed changes

Polish the comments for the Dockerfiles

7666d97

jtattermusch approved these changes Sep 2, 2021

View reviewed changes

lidizheng merged commit 4729f6f into grpc:master Sep 2, 2021

copybara-service bot added the imported Specifies if the PR has been imported to the internal repository label Sep 2, 2021

lidizheng mentioned this pull request Sep 2, 2021

Use manylinux_2_17 instead of manylinux_2_24 tag for manylinux2014 aarch64 wheels #27239

Closed

gnossen mentioned this pull request Sep 3, 2021

Fix Artifact Build Extra #27247

Merged

jtattermusch mentioned this pull request Sep 8, 2021

Use manylinux_2_17 instead of manylinux_2_24 tag for manylinux2014 aarch64 wheels #27280

Merged

ealap mentioned this pull request Oct 6, 2021

Bump grpcio from 1.40.0 to 1.41.0 in /Browser MarketSquare/robotframework-browser#1349

Merged

Conversation

lidizheng commented Apr 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stale bot commented Jul 23, 2021

Uh oh!

lidizheng commented Aug 27, 2021

Uh oh!

Uh oh!

gnossen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidizheng Sep 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

veblush left a comment

Choose a reason for hiding this comment

Uh oh!

jtattermusch left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

lidizheng commented Apr 23, 2021 •

edited

Loading

lidizheng Sep 1, 2021 •

edited

Loading