Workaround #26279 by publishing manylinux_2_24 wheels instead of manylinux2014 on aarch64 by jtattermusch · Pull Request #26430 · grpc/grpc

jtattermusch · 2021-06-04T11:30:53Z

Context:

the dockcross/manylinux2014-aarch64 image provides a broken version of libstdc++ that (if statically linked), causes free(): invalid pointer crash on specific import order (on ARM64 linux) #26279
we need to use a version of dockcross/manylinux2014-aarch64 that has gcc newer than 4.8 (because otherwise we won't be able to build gRPC C core, since the compiler would be too old).
When a version of dockcross/manylinux2014-aarch64 that has gcc 4.9.4, the resulting wheel won't be manylinux2014 compliant since the libstdc++ is too new (but at the same time we can't statically link libstdc++ since we know that's broken for some reason).
the only reasonable solution in short term seems to be to compromise on the level of binary compatibility and publish manylinux_2_24 compliant wheels instead of manylinux2014 (see https://www.python.org/dev/peps/pep-0600/). Then we can stop statically linking libstdc++ and that in turn fixes free(): invalid pointer crash on specific import order (on ARM64 linux) #26279.

(the armv7l wheels were suffering from the same problem as reported in #26279, so I'm removing static linking of libstdc++ for armv7 as well).

jtattermusch · 2021-06-04T11:32:01Z

If the solution from this PR looks good, I'll backport into v1.38.x branch as well.

gnossen

Very nice detective work!

If you get the chance, I'd love to know how you figured out the root cause.

lidizheng

Great finding!

lidizheng · 2021-06-04T17:29:35Z

tools/run_tests/artifacts/artifact_targets.py

-        PythonArtifact('manylinux2014', 'aarch64', 'cp37-cp37m'),
-        PythonArtifact('manylinux2014', 'aarch64', 'cp38-cp38'),
-        PythonArtifact('manylinux2014', 'aarch64', 'cp39-cp39'),
+        PythonArtifact('manylinux_2_24', 'aarch64', 'cp36-cp36m'),


As you mentioned in #26279 (comment), 3.6 isn't impacted by this bug. Do we want to keep 3.6 wheel on manylinux2014?

I think it's better to stay consistent and leave all the aarch64 wheels as manylinux_2_24.

staying consistent is better

it seems that something's wrong with libstdc++ in the dockross manylinux image and we don't know what exactly it is, and it could simply be that with python3.6 the reproduction we have happens not to crash, but things could crash under slighly different conditions. it's better to avoid that.

specialcasing python3.6 doesn't seem to make sense since python3.6 will go end-of-life later this year.

jtattermusch · 2021-06-07T07:23:22Z

Known failures:
#25614

jtattermusch · 2021-06-07T07:56:19Z

FTR, these are the auditwheel outputs for the wheels built on kokoro:

grpcio-1.39.0.dev0-cp38-cp38-manylinux_2_24_aarch64.whl is consistent
with the following platform tag: "manylinux_2_24_aarch64".

The wheel references external versioned symbols in these system-
provided shared libraries: libpthread.so.0 with versions
{'GLIBC_2.17'}, libm.so.6 with versions {'GLIBC_2.17'}, libc.so.6 with
versions {'GLIBC_2.17'}, libstdc++.so.6 with versions {'CXXABI_1.3.5',
'GLIBCXX_3.4.18', 'GLIBCXX_3.4.20', 'CXXABI_1.3', 'GLIBCXX_3.4.9',
'GLIBCXX_3.4.15', 'GLIBCXX_3.4.11', 'GLIBCXX_3.4.17',
'GLIBCXX_3.4.14', 'GLIBCXX_3.4'}

grpcio_tools-1.39.0.dev0-cp38-cp38-manylinux_2_24_aarch64.whl is
consistent with the following platform tag: "manylinux_2_24_aarch64".

The wheel references external versioned symbols in these system-
provided shared libraries: libgcc_s.so.1 with versions {'GCC_3.0',
'GCC_4.2.0'}, libpthread.so.0 with versions {'GLIBC_2.17'}, libc.so.6
with versions {'GLIBC_2.17'}, libstdc++.so.6 with versions
{'GLIBCXX_3.4.11', 'GLIBCXX_3.4.20', 'CXXABI_1.3', 'GLIBCXX_3.4.9',
'GLIBCXX_3.4.14', 'GLIBCXX_3.4', 'CXXABI_1.3.8', 'GLIBCXX_3.4.18',
'CXXABI_1.3.5'}

* workaround #26279 at the expense of binary compatibility * correctly tag aarch64 wheels as manylinux_2_24 * remove unnecessary comments left by #26430

* Bump version to v1.38.0-pre1 (#26208) * Bump version to v1.38.0-pre1 * Regenerate projects * [Backport][v1.38.x] Add EventEngine's port.h to the podspec (#26222) Backport of #26220 to v1.38.x. This is the only backport, since the EventEngine code was introduced after v1.37 was cut. This file is needed for the Objective-C cocoapod build. * fix xds resolver to add XdsClient to channel args even on errors (#26282) (#26288) Co-authored-by: Mark D. Roth <[email protected]> * Backport config update timeout change to v1.38.x (#26276) * Enable Traffic Director time tracer (#26111) * Just update the timeout (#26223) * Increase xds job timeouts (#26187) * Revert grpc_xds_k8s job timeouts back to 120 mins (#26219) * Revert grpc_xds_k8s_python timeout to 120mins (#26221) Seems to have been inadvertently increased in #26187 Co-authored-by: Eric Gribkoff <[email protected]> Co-authored-by: Yash Tibrewal <[email protected]> * Bump v1.38.x to v1.38.0 (#26317) * Bump version to v1.38.0 * Regenerate projects * Provide custom sponge configs to exclude manual runs from testgrid (#26283) (#26313) * Provide custom sponge configs * Correct the script * Allow test scripts to produce artifacts through Docker * Overcome the direcotry clone in our docker script * Expose TESTGRID_EXCLUDE and GIT_ORIGIN_URL through docker run * Make sanity tests happy * Revert "Provide custom sponge configs to exclude manual runs from testgrid (#26283) (#26313)" (#26324) This reverts commit 54dc182. * better way of building protoc aarch64 artifacts (#26423) * Backport #26430 and #26435 to v1.38.x (#26436) * workaround #26279 at the expense of binary compatibility * correctly tag aarch64 wheels as manylinux_2_24 * remove unnecessary comments left by #26430 * Backport #22964 into v1.38.x (#26448) * upgrade dotnet SDK 3 for C# build * add C# global.json * install dotnet SDK 3 on macos * upgrade C# docker images to debian buster * regenerate dockerfiles * Backport fixes from #26465 to v1.38.x (#26466) * switch remaining tests to C# buster image * remove C#'s global.json (which requires dotnet SDK 3+) * dotnet SDK3 currently breaks C# package build * regenerate dockerfiles Co-authored-by: Yash Tibrewal <[email protected]> Co-authored-by: AJ Heller <[email protected]> Co-authored-by: donnadionne <[email protected]> Co-authored-by: Mark D. Roth <[email protected]> Co-authored-by: Lidi Zheng <[email protected]> Co-authored-by: Eric Gribkoff <[email protected]> Co-authored-by: Jan Tattermusch <[email protected]>

jtattermusch added 2 commits June 4, 2021 11:15

workaround grpc#26279 at the expense of binary compatibility

fb6d47f

correctly tag aarch64 wheels as manylinux_2_24

96ab6f4

jtattermusch requested review from apolcyn and nicolasnoble as code owners June 4, 2021 11:30

jtattermusch assigned lidizheng Jun 4, 2021

jtattermusch requested review from gnossen and lidizheng June 4, 2021 11:31

jtattermusch added release notes: yes Indicates if PR needs to be in release notes lang/Python labels Jun 4, 2021

jtattermusch added this to the ARM64 support milestone Jun 4, 2021

jtattermusch mentioned this pull request Jun 4, 2021

free(): invalid pointer crash on specific import order (on ARM64 linux) #26279

Closed

gnossen approved these changes Jun 4, 2021

View reviewed changes

lidizheng approved these changes Jun 4, 2021

View reviewed changes

jtattermusch merged commit 4fb0cce into grpc:master Jun 7, 2021

jtattermusch added a commit to jtattermusch/grpc that referenced this pull request Jun 7, 2021

remove unnecessary comments left by grpc#26430

770d4b6

jtattermusch mentioned this pull request Jun 7, 2021

Remove unnecessary comments left by #26430 #26435

Merged

jtattermusch added a commit to jtattermusch/grpc that referenced this pull request Jun 7, 2021

remove unnecessary comments left by grpc#26430

2888af0

jtattermusch mentioned this pull request Jun 7, 2021

Backport #26430 and #26435 to v1.38.x #26436

Merged

jtattermusch added a commit that referenced this pull request Jun 7, 2021

remove unnecessary comments left by #26430 (#26435)

3379d99

jtattermusch added a commit that referenced this pull request Jun 7, 2021

Backport #26430 and #26435 to v1.38.x (#26436)

f5b8b5e

* workaround #26279 at the expense of binary compatibility * correctly tag aarch64 wheels as manylinux_2_24 * remove unnecessary comments left by #26430

jtattermusch mentioned this pull request Sep 10, 2021

Provide ARM64v8 wheel #21283

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Workaround #26279 by publishing manylinux_2_24 wheels instead of manylinux2014 on aarch64#26430

Workaround #26279 by publishing manylinux_2_24 wheels instead of manylinux2014 on aarch64#26430
jtattermusch merged 2 commits intogrpc:masterfrom
jtattermusch:workaround_26279

jtattermusch commented Jun 4, 2021

Uh oh!

jtattermusch commented Jun 4, 2021

Uh oh!

gnossen left a comment

Uh oh!

lidizheng left a comment

Uh oh!

lidizheng Jun 4, 2021

Uh oh!

jtattermusch Jun 7, 2021

Uh oh!

jtattermusch commented Jun 7, 2021

Uh oh!

jtattermusch commented Jun 7, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jtattermusch commented Jun 4, 2021

Uh oh!

jtattermusch commented Jun 4, 2021

Uh oh!

gnossen left a comment

Choose a reason for hiding this comment

Uh oh!

lidizheng left a comment

Choose a reason for hiding this comment

Uh oh!

lidizheng Jun 4, 2021

Choose a reason for hiding this comment

Uh oh!

jtattermusch Jun 7, 2021

Choose a reason for hiding this comment

Uh oh!

jtattermusch commented Jun 7, 2021

Uh oh!

jtattermusch commented Jun 7, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants