Skip to content

Comments

fix binary wheel tag determination#227

Closed
jrandall wants to merge 9 commits intopython-poetry:mainfrom
genomics-dev:fix-binary-wheel-tag
Closed

fix binary wheel tag determination#227
jrandall wants to merge 9 commits intopython-poetry:mainfrom
genomics-dev:fix-binary-wheel-tag

Conversation

@jrandall
Copy link
Contributor

@jrandall jrandall commented Nov 4, 2021

Resolves: poetry#3509 (python-poetry/poetry#3509)

This change fixes a bug with poetry-core when building a binary wheel (actually any wheel that uses a build script). The previous behaviour was that the wheel tag was determined according to the first entry in the list of compatible tags returned from the python interpreter that is running poetry. However, that is in some cases not the version of python being used to actually build the wheel and so the tag can often be wrong.

In addition, some projects will have outside information as to which of the compatible tags is most appropriate to use (such as selecting a PEP-513 style manylinux1 based tag rather than the default most-specific tag, which tends to currently be one of the new PEP-600 style manylinux_${GLIBCMAJOR}_${GLIBCMINOR}_${ARCH} tags which are only supported by the very latest versions of package managers. For this reason, this change introduces a new build-config option that allows one to specify a filter (regex) in the config and use it to select the first matching tag. This is backwards compatible with previous behaviour (selecting the first listed tag) when no regex is specified.

  • Added tests for changed code.
  • Updated documentation for changed code.

Tests have not been updated to exercise the as I am not certain how to extend the text environment to support testing a binary wheel build, but the existing tests cover the basic build script functionality with the default and those are still passing.

The poetry-schema.json has been updated to add the new wheel-tag-regex field, and that includes a description field which forms some (very basic) documentation on this. It would probably be appropriate to add documentation to poetry itself regarding this change once it is updated to use a version of poetry-core containing the new functionality, although it does not appear that there is currently any user-facing documentation of the [tool.poetry.build] configuration in the poetry docs.

…atible tags using wheel-tag-regex config option
@leotrs
Copy link

leotrs commented Dec 18, 2022

Would love to see this merged soon :)

@jrandall
Copy link
Contributor Author

jrandall commented Feb 8, 2023

I have updated this to the latest main branch. Unfortunately I am somewhat at a loss as to how to update the tests to work.

The issue as far as I can tell is that although these changes appear to work perfectly well when combined with poetry (note that the downstream poetry test suite is running successfully: https://github.com/python-poetry/poetry-core/actions/runs/4127360352/jobs/7130470607), the need to have "packaging" as a dev dependency when using a build script (so that we can get the list of sys_tags that are actually present in the build environment) means that we need to run poetry update in order to make the dev dependencies available.

It seems that the integration tests in poetry-core are not working because the poetry update command is not called as part of the integration tests, and it appears that the update functionality is implemented directly in poetry (by invoking pip?) rather than being implemented in poetry-core, and I'm not sure where we would appropriately hook in to get these tests to work.

@jrandall jrandall force-pushed the fix-binary-wheel-tag branch 2 times, most recently from 1e0047a to 5b20bc3 Compare February 8, 2023 23:02
…than requiring it be added to dev-dependencies
@jrandall jrandall force-pushed the fix-binary-wheel-tag branch from 5b20bc3 to 92953d0 Compare February 8, 2023 23:11
@jrandall
Copy link
Contributor Author

jrandall commented Feb 9, 2023

I've now rewritten the code to get the list of sys tags so that it just pip installs packaging into the build python environment before using it to get the list of tags. This seems better overall as it means that there are no breaking changes to pyproject.toml files (with the previous method, a dev dependency on packaging had to be added whenever a build script was specified).

Please could someone review this? We are still actively using the patched version of this code to do our binary wheel builds using poetry and it would be great to get it upstreamed.

@clintonroy
Copy link

I am not the best person to be looking at this, but the project is trying to move to not using pip to install packages. I wonder if it's possible to get access to the isolated build environment another way?

@jrandall
Copy link
Contributor Author

jrandall commented Feb 9, 2023

I am not the best person to be looking at this, but the project is trying to move to not using pip to install packages. I wonder if it's possible to get access to the isolated build environment another way?

Another option would be taking the sys_tags implementation (and all of its prereqs) from the packaging package and refactoring it into a stand-alone python script so we can just pass that directly to the build python for execution without actually installing anything into the environment.

@jrandall jrandall force-pushed the fix-binary-wheel-tag branch from cfe1974 to 6567665 Compare February 9, 2023 02:06
@jrandall
Copy link
Contributor Author

jrandall commented Feb 9, 2023

I've now implemented a new idea after finding that there is a version of the packaging package in the _vendor directory of this repo, and in most cases it should be possible to point the build python to that version of packaging by setting PYTHONPATH when we invoke it.

This is working on linux and macos but is broken for some reason on windows (except for python 3.11 for some reason). Working on understanding the issues on windows now.

@jrandall jrandall force-pushed the fix-binary-wheel-tag branch from d6ee0ec to b1fe2fb Compare February 9, 2023 13:49
… when invoking build python to get sys_tags in an attempt to fix issue on windows
@jrandall jrandall force-pushed the fix-binary-wheel-tag branch from b1fe2fb to c9e2ac9 Compare February 9, 2023 13:52
@sonarqubecloud
Copy link

sonarqubecloud bot commented Feb 9, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@jrandall
Copy link
Contributor Author

jrandall commented Feb 9, 2023

Am now preserving the environment when calling the build python interpreter (including appending our _vendor directory to PYTHONPATH rather than replacing it). This fixes the windows tests, as it seems the windows python installs prior to 3.11 require some setting in PYTHONPATH in order to be able to seed their PRNG.

All tests pass - could a maintainer please let me know if more is required to get this merged? Not sure who the appropriate contact is -- @radoering ?

@radoering
Copy link
Member

I think I understand the reason for this and I'm not opposed so far. However, I want to hear the opinions of other maintainers before deciding if we want to introduce a new field wheel-tag-regex. I put this on my list for one of the next maintainer meetings. (A little heads-up: It might take some weeks until we discuss this.)

Apart from the general decision, I can already give some minor feedback. (It might make sense to wait for the general decision before working on that.)

If I understand correctly, retrieving the tags via a separate subprocess is not required if poetry-core is used as standalone build backend (and is executed in the build environment), but only if poetry-core is used as part of poetry (and is executed in poetry's environment)? If that's correct we should add it as comment in get_tags().

In poetry we already have a code snippet to get the tags: https://github.com/python-poetry/poetry/blob/8558f0cb94c11d5d1f901900d4a6ea8a8167bd35/src/poetry/utils/env.py#L62-L84 I'd prefer this variant because it does not need to change PYTHONPATH.

If we decide to introduce this feature it will require a test.

@radoering
Copy link
Member

I finally had the opportunity to discuss this topic and we are open to this feature. If you are still interested, please consider the remarks from my previous post.

It might make sense to split this in two PRs or at least two commits:

  • fixing wheel tag determination
  • introducing wheel-tag-regex

I can't come up with a sensible test for the first one because it's only a downstream issue (in poetry), but I'd like to have a test for the second one. Probably, it makes most sense to add a dedicated test for the tag property in test_wheel.py and requires mocking Wheel.get_tags().

@rmarquis
Copy link
Contributor

@jrandall Any news on this? I'd love to see your fixes being merged in main.

This was referenced May 19, 2023
@ralbertazzi
Copy link

@radoering I believe this can now be closed?

@radoering
Copy link
Member

I'll close this one since part of it has already been merged via #591 (the relevant part as per the PR title) and the other part is followed up in #592.

@radoering radoering closed this May 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants