DISCUSS,BUILD: Change built setup to default to a backwards compatible build

Yesterday I proposed a plan to evolve our C-API (where we can/want). I believe that needs to be a two step separate things:

Change how we build backwards compatible NumPy versions:
- Introduce NUMPY_TARGET_VERSION=0x01013000 // NumPy 1.19, when compiling, by default any newer API is hidden behind #if guards.
- If a user wants newer API, they use:
```
/* require NumPy 1.22 for mem-handler API: */
NUMPY_TARGET_VERSION=0x01016000
```
- We "take care of the rest". I.e. checking versions on import.
- oldest-support-numpy would always be happy with >1.25.
- Cython: In this context .pyd files are bit special. In most cases it is OK for them to include definitions that are not actually available (because they too new). There are some exceptions, though. The one I am aware of is that new Python classes must live in a new .pyd file because otherwise Cython would generate backwards incompatible initialization code.
Point 1 means that users can always use the newest headers. For a 2.0 release we could enforce a rebuilt of NumPy so that we know they are using 2.0+ headers. That gives us a huge amount of flexibility in what we could do. (Of course many changes themselves will impact users).
- This point, I suspect needs to be fleshed out a little bit as a document anyway (if just to have something to point to and more formally agree on).

I don't want to discuss 2. here (we can open separate issue, yesterday it sounded like we will give forcing both a rebuild and some code changes a shot).

What I will push now is point 1. Some points for feedback, if we are unsure about it I can flesh this out:

Let's sync up the NumPy C-API feature version and the actual NumPy version?
- Yes, one includes the major version and the other one does not. Our C-ABI version would be left untouched. As soon as NumPy 2.0 gets released the C-ABI version would be identical to NumPyVersion & 0xFF000000 (right now it is 0x01000009)
- We'll bump it anyway in 1.25.
Users of memory handler API will have to add the define. This seems so niche and easy to do that I think it is firmly "no worry".
Are there other things we need to consider? (I.e. testing? For that I suspect it isn't a big issue and downstream wheels will be runtime tested against older NumPy versions.)

Any other bike-sheds, like the name of the define? :)

This is just for potentially longer discussion. I hope to open a PR soon (step 1 is really quite simple).

@rgommers

I am not 100% that going to 1.19 is the best, so it may be good sport to go back a bit further. oldest-support-numpy is currently on 1.17, defaulting to that seems fine. The previous relevant changes were at 1.14 (probably, some irrelevant struct additions maybe further).

There is no hurt in just guarding things further back. But 1.17 seems OK as a "default".

There some actions that need to occur in distribution systems:

oldest-support-numpy doesn't work in its current state. The solution is likely to use:
- numpy >=1.17.5,!=1.18,!=1.19,...,<2 as a version pin. This is ugly, but would mean you fetch a 1.25 release when available (and compatible).
Conda has pin_compatible() which may need similar adaption. Ping @rgommers.

Since the old method would keep working fine for 1.25.x, we should have a good amount of time to adapt these things where necessary.

I changed my mind about changing versioning, because we have the definitions:

NPY_1_21_API_VERSION ...
NPY_1_22_API_VERSION ...

anyway already and you need knowledge of the old version numbering anyway (for the _import_numpy macro).

gh-23528 contains the implementation, and I think it is actually complete and very short.

To be clear, I don't have an strong opinion about whether to default to backwards compatible builds. Instead of defaulting to it, we could also try to include the information of how to achieve a backwards compatible build in the error that _import_array() gives.

It seemed like the more useful default to me. Right now, compiling in a back-compat way is free anyway. In NumPy 2.0 it would not always be (you need shims if you also target 1.x), but overall I doubt those shims matter in practice and packaging is the bigger confusion than local builds having a tiny bit of overheads.

Let's sync up the NumPy C-API feature version and the actual NumPy version?

That sounds good to me.

Any other bike-sheds, like the name of the define? :)

Not from me, too many bike sheds around recently:) Pick what you like best I'd say.

I am not 100% that going to 1.19 is the best, so it may be good sport to go back a bit further. oldest-support-numpy is currently on 1.17, defaulting to that seems fine.

That assumes we're staying with Python 3.8 for a bit longer, and you're looking at the lowest version in oldest-supported-numpy for 3.8, right? That seems fine to me. We can drop 3.8 support, and then 1.19.3 would be the lowest version. In practice it doesn't seem to matter though, according to what you wrote. I do want to check that with you - I remember we had a C API change which TensorFlow was quite unhappy with and pinned to <1.19.2. Just want to make sure that that was nothing that may bite us now if we go that far back?

oldest-support-numpy doesn't work in its current state. The solution is likely to use:
numpy >=1.17.5,!=1.18,!=1.19,...,<2 as a version pin. This is ugly, but would mean you fetch a 1.25 release when available (and compatible).

After some more thought, I think this will be okay and this should be the sequence in time here:

Merge your changed "default to old API" PR for 1.25.0
Release 1.25.0
Update oldest-supported-numpy with the numpy >=1.17.5,!=1.18,!=1.19,...,<2 pins
.... do whatever C-level changes we need for 2.0 ....
release 2.0.0rc1
Ask downstream libraries to stop using oldest-supported-numpy, test rebuilding their latest released version against 2.0.0rc1, and publish wheels at some point before 2.0 (might be rc1, or a later rc depending on our confidence level). Their wheels will work with 1.2x.0-1.25.0 (with x TBD) as well as with >=2.0.
release 2.0.0

The tricky bit here is included in (6). I am not sure that "stop using oldest-supported-numpy" will work. numpy itself will have much better default behavior after gh-23528, but that may not capture all that oldest-supported-numpy does. Or maybe it does. I'm thinking of stuff like this:

    # Note that 1.22.0 broke support for using the Python limited C API
    # (https://github.com/numpy/numpy/pull/20818), so we use 1.22.2 instead
    numpy==1.22.2; platform_machine=='loongarch64' and python_version<'3.11'

    # Note that 1.21.3 was the first version with a complete set of 3.10 wheels,
    # however macOS was broken and it's safe to build against 1.21.6 on all platforms (see gh-28 and gh-45)
    numpy==1.21.6; python_version=='3.10' and platform_machine!='loongarch64' and platform_python_implementation != 'PyPy'

In principle I think such cases can be fixed in numpy, and then should be present in the latest release and things are good. A practical issue is that the fix is harder to apply (needs a new numpy bugfix release, a lot of work compared to a tweak in a pure Python metapackage). But in practice that's not a big deal I'd say; these types of things are rare, and those affected can just do their own workarounds as needed.

So what's left is the advice to downstream packages about what to use as their build-time and runtime dependency specifiers. Perhaps:

runtime: numpy>=1.22.4,<2.3 or numpy>=1.22.4,<3.0 (the former for packages adhering to the N+3 recommendation for deprecated API, the latter for everyone else)
build-time: numpy>=2.0,<2.3 (or <3.0 again)

This does mean that the wheel build situation for downstream packages becomes less stable, rather than a pin it depends on yet-to-be-released numpy versions. So potentially an issue that they need to think about - but that's no different from dealing with any other build-time dependency. So we should be all good there I think.

Conda has pin_compatible() which may need similar adaption.

We'll have this in meson-python as well. What it does is translate the build-time numpy version into a lower bound on the runtime version. So if you build against numpy 2.1, the runtime constraint added is numpy>=2.1. That will now actually change - by default no lower bound is needed anymore; the tricky bit here may be knowing when NUMPY_TARGET_VERSION is used and how to translate that into a lower bound.

This may perhaps turn out to be a little annoying. It's no longer possible to know what the right bound is based on static metadata + the build-time version (which is also captured in metadata for conda-forge). Instead, it comes from a define in your source code somewhere. Which means it can't be handled by conda-forge or a build system, and it turns into a manual exercise of ensuring the define in the source code is in sync with pyproject.toml, meta.yaml, etc. for each package. Again not a huge issue, it just needs to be clearly communicated about how to do this in a standard way and that it's mostly the package authors' responsibility. Conda-forge tooling (Grayskull) should also be updated to check for changes in lower bounds probably, so that it can be automatically translated in the PRs to feedstocks.

Marten mentioned NEP 29 as a good choice for the default. I had changed my draft with that for now, but in practice we could go back as far as 1.14 easily.

I suspect it all doesn't matter much. Small cython libs are almost bound to get version pins wrong, and for the most part nobody will notice either way probably.

But we could default to going so far back that only Python limited API users can possibly notice a problem (and those that opt-in to a newer version but forget to keep their metadata in sync).

I remember we had a C API change which TensorFlow was quite unhappy with and pinned to <1.19.2.

I don't recall anything problematic. I recall that 1.19 added a few const qualifiers and Tensorflow imagined it was an ABI break (IIRC it did required a somewhat ugly casts in C++ but it was no break).
Same for PyArray_DescrCheck (also same version if that applied to them), which was a macro fix. In both cases the newer headers are strictly ABI (and probably even API) compatible.

Update oldest-supported-numpy with the numpy >=1.17.5,!=1.18,!=1.19,...,<2 pins

Wouldn't it be even OK to just update it to <3 even (or 2.x)? We would do this when the rc is out that we want users to start compiling for. The goal being that oldest-support-numpy can remain a no-op for the most part.
But, that would allow oldest-support-numpy to transform into the role of blocklisting bad NumPy releases oldest-support-numpy indefinitely (if that remains worthwhile)?

This may perhaps turn out to be a little annoying. It's no longer possible to know what the right bound is based on static metadata + the build-time version

Yeah, packages/maintainers will have to search for the defines and keep that and metadata in sync :/.

Marten mentioned NEP 29 as a good choice for the default. I had changed my draft with that for now, but in practice we could go back as far as 1.14 easily.

Hmm I'm not sure you can even do that, since NEP 29 compatibility is time-based. Making sure it defaults to all versions that a user is able to install for a given Python version seems a lot safer either way, and would allow reaching the goal you articulate below:

Wouldn't it be even OK to just update it to <3 even (or 2.x)? We would do this when the rc is out that we want users to start compiling for. The goal being that oldest-support-numpy can remain a no-op for the most part.

Yes, I think that may be okay, and could be nicer perhaps. I would like to get rid of oldest-supported-numpy though if we don't need it anymore. The metapackage is pretty annoying for packagers of other dependencies (since it doesn't actually exist there), and a little non-obvious even within PyPI walls.

Going to close this. Lets do follow-up discussions in the NEP (gh-23579) or just on a new issue.

I have not changed the versioning scheme yet, but I think it may make sense to just do it for 2.0 (or now if anyone cares to do it).

Navigation Menu

Repository navigation

DISCUSS,BUILD: Change built setup to default to a backwards compatible build #23525

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

DISCUSS,BUILD: Change built setup to default to a backwards compatible build #23525

Description

Activity

seberg commented on Apr 4, 2023

seberg commented on Apr 4, 2023

seberg commented on Apr 5, 2023

rgommers commented on Apr 19, 2023

rgommers commented on Apr 19, 2023

seberg commented on Apr 19, 2023

rgommers commented on Apr 19, 2023

seberg commented on Apr 28, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions