Dealing with opaque API, 32bit builds and Fields Getting

@da-woods

Let's maybe move the discussion here @da-woods, @kumaraditya303, and @ngoldbaum.

Right now to allow opaque API use, we have a family of:

(PyArray|*)_GET_ITEM_DATA static inline function which use,
A family of _* functions that all do the same thing (remove PyObject * offset for the build).
A family of _fields structs that omit PyObject *.

As noticed by Nathan in gh-31759 this doesn't work for 32bit free-threaded builds, the reason is that sizeof(PyObject) < sizeof(max_align_t) meaning that the _fields struct may have a different layout.

There are two approaches plausible:

Forgo _GET_ITEM_DATA completely... everything would have to go through actual API functions.
Rather adding PyObject to our types, we add PyObject_made_multiple_of_max_align_t.

My gut feeling is: Doing 1 would basically reduce the whole current approach to rubbles and make NumPy less convenient than a full opaque API using module (where Python would add that same padding as @da-woods pointed out).

My current take is that we should go with 2 and pad on Python 3.15+, this breaks ABI for free-threaded 32bit builds, but that release doesn't exist yet.
(After thinking, I suspect Cython is actually OK, since it just uses the sizeof() from the header and doesn't care if there is an additional field conditional to the Python version.)

I'll be annoying and actually ask @da-woods once if you think the family of functions is worth it.
Sorry, it still feels a bit unnecessary to me. Yes, it allows inserting private fields at the start which is cute, but only if we ever break the non opaque ABI fully for that same object. (But yeah, I guess I am the only one who feels this way so whatever...)

Idea from Kumar: maybe instead of doing it for all 3.15+ builds, we only do it for abi3t builds? That nicely sidesteps the ABI break for all GIL-enabled builds because the structs are all opaque in the public API on abi3t. I think we just need to make the padding explicit or use struct packing e.g. and then it will all work OK again and no need for a new function?

That nicely sidesteps the ABI break for all GIL-enabled builds because the structs are all opaque in the public API on abi3t

Conceptually, I think we must always pad, but in practice this is already the case: Only for free-threaded and 32bit builds the padding will be != 0 so on all of those build the headers will evaluate to not adding any padding anyway and there is no ABI break anyway.

I'm a little confused, isn't the padding added only on the GIL-enabled build? That's the struct padding math I ended up with in the description of #31762 anyway.

That said, I think I see what you're getting at that making the padding explicit isn't an ABI break.

Let's try and spell out how it might look like (roughly), guess I am thinking this:

struct PyArrayDescr_Object {
    PyObject_HEAD
#if SIZEOF_PYOBJECT_HEAD % ALIGNOF_MAX_ALIGN_T != 0
    char _npy_obj_padding[ALIGNOF_MAX_ALIGN_T - (SIZEOF_PYOBJECT_HEAD % ALIGNOF_MAX_ALIGN_T)];
#endif
    ...
}

Now, I got it wrong: ALIGNOF_MAX_ALIGN_T is often 16. Which I think works out for all non free-threaded Python builds.

ALIGNOF_MAX_ALIGN_T would be the best, but we need to be sure it works out, since our structs never contain anything with more than 8 bytes alignment, so if ALIGNOF_MAX_ALIGN_T > SIZEOF_PYOBJECT_HEAD on any existing stable API build we would get things wrong!

Just trying to think of ways to break this. Does this break abi3 extensions that want to run on GIL-enabled 3.15 or newer? Because at runtime they'll include the padding but at compile time it won't.

Yeah, you are right, I think it is broken, but this leaves us in a pickle where we have to "promise" alignment things about our object structs. Probably fine, but ugly...

I.e. even if I am not aware that ALIGNOF_MAX_ALIGN_T >= 16 exists, on 32bits ALIGNOF_MAX_ALIGN_T == 16 can be true, but SIZEOF_PYOBJECT_HEAD == 8.

And at that point things still work out fine, but we have to replace ALIGNOF_MAX_ALIGN_T with 8 (at least on 32bit platforms) and guarantee that we'll never add a field to any of our object structs that needs a larger alignment.
I somewhat still suspect it is our best bet, but it would be nicer if we could just roll with ALIGNOF_MAX_ALIGN_T :(.

I think the simpler alternative which will not break any existing extensions is to use packed structs on abi3t with alignment of 4.

I have a working implementation of this at kumaraditya303#2 and corresponding https://github.com/kumaraditya303/rust-numpy/tree/abi3t for rust-numpy. The minimum guaranteed alignment of PyObject is 4 so if we use struct packing of 4 then it will always work.

I think the simpler alternative which will not break any existing extensions is to use packed structs on abi3t with alignment of 4.

Ah, interesting though and maybe best. But while I guess that works around rust-numpy does that conceptually actually safe us from changes to SIZEOF_PYOBJECT_HEAD changing the internal padding?!
(Unless we make all our struct packed to 4 on 32bit builds maybe... but even then I don't really think it saves us from pitfalls if SIZEOF_PYOBJECT_HEAD ever changes?)

EDIT: I suppose to me it feels that ideally any change of SIZEOF_PYOBJECT_HEAD will have no effect on the layout of any of our (effective) _field structs. If we bake that into headers, we are good. But that would mean padding to ALIGNOF_MAX_ALIGN_T.

But while I guess that works around rust-numpy does that conceptually actually safe us from changes to SIZEOF_PYOBJECT_HEAD changing the internal padding?!

Yes, even if sizeof(PyObject) changes, the change would still be multiple of 4 because of the alignment so it will still work.

Fair, we can probably assume the multiple of 4 well enough. But if we reduce the alignment of our object structs, then that in itself may change their size/alignment which is something we can only do on free-threaded builds.
So it would have to not do that on GIL builds. But unless we bake in explicit padding, we then still rely on Python GIL builds to never make an PyObject_HEAD with a size that isn't a multiple of 8/16?
I am sure that is true... But if that is the case I am not sure I see any advantage in using packed structs over just padding manually.

I think that means you should go ahead and open a NumPy PR with the struct packing fix and mark it as closing both issues 😀

We can backport that too because it only impacts abi3t builds that people aren't actually using yet.

OK, I'll just wait for Kumar's PR ideally, thanks! I still somewhat think that changing packing alignment may be part of the solution, but isn't the solution, because we have void *, 4x char, int, int64 == 3 * 4 bytes followed by int64 meaning that I think a normal build has 4 padding bytes and changing the packing will remove them? (But of course it is also just as likely that I am just missing what you mean.)

@da-woods

I'll just answer the one thing that I was directly asked

I'll be annoying and actually ask @da-woods once if you think the family of functions is worth it.
Sorry, it still feels a bit unnecessary to me. Yes, it allows inserting private fields at the start which is cute, but only if we ever break the non opaque ABI fully for that same object. (But yeah, I guess I am the only one who feels this way so whatever...)

I think the functions are only really worthwhile if you commit to it fully. I.e. if you didn't have (PyArray|*)_GET_ITEM_DATA but instead made the functions the only way to access the structs when using opaque objects. If that was the case then the alignment wouldn't be an issue because you'd never be able to get the "fields only object".

It's one solution to the problem, but it's a fairly tedious and verbose one.

In a world where (PyArray|*)_GET_ITEM_DATA exists and has to work, then the only real point of the functions is as a convenience. Which I guess might be justified for commonly accessed things.

I think that's broadly in agreement with your summary.

I've setup a test repo at https://github.com/kumaraditya303/np-test for this issue. It create a stable ABI extension compiled against Python 3.12 stable ABI and numpy 2.5 and tests it against kumaraditya303@89596c6 where the alignment the alignment of first field is set to 16 bytes. See https://github.com/kumaraditya303/np-test/actions/runs/28388653853 where it fails on windows x86 but passes on all 64 bit versions. So if any alignment higher than 8 is used, it will break any stable ABI extensions compiled on 32 bit systems.

It passes on #31771 where the alignment is set to 8 bytes which is the correct fix.

OK, I also thought this through a bit more and my basis for things is this:

The current abi3 gives us two limitations:
a) On 64bits, the actual alignment we have is at most 16. (No limitation as I don't think there is any real system where alignof(max_align_t) > 16)
b) On 32bit systems the actual alignment we have is at most 8 (I missed this part at times).
- In theory this is a limitation as alignof(max_align_t) can be larger on such systems. ¹
IMO we cannot make any assumptions about future sizeof(PyObject) and assume we might not notice when it changes:
- This means that whatever alignment we put now is the maximum alignment we can ever use in public fields.
I would prefer to not make any assumptions about alignof(max_align_t).

Kumar and I had talked about this, but didn't really take the same viewpoint. I don't like that rather than just fixing a bug, we potentially make an ABI promise that we might regret in theory.
(This is all pretty theoretical, but I need to know that we didn't just ignore this because it's complicated.)

To some degree, I would love to just use alignof(max_align_t) it feels conceptually cleanest and is the same as PEP 967 as well. But unless someone can argue that there are no realistic abi3 builds on 32bit systems with alignof(max_align_t) > 8 that can't work...
I still am wondering about using 16/max_align_t on 64bit systems though. Do we lose anything doing that?

Of course, if the best anyone can see my worry here is: "hahaha you are crazy but maybe/meh" we can also add a static_assert(offsetof(first_field) % 16) == 0 or so and make it "theoretical future us" problem.

Other small things (first needs fixing, last is just "thought about it, but I guess it doesn't matter"):

As mentioned at the top, I think we have to limit this to Python 3.15+ (for free-threaded builds). We are changing the ABI there and we cannot do that for any Python version for which a NumPy release already exists.
As Kumar told me, Python also uses the C11 _Alignas(). Python does provide an escape hatch for theoretical niche compilers. Should we also add one via #ifndef NPY_DECL_ALIGNED?
By using alignas(8) rather than alignas(min(8, alignof(max_align_t)), we might be making an ABI promise we don't directly control?
- I think this doesn't matter, because NumPy itself would notice any problem before downstream could. (the alignas() is part of our struct, not the opaque API one)

Python could maybe change this if it became a problem, but it seems it may already a lingering ABI problem because Python seems to only guarantee 8 byte alignment for objects, which means that NumPy longdouble scalars might be broken on 32bit RISC V. Which is probably just not a real system to run NumPy in practice... ↩

@seberg

Over in #31771 (comment), @seberg asked me to do one last pass over the public C API surface before merging Kumar's PR.

I'm glad he did because (with some AI assistance) I realized that we missed PyArray_StringDTypeObject in our abi3t support pass for NumPy 2.5 and that struct is completely broken under abi3t. For NumPy 2.5.2 I'm just going to exclude it and any function that uses from the C API surface under abi3t builds. I'll just push that directly to Kumar's PR later today. For NumPy 2.6 I think we'll need a real accessor function in the C API.

For testing, Kumar wrote an ABI checker tool that we'd like to incorporate into the NumPy CI: https://github.com/kumaraditya303/np-test

I also had an AI generate an ABI checker tool for the alignment issue. It also includes an exhaustiveness check and it can detect when a new struct has incorrect alignment: https://github.com/ngoldbaum/numpy-abi-check

Both of these abi checker tools probably need some consolidation and iteration before they can go into the NumPy CI. So for now I'd like to go ahead and merge Kumar's PR with the fix for the StringDType descriptor and leave this issue open to track adding improved ABI checking tests as a followup.

Please let me know if you all have issues with that plan. I'll probably hold off on hitting the merge button on Kumar's PR until the triage meeting tomorrow.

Cool! Sounds like covering a lot of ground (probably more than needed), neat! I haven't looked at the things, I agree there is no need to rush merging them and that it may be nice to do so.

One thing I was a bit worried about is that we just add(ed) #ifndef Py_TARGET_ABI3T thinking that if someone asks we can just expose it to Py_TARGET_ABI3T later.
But with these ABI issues now, that later exposure would become just a tad trickier (i.e. may have to wait a Python release at least for 32bit systems).

Not sure if you want to add that. I am not super worried but I think I might at least once have an agent make me a list of structs with PyObject_HEAD that are not exposed to Py_TARGET_ABI3T to see if any of them look like we might want to add it.

but I think I might at least once have an agent make me a list of structs with PyObject_HEAD that are not exposed to Py_TARGET_ABI3T to see if any of them look like we might want to add it

I did that and couldn't find anything. The only thing it flagged is that PyArrayMethod.legacy_initial being defined as char legacy_initial[sizeof(npy_clongdouble)]; is now load-bearing for alignment. But also I don't see why we would ever change that?

All it recommends are dev docs documenting these limitations, which makes sense to me too.

Ah OK, I guess I am almost tempted that for such structs adding the padding doesn't hurt, but if the ABI checker flags it as "hey, this might not be exposable" then it also doesn't matter, which is nice!

then it also doesn't matter, which is nice!

Although, it does matter a little? PyArrayMethod is private so doesn't matter (even if we expose it we could never expose the object struct as well).

But if the struct was public abi3 API available then it would be nice to list/flag it. The point is that by not using the padded PyObject_HEAD you'd make adding 8 byte aligned fields at the end tricky.

Navigation Menu

Repository navigation

Dealing with opaque API, 32bit builds and Fields Getting #31764

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Dealing with opaque API, 32bit builds and Fields Getting #31764

Description

Activity

ngoldbaum commented on Jun 26, 2026

seberg commented on Jun 26, 2026

ngoldbaum commented on Jun 26, 2026

seberg commented on Jun 26, 2026

ngoldbaum commented on Jun 26, 2026

seberg commented on Jun 26, 2026

kumaraditya303 commented on Jun 26, 2026

seberg commented on Jun 26, 2026

kumaraditya303 commented on Jun 26, 2026

seberg commented on Jun 26, 2026

ngoldbaum commented on Jun 26, 2026

ngoldbaum commented on Jun 26, 2026

seberg commented on Jun 26, 2026

da-woods commented on Jun 26, 2026

kumaraditya303 commented on Jun 30, 2026

seberg commented on Jun 30, 2026

Footnotes

ngoldbaum commented on Jul 7, 2026

seberg commented on Jul 7, 2026

ngoldbaum commented on Jul 7, 2026

seberg commented on Jul 7, 2026

seberg commented on Jul 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions