ENH: Help with custom alignments for (structured) dtypes

@leofang

It has come up more often recently (also CC @leofang, but I remember another issue that I think is probably closed).

In practice, it is becoming relatively common for users to have custom alignment needs that fit a structured dtype relatively well. I.e. in C++ one would use a:

struct {
    int a;
    struct alignas(8) {
        int4_t b;
        int4_t c;
    }; 
}

because later compute kernels require that alignment. NumPy can handle this type of dtype by using custom offsets=[] when creating the dtype.

The issue is that there are two notions of "alignment" here:

The user desired one needed for a specific compute kernel, that NumPy may or may not be able to (or even need to) ensure.
One used by NumPy (not often for structured dtypes, since we don't do much with them). This is used to decide for example whether we need to copy an array before running a ufunc on it.

So allowing users to just set/change the dtype alignment may currently lead to awkward copies at least.

Solutions: I can see three things that we can do (which are not necessary exclusive)

Add a helper to np.dtypes.<...> to construct a properly aligned structured dtype (or make a dtype properly aligned). (could be part of np.dtype, but a helper seems easier).
Just allow setting the alignment at least of structured dtype. Either we live with the two concepts colliding, or we ask for custom_alignment <= max_alignment.
If the use-cases are mostly custom_alignment <= max_alignment, I actually don't see a problem with this.
We could actually have a second, custom, alignment that can be larger and is enforced only while constructing a new structured dtype (instead of the CPU alignment need).
This would mainly allow nesting and indexing have the custom alignment survive. It would be very targeted to structured dtypes, I think (i.e. other dtypes don't carry a custom alignment).

Some form of helper may make sense either way, I think. All of these seem pretty reasonable to me (2. without custom_alignment <= max_alignment may be a bit "I know what I am doing").

@gmarkall

Thanks, Sebastian! I think the first need

The user desired one needed for a specific compute kernel, that NumPy may or may not be able to (or even need to) ensure.

can be further decomposed into 2 sub-categories:

The user requires the entire structured dtype to have a specific alignment
The user requires one or multiple members of a structured dtype to have specific alignments

In case 2, if the special members are also structured dtypes, then they are basically case 1 since one can build up a structured dtype using one or more structured dtypes (and I assume the alignments of the member structured dtypes can be honored?). So case 2 is really about enforcing a specific alignment on bulit-in dtypes, which I guess is not yet possible? (IIRC, the bulit-in dtypes are by default aligned to their sizeof sizes.)

Solutions: I can see three things that we can do (which are not necessary exclusive)

I like helper functions better but I do not have strong opinions here. Two considerations I had in mind:

the current align= argument in np.dtype() has been misleading for this purpose (it's all about matching the struct layout to the C compiler's behavior), and I do not suggest to further overload it and accept things other than a boolean to meet this need.
Also, the np.dtype() constructor has multiple overloads as per the documentation, which at this point is way too complicated to learn, and worse if we want to add yet another overload to take user-provided alignments.

Hence my slight preference for helpers.

From our perspective we mainly want the following capabilities:

when allocating a NumPy array with a structured dtype of specific alignment, the allocated pointer is aligned
when slicing a 1D array of a structured dtype of specific alignment, the resulting pointer is still aligned if stride=1
the alignments of a structured dtype and its members, if specified, are queryable through a dtype attribute or a helper function, so that downstream projects like Numba can check the alignments and incorporate this information in their typing systems

cc @gmarkall @isVoid @brycelelbach @gevtushenko @tpn @nvlcambier @hawkinsp for vis

So case 2 is really about enforcing a specific alignment on bulit-in dtypes, which I guess is not yet possible?

Well, it is it just comes with the caveats:

NumPy may get confused and make unnecessary copies (because it is sufficiently aligned for NumPy, but not in practice).
If NumPy creates a new dtype for some reason, it may lose the custom alignment.

In practice those are actually miniscule problems. So it is perfectly viable to create a helper that (inside or outside of NumPy):

Always uses align=True
Ensures itemsize fits the custom alignment.
Just sets ->alignment on the dtype via the C-API (must be used on a newly created dtype, but all structured dtypes are new).

From our perspective we mainly want the following capabilities:

Right, the main limitation here is that NumPy doesn't ensure alignment beyond what e.g. malloc would (any new array is max-align-type aligned, and NumPy will do a copy if something is unaligned).

@seberg

So it is perfectly viable to create a helper that (inside or outside of NumPy):

Always uses align=True

Ensures itemsize fits the custom alignment.

Just sets ->alignment on the dtype via the C-API (must be used on a newly created dtype, but all structured dtypes are new).

Interesting, I was unaware that this is feasible without changes in NumPy. @seberg could you elaborate on which C APIs to use? Perhaps we could build a prototype in Numba-cuda to test this out (and address critical needs of all Numba-cuda downstreams), and once verified we can upstream the helper to NumPy?

@seberg

According to posix_memalign's manual page, malloc will only ever return addresses with 8-byte alignment.

So, @seberg, regarding this comment:

Right, the main limitation here is that NumPy doesn't ensure alignment beyond what e.g. malloc would (any new array is max-align-type aligned, and NumPy will do a copy if something is unaligned).

Am I to interpret that as meaning there's no support for getting address alignment greater than 8? (Would it be possible to dispatch to posix_memalign() instead of malloc() if the requested alignment exceeds 8?)

@tpn

@tpn you would have to set a custom aligned allocator. (Right now that has no access to alignment requests of the dtype.)

EDIT: Not that we couldn't expand NumPy, but there are some slightly less clear internal places (although you probably don't care about short-lived buffers).

Sorry, I should give a working example of what I mean with the forcing the alignment.

This will fail to compile with NumPy 1.x, but should be NumPy 1.x compatible:

Cython code to mutate alignment/itemsize

Note, as noted in-line. Be aware of the notes in the "docs". Also note that this may be missing one check to ensure we never mutate an existing structured dtype (passed in by the user).

cimport numpy as cnp

import numpy as np

cdef extern from "numpy/arrayobject.h":
    # Dance to be NumPy 1.x compatible.  Nobody should do this so
    # we didn't include a helper in NumPy itself...
    int NPY_2_0_API_VERSION
    int PyArray_RUNTIME_VERSION

    void PyDataType_SET_ELSIZE(cnp.dtype, cnp.npy_intp itemsize)

    ctypedef struct _PyArray_DescrNumPy2:
        cnp.npy_intp alignment

    ctypedef struct PyArray_DescrProto:
        int alignment


def make_aligned_struct_dtype(*args, int alignment=1, **kwargs):
    """Create a custom aligned NumPy structured dtype.

    .. note::
        An alignment larger than the maximum `malloc` alignment
        may cause NumPy to do unnecessary copies as even freshly
        allocated arrays may have `arr.flags.aligned == False`.
        (NumPy internals never guarantee such aligned access.)

    .. note::
        When nesting an aligned structured dtype, always use
        `align=True`.

    .. note::
        NumPy promotion (e.g. in concatenate) may "canonicalize" the dtype
        and drop custom alignment, so you may have to explicitly use ``dtype=``
        in such cases.

    Parameters
    ----------
    Same as np.dtype, but additionally `alignment=`
    """
    cdef cnp.dtype dtype = np.dtype(*args, **kwargs, align=True)

    if alignment <= 0 or alignment.bit_count() != 1:
        raise ValueError("Reasonable alignments must be >0 and a power of 2.")

    if dtype.kind != b"V":
        # TODO: If args[0] is already structured we might not return
        # a copy here.  In that case mutating would be wrong!
        raise TypeError("Created dtype was not strutured!")

    # Adjust itemsize and alignment to work with custom one.
    cdef int itemsize = dtype.itemsize
    if itemsize % alignment != 0:
        # Increase itemsize to next multiple of alignment:
        itemsize += alignment - itemsize % alignment

    PyDataType_SET_ELSIZE(dtype, itemsize)  # could also create a new one
    if PyArray_RUNTIME_VERSION >= NPY_2_0_API_VERSION:
        (<_PyArray_DescrNumPy2 *>dtype).alignment = alignment
    else:
        (<PyArray_DescrProto *>dtype).alignment = alignment

    return dtype

@seberg

Thanks, @seberg! It seems to work fine once I remove align=True from this line

    cdef cnp.dtype dtype = np.dtype(*args, **kwargs, align=True)

to avoid the error TypeError: function() got multiple values for keyword argument 'align'.

Compiling your code with

C_INCLUDE_PATH=$(python -c 'import numpy; print(numpy.get_include())') cythonize -i -3 test_numpy_alignment.pyx

and run it, I see a custom, overall alignment can be enforced:

>>> import numpy as np
>>> import test_numpy_alignment
>>>
>>> complex128 = np.dtype([('x', np.float64), ('y', np.float64)], align=True)
>>> complex128
dtype([('x', '<f8'), ('y', '<f8')], align=True)
>>> complex128.alignment
8
>>> complex128_aligned = test_numpy_alignment.make_aligned_struct_dtype([('x', np.float64), ('y', np.float64)], alignment=16, align=True)
>>> complex128_aligned
dtype([('x', '<f8'), ('y', '<f8')], align=True)
>>> complex128_aligned.alignment
16

Ah, thought I imply align=True, but probably not quite right. Internal alignment is different anyway, and to get that to what you want, you still need to expand this to pass offsets= manually (overriding internal alignment for non-structured dtypes may work, but must take a lot of care to not mutate the singletons).

Sorry, didn't mean to close this, although I am not sure there we need to do much (this needs to be figured out with additional needs. CuPy now has a helper for structured dtypes specifically.

What is the status of upstreaming the capability being using in cupy to allow for over aligned allocations?

@dbs4261

CuPy now has a cp.make_aligned_dtype() which helps with this type of thing. It's a bit different from the above hack as it doesn't actually modify the dtype alignment (as it is a CPU hint for NumPy). I am not sure if we should put it into NumPy (or how).

Were you looking for more @dbs4261?

I don't think so. Modifying the item size works for setting the stride between items. But I don't see anything in these approaches that addresses if the allocation is actually aligned. Specifically I am wondering how the allocation is handled when the alignment is greater than max_align_t.

Specifically I am wondering how the allocation is handled when the alignment is greater than max_align_t.

NumPy doesn't guarantee that right now. There is simply no mechanism to do this except using a custom aligned allocator (which does exist, I have one on a CuPy branch also that is probably going to get merged).

I am not sure if there is one readily available on PyPI to install, although it's not hard to create one.

Maybe we could add such an allocator as an option to NumPy itself (if there is a clear use-case). I am not sure about .
The thing about "alignment" is that it is a NumPy information, does NumPy need it to work with the dtype or not, and internally, NumPy doesn't bother to guarantee more than max_align_t/malloc alignment; so NumPy doesn't need it.

(And a decent number of times you may want an alignment that is larger than the itemsize, that seems almost a given if you are asking for more than max_align_t. This would be problematic for NumPy, because arr[1:] would be considered unaligned and unnecessarily copied around. There is a real thing that what is the right alignment is really tied to the use-case.)

Ok, correct me if I’ve misunderstood, but I think you’re telling me this is a two pronged problem. First there is no way to set a custom alignment for a structured dtype (without direct use of the CAPI). And second, there is no mechanism to guarantee that an allocation will be aligned more than max_align_t even if that’s what the dtype requested. And though an allocator capable of greater alignment could be swapped in, because numpy does not currently use one, there is no reason to solve the first problem.

For context, I’m looking to generate buffers for rendering with Intel’s Embree. A lot of structures like ray payloads must be 16 byte aligned. And buffers, particularly vertex buffers, benefit greatly from 16 byte alignment. This is all to allow for better vectorization.

But that raises the question, if numpy arrays can only be aligned to the alignment of max_align_t (generally 8 byte alignment), how does numpy handle larger vectorization chunks?

Also I don’t understand your final comment in parentheses. Over aligning the structured dtype would mean that every instance of that structure would start at the correct alignment. That’s the whole point, as I understand it, of the itemsize hack laid out in this thread. The only way arr[1:] would not be aligned would be if the start of the allocation and arr[0] was also not aligned. Which is why I asked if numpy could create allocations with larger alignments.

Right, roughly, with one additional awkwardness:

You have your "allocation alignment".
You have the internally "required alignment" (which is what NumPy would choose with align=True, not your larger one). NumPy operations won't actually need your requested allocation alignment.
This is not a huge deal but a bit of a dissonance (at least for structured dtypes).
NumPy allocations ignore this entirely. Array allocations could be easy to fix, but I have real doubts you could fix it everywhere. It might be that this can be partially resolved by actually distinguishing allocation and required alignment and further refactors internally (which would be nice anyway maybe). (I.e. we'd have to figure out if we can always guarantee a high alignment, but if the two are distinct concepts that may be OK, and maybe NumPy could still indicate that a high alignment is there.)

Either way all a bit more complicated to the point that it needs some experiments and possibly a NEP.
Which is why I am pointing you towards an allocator and I guess we could consider integrating that into NumPy with np.aligned_alloc():

max_align_t (generally 8 byte alignment), how does numpy handle larger vectorization chunks?

The vectorized loops deal with loads. We don't only do so for basic types, it's not like we use this for structured dtypes.

The only way arr[1:] would not be aligned would be if the start of the allocation and arr[0] was also not aligned

True for structs where alignment <= itemsize, I was thinking of a float32 array where you want arr[0] to have higher alignment.

Ok, correct me if I’ve misunderstood, but I think you’re telling me this

is a two pronged problem. You have understood correctly I believe.

…

On Wed, Sep 16, 2026 at 7:49 AM Daniel Simon ***@***.***> wrote: *dbs4261* left a comment (numpy/numpy#28318) <#28318 (comment)> Ok, correct me if I’ve misunderstood, but I think you’re telling me this is a two pronged problem. First there is no way to set a custom alignment for a structured dtype (without direct use of the CAPI). And second, there is no mechanism to guarantee that an allocation will be aligned more than max_align_t even if that’s what the dtype requested. And though an allocator capable of greater alignment could be swapped in, because numpy does not currently use one, there is no reason to solve the first problem. For context, I’m looking to generate buffers for rendering with Intel’s Embree. A lot of structures like ray payloads must be 16 byte aligned. And buffers, particularly vertex buffers, benefit greatly from 16 byte alignment. This is all to allow for better vectorization. But that raises the question, if numpy arrays can only be aligned to the alignment of max_align_t (generally 8 byte alignment), how does numpy handle larger vectorization chunks? Also I don’t understand your final comment in parentheses. Over aligning the structured dtype would mean that every instance of that structure would start at the correct alignment. That’s the whole point, as I understand it, of the itemsize hack laid out in this thread. The only way arr[1:] would not be aligned would be if the start of the allocation and arr[0] was also not aligned. Which is why I asked if numpy could create allocations with larger alignments. — Reply to this email directly, view it on GitHub <#28318?email_source=notifications&email_token=AADBG4WMZVGOBZ76FZ2SP4D5PJ45ZA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNRZGY4TIMRTG432M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5696942377>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADBG4XQIN7HZXER765I2AD5PJ45ZAVCNFSNUABCKJSXA33TNF2G64TZHM4TAOBWGA3TWSLTON2WKOZSHA2DINRRGA2TQONBOYBA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Bryce Adelstein Lelbach Principal Architect @ NVIDIA --

I remember that PyTorch also has challenges here. I wonder if this is something that should be addressed in the context of the Array API spec.

…

On Wed, Sep 16, 2026 at 10:53 AM Sebastian Berg ***@***.***> wrote: *seberg* left a comment (numpy/numpy#28318) <#28318 (comment)> Right, roughly, with one additional awkwardness: 1. You have your "allocation alignment". 2. You have the internally "required alignment" (which is what NumPy would choose with align=True, not your larger one). NumPy operations won't actually need your requested allocation alignment. This is not a huge deal but a bit of a dissonance (at least for structured dtypes). 3. NumPy allocations ignore this entirely. Array allocations could be easy to fix, but I have real doubts you could fix it *everywhere*. It might be that this can be partially resolved by actually distinguishing allocation and required alignment and further refactors internally (which would be nice anyway maybe). (I.e. we'd have to figure out if we can always *guarantee* a high alignment, but if the two are distinct concepts that may be OK, and maybe NumPy could still indicate that a high alignment is there.) Either way all a bit more complicated to the point that it needs some experiments and possibly a NEP. Which is why I am pointing you towards an allocator and I guess we could consider integrating that into NumPy with np.aligned_alloc(): max_align_t (generally 8 byte alignment), how does numpy handle larger vectorization chunks? The vectorized loops deal with loads. We don't only do so for basic types, it's not like we use this for structured dtypes. The only way arr[1:] would not be aligned would be if the start of the allocation and arr[0] was also not aligned True for structs where alignment <= itemsize, I was thinking of a float32 array where you want arr[0] to have higher alignment. — Reply to this email directly, view it on GitHub <#28318?email_source=notifications&email_token=AADBG4S3JUSMLEZFQJU2RHD5PKSNVA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNRZHE2DONRQG4Y2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5699476071>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADBG4XC4WLAJKYGA3BEIML5PKSNVAVCNFSNUABCKJSXA33TNF2G64TZHM4TAOBWGA3TWSLTON2WKOZSHA2DINRRGA2TQONBOYBA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Bryce Adelstein Lelbach Principal Architect @ NVIDIA --

@seberg

@seberg Maybe I am misremembering the standard, I am more familiar with C++ than C after all, but I thought structures get padded out to their alignment if their size is less than that of the alignment. So an array of struct alignas(4) uint8_a4 { std::uint8_t v; }; would pad the structure so that each v starts on a 4 byte boundary.

Yeah, I guess an allocation for which alignment is larger than sizeof(dtype) is maybe just a different matter. It isn't the issue around structs.

NumPy doesn't JIT compile, so it's (ancient) internal machinery won't make use of the alignas(4) as the compiler might (and if it can, I suspect it is best to just check the actual alignment in practice).
But, also: I am probably diverging a bit, this is a bit awkward, but with some choices about what larger than max_align_t alignments actually guarantee in practice, how NumPy works with such structured dtype is a real but probably minor inconsistency in practice.

My thinking is that there should be a way to pass an alignment value when creating a dtype and the allocator should take that into account and throw a warning (error?) if it is unable to ensure an alignment of that size. Using NEP 49 I could provide my own allocator that will over allocate all data blocks. Long term it would be better to have signatures in NEP 49's PyDataMemAllocator that are alignment aware, but that would likely require some serious changes.

Took me a while to come back to this but I understand what you are saying about over aligning the array allocation of just a bunch of floats, and the problems you'd run into with slicing. I would argue that we shouldn't worry about that problem. This could be addressed by using a structured dtype. So there could be a dtype for float32x8 with 32 byte alignment. It would allocate with the correct alignment, and if you wanted to use it as just a bunch of floats, you could astype(np.float32). Slicing the float view would discard the over alignment, but this is fine, because it would still be aligned to its dtype. If you wanted to slice the array and maintain SSE alignment, you'd use the view with the float32x8 dtype. If someone reinterprets an array allocated with a dtype of np.float32 into the float32x8 vector dtype, the alignment issues are on them.

Navigation Menu

Repository navigation

ENH: Help with custom alignments for (structured) dtypes #28318

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ENH: Help with custom alignments for (structured) dtypes #28318

Description

Activity

leofang commented on Feb 22, 2025

seberg commented on Feb 22, 2025

leofang commented on Feb 24, 2025

tpn commented on Feb 24, 2025

seberg commented on Feb 24, 2025

seberg commented on Feb 24, 2025

leofang commented on Feb 25, 2025

seberg commented on Feb 25, 2025

seberg commented on Jun 16, 2026

dbs4261 commented on Sep 15, 2026

seberg commented on Sep 15, 2026

dbs4261 commented on Sep 15, 2026

seberg commented on Sep 16, 2026

dbs4261 commented on Sep 16, 2026

seberg commented on Sep 16, 2026

brycelelbach commented on Sep 16, 2026

brycelelbach commented on Sep 16, 2026

dbs4261 commented on Sep 16, 2026

seberg commented on Sep 16, 2026

dbs4261 commented on Sep 16, 2026

dbs4261 commented on Oct 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions