BUG: Possible undefined behavior for numpy.void.byteswap()

Describe the issue:

Hi everyone,

I am loading data from a binary file. The file has a header and I read it with:

header_type = np.dtype([('time', '>f8'), ('n', '>i4'), ('dims', '>i4'),
                       ('n_gas', '>i4'), ('n_dark', '>i4'), ('n_star', '>i4'), ('pad', '>i4')])
with open('test.tipsy', 'rb') as binary_in:
	header = np.fromfile(header = np.fromfile(binary_in, count=1, dtype=header_type)[0]

Note, how I extract the single element with [0] instead of using the array as is. header has the type numpy.void.

I then try to swap its byte order. This normally works on numpy.ndarray but seems to do something completely random on `numpy.void'

print(header.tobytes().hex())
res = header.byteswap(inplace=False)
print(res.tobytes().hex())

output of several runs

3f947ae147ae147b000400000000000300000000000400000000000000000000
70cc49706c7f0000a0a33c491c5600000200000000000000040000006c7f0000

3f947ae147ae147b000400000000000300000000000400000000000000000000
50fbdfea127f0000a0b330d9e3550000020000000000000004000000127f0000

3f947ae147ae147b000400000000000300000000000400000000000000000000
107eca8d477f0000a05382ffa0550000020000000000000004000000477f0000

As you can see, while the header before byteswap() stays the same it's output seems to be completely random.

I don't really know why this method even exists on numpy.void. I couldn't find it when searching in the docs. Should this work? Or should the method just not be present?

Reproduce the code example:

import numpy as np

header_type = header_type = np.dtype([('time', '>f8'), ('n', '>i4'), ('dims', '>i4'), ('n_gas', '>i4'), ('n_dark', '>i4'), ('n_star', '>i4'), ('pad', '>i4')])
header = np.array([(31.2, 12, 1, 4, 4, 4, 0)], dtype=header_type)[0]
print(f"{type(header) = }")
print(header.tobytes().hex())
header = header.byteswap(inplace=False)
print(header.tobytes().hex())

Error message:

type(header) = <class 'numpy.void'>
403f3333333333330000000c0000000100000004000000040000000400000000
909719af607f0000a043b0e6cf55000002000000000000000085cd3a82fd0100

Runtime information:

1.25.2
3.9.5 (default, Jun 23 2021, 13:33:38)
[GCC 10.2.1 20210110]
WARNING: threadpoolctl not found in system! Install it by pip install threadpoolctl. Once installed, try np.show_runtime again for more detailed build information
[{'numpy_version': '1.25.2',
'python': '3.9.5 (default, Jun 23 2021, 13:33:38) \n[GCC 10.2.1 20210110]',
'uname': uname_result(system='Linux', node='phoenix', release='6.1.0-10-amd64', version='#1 SMP PREEMPT_DYNAMIC Debian 6.1.38-1 (2023-07-14)', machine='x86_64')},
{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
'found': ['SSSE3',
'SSE41',
'POPCNT',
'SSE42',
'AVX',
'F16C',
'FMA3',
'AVX2'],
'not_found': ['AVX512F',
'AVX512CD',
'AVX512_KNL',
'AVX512_KNM',
'AVX512_SKX',
'AVX512_CLX',
'AVX512_CNL',
'AVX512_ICL']}}]

Context for the issue:

It just took me several hours to track down why the header wouldn't be written correctly in the little endian format. Now I can work around it but it would be nice if it just didn't happen.

Yeah my main, or first, point of concern is that should numpy.void types support byteswap()? Given its representation of potentially diverse data types, the question of byte swapping becomes non-trivial.

Please update/repost this with an example that is executable. Otherwise, I will assume you just didn't flush the file writing, but I don't know.

@seberg

Please update/repost this with an example that is executable. Otherwise, I will assume you just didn't flush the file writing, but I don't know.

@seberg What do you mean? I included a minimal example that works in the "Reproduce the code example:" section. I don't need to write to a file there for the error to show.

Yeah my main, or first, point of concern is that should numpy.void types support byteswap()? Given its representation of potentially diverse data types, the question of byte swapping becomes non-trivial.

For an array with a single numpy.void element the operation works and is seemingly well defined. It just swappes the bytes of all contained datatypes. I.e. if the datatype is comprised of a (">f4", "<i8") it just swappes both inner datatypes as documented ("<f4", ">i8"). In this regard it seems to follow the documentation of ndarray.newbyteorder()

Notes

Changes are also made in all fields and sub-arrays of the data type.

Which makes sense since ndarray.newbyteorder().byteswap() is documented to give an array with the same semantics but different byteorder.

Ah, right. To be fair to me: the first part of the issue has a small code-block with at least 2 syntax errors and a distracting, unrelated, file loading.

So to diagnose the problem: If you convert it to an array (e.g. use [0, ...] to index) everything is fine, the problem only occurs with byteswapping the void scalar.

So, yeah, there is probably a bug in the void byteswap, or how it falls back to an array byteswap.

Ah, right. To be fair to me: the first part of the issue has a small code-block with at least 2 syntax errors and a distracting, unrelated, file loading.

Yeah sorry for that. I thought giving more context would be better but I can see how that can distract from the correct error reproduction.

Ah, right.
So to diagnose the problem: If you convert it to an array (e.g. use [0, ...] to index) everything is fine, the problem only occurs with byteswapping the void scalar.

So, yeah, there is probably a bug in the void byteswap, or how it falls back to an array byteswap.

Yes exactly.

@seberg

@seberg should I try to come up with a fix?

Sure, if you have some time and want to dive into the code! It will be somewhat obfuscated in C code, I think in scalars.c.csr. It might be "gentype" or void specifically.

Thanks for the hint. I'll try and get back with a PR if I succeed or a message here if I fail. If I haven't reacted to it by Friday you can consider it as failed^^.

@SebastianJL

Yeah @SebastianJL if you post developments on here, I am more than happy to lend a hand with it too.

I'm trying to find my way around the project.

// In arrayscalars.h

typedef struct {
        PyObject_VAR_HEAD
        char *obval;
        PyArray_Descr *descr;
        int flags;
        PyObject *base;
    #if NPY_FEATURE_VERSION >= NPY_1_20_API_VERSION
        void *_buffer_info;  /* private buffer info, tagged to allow warning */
    #endif
} PyVoidScalarObject;

Is this the definition of numpy.void in the c layer?

Yes and right its arrayscalars.c.src. You don't need the struct layout though, the important thing is the type definition and the methods defined on them, which is in the C-file and has little to do with the actual object struct.

Yes I was just sanity checking if I was on the right track.
I set up the project in PyCharm so it can't index the c code. So I've been limited to string search. I might switch to Clion because I think it should be able index c and python in the same project.

Guys I'm sorry but I need some help.
I setup a conda env with

conda env create -f environment.yml

and activated it with

conda activate numpy-dev

I am able to build numpy with

spin build

But no matter what I change in the numpy source code (e.g. a random print statement), I cannot seem to make it appear when I import and run numpy functions.
Does spin build install numpy in the numpy-dev environment? If not how do I access the freshly built version of numpy?

For example I added a print statement here

# numpy/numpy/ma/core.py

def array(data, dtype=None, copy=False, order=None,
          mask=nomask, fill_value=None, keep_mask=True,
          hard_mask=False, shrink=True, subok=True, ndmin=0):
    """
    Shortcut to MaskedArray.

    The options are in a different order for convenience and backwards
    compatibility.

    """
    print("helloooo")
    return MaskedArray(data, mask=mask, dtype=dtype, copy=copy,
                       subok=subok, keep_mask=keep_mask,
                       hard_mask=hard_mask, fill_value=fill_value,
                       ndmin=ndmin, shrink=shrink, order=order)
array.__doc__ = masked_array.__doc__

and tried to run a python script from within the environment

import numpy as np

a = np.array([1,2,3])

I expected

helloooo

to be printed.

I just realized, that that's not the np.array() function but np.ma.core.array(). Still doesn't work, but I found

spin ipython

That drops me into an ipython sesssion that has the locally built numpy installed.

Is there another way to do this?
I'd like to write some scratch file for testing before I write an actual test in numpy.

Ok, another update.
I was able to do it with

pip install -e . --no-build-isolation

which apparently installs it in the conda env if the env is activated. I did not know that conda allows for pip installs.

Yes and right its arrayscalars.c.src. You don't need the struct layout though, the important thing is the type definition and the methods defined on them, which is in the C-file and has little to do with the actual object struct.

There doesn't seem to be a arrayscalars.c.src. Also greping for PyVoidScalarType, which I expected to be the type definition based on this documentation didn't give any results.

Ok, I'm a bit further now. Important stuff, e.g. the byteswap() implementation seems to be in scalartypes.c.src

Question: Does it make sense that numpy.float64 can be byteswapped inplace and numpy.void can't with the error message:

ValueError: cannot byteswap a scalar in-place

No it doesn't make sense that np.float64 can be byteswapped inplace (if it can). Voids scalars are sometimes mutable, but they shouldn't be, so that makes sense.

I ran into this from a completely different direction but found this while searching open issues. I opened #32254 which should fix this.

On my PR:

In [1]: import numpy as np
   ...:
   ...: header_type = header_type = np.dtype([('time', '>f8'), ('n', '>i4'), ('dims', '>i4'), ('n_gas', '>i4'), ('n_dark', '>i4'), ('n_star', '>i4'), ('pad', '>i4')])
   ...: header = np.array([(31.2, 12, 1, 4, 4, 4, 0)], dtype=header_type)[0]
   ...: print(f"{type(header) = }")
   ...: print(header.tobytes().hex())
   ...: header = header.byteswap(inplace=False)
   ...: print(header.tobytes().hex())
type(header) = <class 'numpy.void'>
403f3333333333330000000c0000000100000004000000040000000400000000
3333333333333f400c0000000100000004000000040000000400000000000000

Navigation Menu

Repository navigation

BUG: Possible undefined behavior for numpy.void.byteswap() #24694

Describe the issue:

Reproduce the code example:

Error message:

Runtime information:

Context for the issue:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

BUG: Possible undefined behavior for numpy.void.byteswap() #24694

Description

Describe the issue:

Reproduce the code example:

Error message:

Runtime information:

Context for the issue:

Activity

jacobmcasey commented on Sep 15, 2023

seberg commented on Sep 18, 2023

SebastianJL commented on Sep 19, 2023

SebastianJL commented on Sep 19, 2023

seberg commented on Sep 19, 2023

SebastianJL commented on Sep 19, 2023

SebastianJL commented on Sep 19, 2023

SebastianJL commented on Sep 19, 2023

seberg commented on Sep 19, 2023

SebastianJL commented on Sep 19, 2023

jacobmcasey commented on Sep 19, 2023

SebastianJL commented on Sep 19, 2023

seberg commented on Sep 20, 2023

SebastianJL commented on Sep 20, 2023

SebastianJL commented on Sep 20, 2023

SebastianJL commented on Sep 20, 2023

SebastianJL commented on Sep 20, 2023

SebastianJL commented on Sep 20, 2023

SebastianJL commented on Sep 20, 2023

SebastianJL commented on Sep 20, 2023

seberg commented on Sep 21, 2023

ngoldbaum commented on Aug 11, 2026

ngoldbaum commented on Aug 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions