TST: a failing table test (test_info_preserved_pickle_copy_init) on CPython 3.14

This test checks conservation of column attributes through various copy-like operators, at least one of which is currently failing.

reprod (requires Python 3.14.0b1 + numpy 1.23dev)

pytest astropy/table/tests -k test_info_preserved_pickle_copy_init

(trimmed) output

...

 >                   assert getattr(m2.info, attr) == original
E                   AssertionError: assert dtype('>i4') == dtype('int32')
E                    +  where dtype('>i4') = getattr(name = m\ndtype = int32\nformat = {0}\ndescription = d\nclass = ArrayWrapper\nn_bad = 0\nlength = 4\n, 'dtype')
E                    +    where name = m\ndtype = int32\nformat = {0}\ndescription = d\nclass = ArrayWrapper\nn_bad = 0\nlength = 4\n = <ArrayWrapper name='m' data=[0 1 2 3]>.info

I'm not sure exactly which operation is affected yet as I'm reporting all failures at once, but I'll dig into this one in time.
From afar, it seems likely that either nothing is actually broken (dtypes under comparison are actually equivalent), or that the bug is upstream in numpy. Either way, more digging needed.

I think I can safely reject the "no bug" scenario: >i4 is big-endian aligned and the machine I'm running on is using little endianness, so dtype('>i4') and dtype('int32') should indeed compare as different.
Furthermore, the exception is raised for the column data labeled as 'arrayswap' (that has class astropy.table.table_helpers.ArrayWrapper and has flipped endianness as compared with whichever is native), and with the init_from_class copy-operator.

@dhomeier

@dhomeier might be able to help. Do you need help investigating? I see a "needs clarification" label but not sure what it means in the context of this issue. Thanks for looking into this!

We need to clarify wether the defect is on our side or in numpy (or even CPython). I could indeed use a hand if this is obvious to someone else, but otherwise I can also investigate myself.

I guess it's actually a feature, not a defect ;-)
More precisely, it seems pickle_roundtrip is now also preserving non-native byteorder, which can be considered an improvement. This seems to be due to the default pickle protocol having been upgraded to 5 – I could not get the details out of PEP 574, but apparently in that version byteorder is now preserved. So we could restore the previous test behaviour by pinning to pickle.dumps(c, protocol=4), but rather better probably, could set protocol=5 for all Python versions and move pickle_roundtrip to the byteorder-preserving functions.

ah ! got it. Indeed, pickle_roundtrip now correctly preserves byteorder, so that's not a bug in itself. What confused me though is that the following patch wasn't enough to pass the test:

diff --git a/astropy/table/tests/test_mixin.py b/astropy/table/tests/test_mixin.py
index be7d66a9ed..83a18753d0 100644
--- a/astropy/table/tests/test_mixin.py
+++ b/astropy/table/tests/test_mixin.py
@@ -527,8 +527,7 @@ def test_info_preserved_pickle_copy_init(mixin_cols):
                 ):
                     original = getattr(m.info, attr)
                 else:
-                    # func does not preserve byteorder, check against (native) type.
-                    original = m.info.dtype.newbyteorder("=")
+                    original = m.info.dtype
                 assert getattr(m2.info, attr) == original

and the reason is that the next iteration of the loop fails, because ArrayWrapper doesn't roundtrips correctly, as is evident from the following, reduced example:

from astropy.table.table_helpers import ArrayWrapper
import numpy as np

arr = np.array([1,2,3], dtype=">i4")
aw1 = ArrayWrapper(arr)
aw2 = ArrayWrapper(aw1)

assert aw1.dtype == arr.dtype
assert aw2.dtype == arr.dtype # fails !

so I guess it's a good opportunity to try and fix that too if we can. The following patch achieves this:

diff --git a/astropy/table/table_helpers.py b/astropy/table/table_helpers.py
index e3c8a96805..bf61191359 100644
--- a/astropy/table/table_helpers.py
+++ b/astropy/table/table_helpers.py
@@ -132,7 +132,12 @@ class ArrayWrapper:
     info = ArrayWrapperInfo()
 
     def __init__(self, data, copy=True):
-        self.data = np.array(data, copy=copy)
+        if isinstance(data, ArrayWrapper):
+            # this is done to preserve byteorder through copies
+            arr = data.data
+        else:
+            arr = data
+        self.data = np.array(arr, copy=copy)
         if "info" in getattr(data, "__dict__", ()):
             self.info = data.info

With this, I think I have every pieces of the puzzle. Will open a PR shortly. Thanks for your help, Derek !

Navigation Menu

Repository navigation

TST: a failing `table` test (`test_info_preserved_pickle_copy_init`) on CPython 3.14 #18127

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

TST: a failing table test (test_info_preserved_pickle_copy_init) on CPython 3.14 #18127

Description

Activity

neutrinoceros commented on May 12, 2025

pllim commented on May 12, 2025

neutrinoceros commented on May 12, 2025

dhomeier commented on May 13, 2025

neutrinoceros commented on May 13, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

TST: a failing `table` test (`test_info_preserved_pickle_copy_init`) on CPython 3.14 #18127