Repository navigation
Closed
Milestone
Description
It's still very early in the morning and I may simply not be awake yet, but it looks to me like the __array__ changes in gh-25168 are incomplete and don't have a regression test for the old signature:
In [1]: import numpy as np
In [2]: class MyArr:
...: def __init__(self, x):
...: self._x = x
...: def __array__(self, dtype=None):
...: return self._x
...:
In [3]: x = np.arange(3)
In [4]: np.asarray(x)
Out[4]: array([0, 1, 2])
In [5]: np.asarray(MyArr(x))
...
TypeError: MyArr.__array__() got an unexpected keyword argument 'copy'
In [6]: np.__version__
Out[6]: '2.0.0.dev0+git20240301.06b169b'I didn't follow the review of the C code in detail, so I'm probably missing some details, but it doesn't look like the current implementation does what was intended. In particular:
- only pass along the new
copykeyword if it's not the default value ofNone, to have a smaller impact, - pass the actual True/False/None values; it now only passes
FalseorNone(never_copy ? Py_False : Py_None;), - when retrying without the new keyword, the
dtypekeyword should still be passed, so the code should popcopyand retry withkwargsrather than passing NULL.
It looks like the changes should be along these lines (incomplete):
$ git diff
diff --git a/numpy/_core/src/multiarray/ctors.c b/numpy/_core/src/multiarray/ctors.c
index 4b9c3add3f..897e141336 100644
--- a/numpy/_core/src/multiarray/ctors.c
+++ b/numpy/_core/src/multiarray/ctors.c
@@ -2442,26 +2442,46 @@ PyArray_FromArrayAttr_int(
PyObject *copy = never_copy ? Py_False : Py_None;
PyObject *kwargs = PyDict_New();
- PyDict_SetItemString(kwargs, "copy", copy);
+
+ /*
+ * Only if the value of `copy` isn't the default one, we try to pass it
+ * along; for backwards compatibility we then retry if it fails because the
+ * signature of the __array__ method being called does not have `copy`.
+ */
+ int copy_passed = 0;
+ if (copy != Py_None) {
+ /* XXX: why can `copy` only be False or None, not True? */
+ copy_passed = 1;
+ PyDict_SetItemString(kwargs, "copy", copy);
+ }
PyObject *args = descr != NULL ? PyTuple_Pack(1, descr) : PyTuple_New(0);
new = PyObject_Call(array_meth, args, kwargs);
if (PyErr_Occurred()) {
+ PyObject *errmsg_substr = PyUnicode_FromString(
+ "__array__() got an unexpected keyword argument 'copy'");
+ if (errmsg_substr == NULL) {
+ return NULL;
+ }
PyObject *type, *value, *traceback;
PyErr_Fetch(&type, &value, &traceback);
- if (PyUnicode_Check(value) && PyUnicode_CompareWithASCIIString(value,
- "__array__() got an unexpected keyword argument 'copy'") == 0) {
+ if (PyUnicode_Check(value) && PyUnicode_Contains(value, errmsg_substr) > 0) {
Py_DECREF(type);
Py_XDECREF(value);
Py_XDECREF(traceback);
+ Py_DECREF(errmsg_substr);
if (PyErr_WarnEx(PyExc_UserWarning,
"__array__ should implement 'dtype' and 'copy' keywords", 1) < 0) {
return NULL;
}
- Py_SETREF(new, PyObject_Call(array_meth, args, NULL));
+ if (copy_passed) { /* try again */
+ PyDict_DelItemString(kwargs, "copy");
+ Py_SETREF(new, PyObject_Call(array_meth, args, kwargs));
+ }
} else {
PyErr_Restore(type, value, traceback);
+ Py_DECREF(errmsg_substr);
return NULL;
}
}Reacted by Mateusz Sokół
Activity
oscarbenjamin commentedon Mar 3, 2024
This breaks a number of tests in sympy:
At the same time we are also seeing a segfault in CI with the numpy nightly wheels but I don't know if that is related to this (they both seem to happen at the same time):
https://github.com/sympy/sympy/actions/runs/8116952589/job/22188138475?pr=26294
https://groups.google.com/g/sympy/c/OAcu4xV-R7k
oscarbenjamin commentedon Mar 3, 2024
I can't reproduce the segfault locally so it is only in CI for now...
mattip commentedon Mar 3, 2024
Can you try a CI run where you only use one of nightly numpy/nightly scipy to try to narrow it down a bit?
seberg commentedon Mar 3, 2024
Almost certainly related, so not sure it is worth digging. this is the test in question (where it segfaults):
rgommers commentedon Mar 3, 2024
Yes, the
Py_SETREF(new, PyObject_Call(array_meth, args, NULL));line seems to be problematic.oscarbenjamin commentedon Mar 4, 2024
Thanks all. The sympy CI jobs are green now so it looks like it's working as expected.
If anything about
__array__should be changed on sympy's end let us know. Currently Matrix has:We could add a
copyparameter there but it can't do anything useful. There is no possibility of avoiding a "copy" because asympy.Matrixneeds to become anumpy.ndarrayso it needs to be a newly created array.Perhaps it should be something like:
The dev docs suggest that the signature should have
copy=Noneas a parameter:https://numpy.org/devdocs/user/basics.interoperability.html#the-array-method
Is the intention that not having the
copyparameter will lead to deprecation warnings in future? I don't see any warnings in CI right now.rgommers commentedon Mar 4, 2024
That signature and logic look right to me.
Indeed. Right now it's a
UserWarning, not a deprecation. It'll probably be a while before we deprecate it (a couple of releases at least I guess).charris commentedon Mar 4, 2024
I was thinking a couple of years :)
ngoldbaum commentedon Mar 4, 2024
Also you’ll only see the warning if you explicitly pass
copy=False, the keyword is not passed for the defaultcopy=Nonecase for better back compat in the common case.