You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
TSK: deprecate setting strides, dtype, and shape on arrays. #28800
As seen in #28727, mutating arrays like this is dangerous in surprising ways, particularly on the free-threaded build.
We should guide people towards safer alternatives that create new views or copies of arrays instead of mutating.
We should first try deprecating strides and dtype, then shape, since we expect that setting shape is used the most (before np.reshape it was the only way to do it).
To do the actual deprecation, first someone needs to add the warning and deal with any fallout in the NumPy tests. Make sure the deprecation warnings indicate clearly how to update affected code. It would also be good to look at some downstream libraries like SciPy, matplotlib, pandas, and scikit-learn to see how much noise these deprecations cause in downstream tests.
If it isn't too bad and the fixes are straightforward, we should be good to go.
Tiny correction: Reshape always existed, but raising an error when it failed with copy=False is new (i.e. knowing that the result is a view). Although, the copy was made even on error before.
But yeah, shape assignment was/is pretty common (I have seen it in educational materials, maybe it still is).
It may be useful to make as_strided faster. It's rather slow compared to transpose / reshape (for the cases where it is identical), even though it shouldn't be doing more work than either of those
From a user and astropy-dev perspective, I have always conceptualized shape as being a lightweight attribute that can be changed freely and merely informs the array how to interpret the item in __getitem__.
I suspect that deprecating shape as a mutable attribute will impact a lot of user code. E.g. in astropy tables it would always be preferable to change the shape in-place.
>>> from astropy.table import Table
>>> t = Table({"a": [1, 2, 3, 4]})
>>> t["a"].shape = (2, 2) # In-place, column id unchanged
>>> t["a"] = np.reshape(t["a"], (2, 2)) # Bunch of Python code to validate and replace column, 4 times slower
This is not the end of the world, but I just want to note the impact.
One naive question I have is how changing the shape attribute is more thread-unsafe than mutating the array data itself.
One naive question I have is how changing the shape attribute is more thread-unsafe than mutating the array data itself.
It's also about static type safety. @jorenham might be able to elaborate more.
Setting the shape mutates the dimensions array, which is especially problematic if another thread is iterating over the array while it's getting reshaped. It's similar to the problem that happens when you change the dtype - you mutate the array metadata.
See array_desr_set and array_shape_set in getset.c for the gory details.
IMO deprecating setting shape is the weakest case - I've definitely mutated shapes before. I see both sides here and it's hard to choose what the best answer here is. If we leave it as-is, then there's a hole in the type system where there's a commom operation that can cause the static type of an object to change underneath an object via mutation of an attribute. If we change it, we cause churn and have an API that's a little harder to understand.
Reacted by Sebastian Berg, Derek Homeier and Thibaud Wilson Stettler
It's also about static type safety. @jorenham might be able to elaborate more.
Ah! An opportunity to rant about static typing; don't mind if I do:
In static-typing jargon the numpy.ndarray class is called a generic type. It has two (covariant) type parameters: One for the .shape, and another for .dtype. The shape type must be assignable to tuple[int, ...], which includes any integer tuple. The dtype type (not a typo, and no pub intended) must be assignable to np.dtype, for obvious reasons. So if a is a one-dimensional float64 array, it can be annotated as
a: np.ndarray[tuple[int], np.dtype[np.float64]]
When you do something like
a.shape= (1, 2, 3)
then a is no longer one-dimensional, but the type says it is. Type-checkers will allow you to pass a to a function that only accepts 1d arrays, otherwise it'll nuke the entire universe. But a is no longer one-dimensional, even though type-checkers are convinced that it is. So type-checkers are no longer to protect you from nuking the universe. And clearly, the same problems arise when you mutate dtype in-place.
I can see delaying this, or at least keeping it as a deprecation warning for a very long time.
Would it help to advertise the arr._set_shape(new_shape) around and advertise it (as in the deprecation/error can say: you can use _set_shape but it is unsafe on arrays that may be used elsewhere and breaks typing)?
EDIT: To be clear, while I always tried to avoid .shape setting, I realize that it is used a lot, and I can see a case of giving up on this just because it's not worth the downstream annoyance.
Thanks @ngoldbaum and @jorenham (and @seberg's comment that came in while I was writing this) for the explanations. I would just urge that numpy carefully weigh the impact to the wide user community vs. the benefits. E.g. is it sufficient to document clearly that mutating shape is not thread safe instead of forbidding it? I don't know the answer but it seems a reasonable question.
On the issue of static typing -- if this is part of the risk/benefit equation, then I think it is important to consider how broadly static type checking is actually used in practice. I can only relate concrete information about Astropy core. The policy now is that developers are free to add type annotation, but it is not required. There is a formal recognition that some core developers are not interested in type annotation, and this is accepted. Static type checking is not done and there are no plans to do so in the near future.
On the issue of static typing -- if this is part of the risk/benefit equation, then I think it is important to consider how broadly static type checking is actually used in practice.
The authors of that link included an important caveat: It makes sense that a survey about types is going to get higher responses from type enthusiasts. 👀 🤔
Sounds like a good idea! I'm traveling this week so if you or someone else would lime to edit the original issue please feel free, otherwise I'll get to it when I get back.
No problem, I added a checklist - @eendebakpt, I'm not quite sure which of the shape/dtype deprecation ones is the correct one, so I linked to both for now.
FWIW, I think the second PR is good and we could just give it a shot. I would be even happy to advertise _set_dtype as unsafe "if you use this and things crash it's your problem you were warned" API.
One naive question I have is how changing the shape attribute is more thread-unsafe than mutating the array data itself.
Concurrent mutation of the data (e.g. array[2, 3]= 5) can lead to funny results (you could call them incorrect, but that depends on your point of view). That is hard to debug and surprising, but one could argue this is the user responsibility.
But changing the shape in a concurrent thread can lead to a crash of the interpreter. A simple example:
Two threads A and B are working on an array of shape (10, 2).
Thread A wants to loop over the 2 dimensions of the array. It first reads the size of first dimension 0 (e.g. 10).
A concurrent thread B resizes the array to shape (2, 10)
Thread A then reads the size of the second dimension. This was 2, but (because of the resize by thread B) has been updated to value 10.
Thread A performs an iteration over the data of the array assuming the shape of the array (10, 10). This leads to an IndexError if executed from the Python side, but can lead corruption of the the interpreter when executed from the C side. (even when working full from the python side, there are some numpy methods not thread safe and can crash)
PyArray_Resize is occasionally useful in array creation routines where the total number of elements is not know at creation time, allowing the array to grow like a C++ vector. Constructing a new array and manually copying the data. I acknowledge that this is a little niche, but it does come up from time to time.
This, of course, is only safe on arrays where there is single owner, but I think this is already a condition in the existing API.
FWIW, we were only mainly thinking about the Python side one right now. So the C-side version is actually a work-around in that case.
In most cases concatenate(chunks) is a reasonable alternative really, but the reason I wrote is that I am not sure that it really is in almost all of these (yes rather niche) use-cases.
@inakleinbottle - PyArray_Resize is used similarly inside numpy itself (more obviously after I refactored ad-hoc equivalents to using it in #29925), so indeed that is not going to go anywhere any time soon.
Hello, I am working on removing all deprecated (or planned to be deprecated) functionality in a downstream project. Is there a version of NumPy that has all of these deprecations applied? I would like to see a warning for all cases where .strides, .dtype, and .shape are set on a NumPy array. Is installing the current main or nightly sufficient? Or would I need to make a version that is a merge of main, 29575, and 29536?
Or would I need to make a version that is a merge of main, 29575, and 29536?
You would need to merge the open PRs and compile on your own system. The two open PRs will probably not be merged before development of numpy 2.5 starts.
Ah, resize isn't fully deprecated, although on new Python versions very strict. We can open a new issue about it (I am personally a bit unsure about it, I don't like the function, but I am not sure it is worth doing more than documenting it as "don't normally use" which we did. The problem with these attributes is that they seem very reasonable to set without docs to the contrary).
🤷 no huge opinion from me. I have some reservation to just deleting it fully (hiding it away would make more sense to me, but not sure how useful it is). While I agree most should be avoided, I am not really sure it isn't useful and sensible enough in the cases where it is used...
That said, I don't care if we have this type-only deprecation around indefinitely, although it is a bit misleading if it is only deprecated in typing (if just from the message) if we don't plan on following through soon.
Activity
seberg commentedon Apr 23, 2025
Tiny correction: Reshape always existed, but raising an error when it failed with
copy=Falseis new (i.e. knowing that the result is a view). Although, the copy was made even on error before.But yeah, shape assignment was/is pretty common (I have seen it in educational materials, maybe it still is).
jorenham commentedon Apr 23, 2025
Yes please! The no. 1 rule of typing is that types don't change, and
dtypeandshapeare part of thendarraytype.ricardoV94 commentedon May 8, 2025
It may be useful to make
as_stridedfaster. It's rather slow compared to transpose / reshape (for the cases where it is identical), even though it shouldn't be doing more work than either of thosetaldcroft commentedon Aug 28, 2025
From a user and astropy-dev perspective, I have always conceptualized
shapeas being a lightweight attribute that can be changed freely and merely informs the array how to interpret theitemin__getitem__.I suspect that deprecating
shapeas a mutable attribute will impact a lot of user code. E.g. in astropy tables it would always be preferable to change the shape in-place.This is not the end of the world, but I just want to note the impact.
One naive question I have is how changing the
shapeattribute is more thread-unsafe than mutating the array data itself.ngoldbaum commentedon Aug 28, 2025
It's also about static type safety. @jorenham might be able to elaborate more.
Setting the shape mutates the dimensions array, which is especially problematic if another thread is iterating over the array while it's getting reshaped. It's similar to the problem that happens when you change the dtype - you mutate the array metadata.
See
array_desr_setandarray_shape_setingetset.cfor the gory details.IMO deprecating setting
shapeis the weakest case - I've definitely mutated shapes before. I see both sides here and it's hard to choose what the best answer here is. If we leave it as-is, then there's a hole in the type system where there's a commom operation that can cause the static type of an object to change underneath an object via mutation of an attribute. If we change it, we cause churn and have an API that's a little harder to understand.jorenham commentedon Aug 28, 2025
Ah! An opportunity to rant about static typing; don't mind if I do:
In static-typing jargon the
numpy.ndarrayclass is called a generic type. It has two (covariant) type parameters: One for the.shape, and another for.dtype. The shape type must be assignable totuple[int, ...], which includes any integer tuple. The dtype type (not a typo, and no pub intended) must be assignable tonp.dtype, for obvious reasons. So ifais a one-dimensionalfloat64array, it can be annotated asWhen you do something like
then
ais no longer one-dimensional, but the type says it is. Type-checkers will allow you to passato a function that only accepts 1d arrays, otherwise it'll nuke the entire universe. Butais no longer one-dimensional, even though type-checkers are convinced that it is. So type-checkers are no longer to protect you from nuking the universe. And clearly, the same problems arise when you mutatedtypein-place.seberg commentedon Aug 28, 2025
I can see delaying this, or at least keeping it as a deprecation warning for a very long time.
Would it help to advertise the
arr._set_shape(new_shape)around and advertise it (as in the deprecation/error can say: you can use_set_shapebut it is unsafe on arrays that may be used elsewhere and breaks typing)?EDIT: To be clear, while I always tried to avoid
.shapesetting, I realize that it is used a lot, and I can see a case of giving up on this just because it's not worth the downstream annoyance.taldcroft commentedon Aug 28, 2025
Thanks @ngoldbaum and @jorenham (and @seberg's comment that came in while I was writing this) for the explanations. I would just urge that numpy carefully weigh the impact to the wide user community vs. the benefits. E.g. is it sufficient to document clearly that mutating
shapeis not thread safe instead of forbidding it? I don't know the answer but it seems a reasonable question.On the issue of static typing -- if this is part of the risk/benefit equation, then I think it is important to consider how broadly static type checking is actually used in practice. I can only relate concrete information about Astropy core. The policy now is that developers are free to add type annotation, but it is not required. There is a formal recognition that some core developers are not interested in type annotation, and this is accepted. Static type checking is not done and there are no plans to do so in the near future.
jorenham commentedon Aug 28, 2025
According to last year's Python typing survey that's 91%.
taldcroft commentedon Aug 28, 2025
The authors of that link included an important caveat: It makes sense that a survey about types is going to get higher responses from type enthusiasts. 👀 🤔
4 remaining items
mhvk commentedon Oct 7, 2025
No problem, I added a checklist - @eendebakpt, I'm not quite sure which of the shape/dtype deprecation ones is the correct one, so I linked to both for now.
seberg commentedon Oct 7, 2025
FWIW, I think the second PR is good and we could just give it a shot. I would be even happy to advertise
_set_dtypeas unsafe "if you use this and things crash it's your problem you were warned" API.eendebakpt commentedon Nov 9, 2025
Concurrent mutation of the data (e.g.
array[2, 3]= 5) can lead to funny results (you could call them incorrect, but that depends on your point of view). That is hard to debug and surprising, but one could argue this is the user responsibility.But changing the
shapein a concurrent thread can lead to a crash of the interpreter. A simple example:(10, 2).(2, 10)(10, 10). This leads to anIndexErrorif executed from the Python side, but can lead corruption of the the interpreter when executed from the C side. (even when working full from the python side, there are some numpy methods not thread safe and can crash)@taldcroft
inakleinbottle commentedon Nov 13, 2025
PyArray_Resize is occasionally useful in array creation routines where the total number of elements is not know at creation time, allowing the array to grow like a C++ vector. Constructing a new array and manually copying the data. I acknowledge that this is a little niche, but it does come up from time to time.
This, of course, is only safe on arrays where there is single owner, but I think this is already a condition in the existing API.
seberg commentedon Nov 13, 2025
FWIW, we were
onlymainly thinking about the Python side one right now. So the C-side version is actually a work-around in that case.In most cases
concatenate(chunks)is a reasonable alternative really, but the reason I wrote is that I am not sure that it really is in almost all of these (yes rather niche) use-cases.mhvk commentedon Nov 13, 2025
@inakleinbottle -
PyArray_Resizeis used similarly inside numpy itself (more obviously after I refactored ad-hoc equivalents to using it in #29925), so indeed that is not going to go anywhere any time soon.nickodell commentedon Nov 22, 2025
Hello, I am working on removing all deprecated (or planned to be deprecated) functionality in a downstream project. Is there a version of NumPy that has all of these deprecations applied? I would like to see a warning for all cases where .strides, .dtype, and .shape are set on a NumPy array. Is installing the current main or nightly sufficient? Or would I need to make a version that is a merge of main, 29575, and 29536?
eendebakpt commentedon Nov 22, 2025
You would need to merge the open PRs and compile on your own system. The two open PRs will probably not be merged before development of numpy 2.5 starts.
seberg commentedon Apr 9, 2026
I think these are actually now all in impressively enough.
If we have to pull it out again (probably the
dtypeone most likely), I think we should open a new issue.seberg commentedon Apr 9, 2026
Ah,
resizeisn't fully deprecated, although on new Python versions very strict. We can open a new issue about it (I am personally a bit unsure about it, I don't like the function, but I am not sure it is worth doing more than documenting it as "don't normally use" which we did. The problem with these attributes is that they seem very reasonable to set without docs to the contrary).jorenham commentedon Apr 9, 2026
Oh, the stubs say it is:
numpy/numpy/__init__.pyi
Lines 2250 to 2255 in df35519
Should that be corrected?
seberg commentedon Apr 9, 2026
🤷 no huge opinion from me. I have some reservation to just deleting it fully (hiding it away would make more sense to me, but not sure how useful it is). While I agree most should be avoided, I am not really sure it isn't useful and sensible enough in the cases where it is used...
That said, I don't care if we have this type-only deprecation around indefinitely, although it is a bit misleading if it is only deprecated in typing (if just from the message) if we don't plan on following through soon.