Skip to content

TSK: deprecate setting strides, dtype, and shape on arrays. #28800

Closed

Description

@ngoldbaum

As seen in #28727, mutating arrays like this is dangerous in surprising ways, particularly on the free-threaded build.

We should guide people towards safer alternatives that create new views or copies of arrays instead of mutating.

We should first try deprecating strides and dtype, then shape, since we expect that setting shape is used the most (before np.reshape it was the only way to do it).

To do the actual deprecation, first someone needs to add the warning and deal with any fallout in the NumPy tests. Make sure the deprecation warnings indicate clearly how to update affected code. It would also be good to look at some downstream libraries like SciPy, matplotlib, pandas, and scikit-learn to see how much noise these deprecations cause in downstream tests.

If it isn't too bad and the fixes are straightforward, we should be good to go.

Activity

  1. added
    39 - free-threadingPRs and issues related to support for free-threading CPython (a.k.a. no-GIL, PEP 703)
    on Apr 23, 2025
  2. seberg commented on Apr 23, 2025

    @seberg
    Member

    (before np.reshape it was the only way to do it).

    Tiny correction: Reshape always existed, but raising an error when it failed with copy=False is new (i.e. knowing that the result is a view). Although, the copy was made even on error before.

    But yeah, shape assignment was/is pretty common (I have seen it in educational materials, maybe it still is).

  3. jorenham commented on Apr 23, 2025

    @jorenham
    Member

    Yes please! The no. 1 rule of typing is that types don't change, and dtype and shape are part of the ndarray type.

  4. added this to the 2.3.0 release milestone on Apr 23, 2025
  5. ricardoV94 commented on May 8, 2025

    @ricardoV94

    It may be useful to make as_strided faster. It's rather slow compared to transpose / reshape (for the cases where it is identical), even though it shouldn't be doing more work than either of those

  6. taldcroft commented on Aug 28, 2025

    @taldcroft

    From a user and astropy-dev perspective, I have always conceptualized shape as being a lightweight attribute that can be changed freely and merely informs the array how to interpret the item in __getitem__.

    I suspect that deprecating shape as a mutable attribute will impact a lot of user code. E.g. in astropy tables it would always be preferable to change the shape in-place.

    >>> from astropy.table import Table
    >>> t = Table({"a": [1, 2, 3, 4]})
    >>> t["a"].shape = (2, 2)  # In-place, column id unchanged
    >>> t["a"] = np.reshape(t["a"], (2, 2))  # Bunch of Python code to validate and replace column, 4 times slower
    

    This is not the end of the world, but I just want to note the impact.

    One naive question I have is how changing the shape attribute is more thread-unsafe than mutating the array data itself.

  7. ngoldbaum commented on Aug 28, 2025

    @ngoldbaum
    MemberAuthor

    One naive question I have is how changing the shape attribute is more thread-unsafe than mutating the array data itself.

    It's also about static type safety. @jorenham might be able to elaborate more.

    Setting the shape mutates the dimensions array, which is especially problematic if another thread is iterating over the array while it's getting reshaped. It's similar to the problem that happens when you change the dtype - you mutate the array metadata.

    See array_desr_set and array_shape_set in getset.c for the gory details.

    IMO deprecating setting shape is the weakest case - I've definitely mutated shapes before. I see both sides here and it's hard to choose what the best answer here is. If we leave it as-is, then there's a hole in the type system where there's a commom operation that can cause the static type of an object to change underneath an object via mutation of an attribute. If we change it, we cause churn and have an API that's a little harder to understand.

  8. jorenham commented on Aug 28, 2025

    @jorenham
    Member

    It's also about static type safety. @jorenham might be able to elaborate more.

    Ah! An opportunity to rant about static typing; don't mind if I do:

    In static-typing jargon the numpy.ndarray class is called a generic type. It has two (covariant) type parameters: One for the .shape, and another for .dtype. The shape type must be assignable to tuple[int, ...], which includes any integer tuple. The dtype type (not a typo, and no pub intended) must be assignable to np.dtype, for obvious reasons. So if a is a one-dimensional float64 array, it can be annotated as

    a: np.ndarray[tuple[int], np.dtype[np.float64]]

    When you do something like

    a.shape = (1, 2, 3)

    then a is no longer one-dimensional, but the type says it is. Type-checkers will allow you to pass a to a function that only accepts 1d arrays, otherwise it'll nuke the entire universe. But a is no longer one-dimensional, even though type-checkers are convinced that it is. So type-checkers are no longer to protect you from nuking the universe. And clearly, the same problems arise when you mutate dtype in-place.

  9. seberg commented on Aug 28, 2025

    @seberg
    Member

    I can see delaying this, or at least keeping it as a deprecation warning for a very long time.

    Would it help to advertise the arr._set_shape(new_shape) around and advertise it (as in the deprecation/error can say: you can use _set_shape but it is unsafe on arrays that may be used elsewhere and breaks typing)?

    EDIT: To be clear, while I always tried to avoid .shape setting, I realize that it is used a lot, and I can see a case of giving up on this just because it's not worth the downstream annoyance.

  10. taldcroft commented on Aug 28, 2025

    @taldcroft

    Thanks @ngoldbaum and @jorenham (and @seberg's comment that came in while I was writing this) for the explanations. I would just urge that numpy carefully weigh the impact to the wide user community vs. the benefits. E.g. is it sufficient to document clearly that mutating shape is not thread safe instead of forbidding it? I don't know the answer but it seems a reasonable question.

    On the issue of static typing -- if this is part of the risk/benefit equation, then I think it is important to consider how broadly static type checking is actually used in practice. I can only relate concrete information about Astropy core. The policy now is that developers are free to add type annotation, but it is not required. There is a formal recognition that some core developers are not interested in type annotation, and this is accepted. Static type checking is not done and there are no plans to do so in the near future.

  11. jorenham commented on Aug 28, 2025

    @jorenham
    Member

    On the issue of static typing -- if this is part of the risk/benefit equation, then I think it is important to consider how broadly static type checking is actually used in practice.

    According to last year's Python typing survey that's 91%.

  12. taldcroft commented on Aug 28, 2025

    @taldcroft

    According to last year's Python typing survey that's 91%.

    The authors of that link included an important caveat: It makes sense that a survey about types is going to get higher responses from type enthusiasts. 👀 🤔

  13. 4 remaining items

  14. mhvk commented on Oct 7, 2025

    @mhvk
    Contributor

    Sounds like a good idea! I'm traveling this week so if you or someone else would lime to edit the original issue please feel free, otherwise I'll get to it when I get back.

    No problem, I added a checklist - @eendebakpt, I'm not quite sure which of the shape/dtype deprecation ones is the correct one, so I linked to both for now.

  15. seberg commented on Oct 7, 2025

    @seberg
    Member

    FWIW, I think the second PR is good and we could just give it a shot. I would be even happy to advertise _set_dtype as unsafe "if you use this and things crash it's your problem you were warned" API.

  16. eendebakpt commented on Nov 9, 2025

    @eendebakpt
    Contributor

    One naive question I have is how changing the shape attribute is more thread-unsafe than mutating the array data itself.

    Concurrent mutation of the data (e.g. array[2, 3]= 5) can lead to funny results (you could call them incorrect, but that depends on your point of view). That is hard to debug and surprising, but one could argue this is the user responsibility.

    But changing the shape in a concurrent thread can lead to a crash of the interpreter. A simple example:

    • Two threads A and B are working on an array of shape (10, 2).
    • Thread A wants to loop over the 2 dimensions of the array. It first reads the size of first dimension 0 (e.g. 10).
    • A concurrent thread B resizes the array to shape (2, 10)
    • Thread A then reads the size of the second dimension. This was 2, but (because of the resize by thread B) has been updated to value 10.
    • Thread A performs an iteration over the data of the array assuming the shape of the array (10, 10). This leads to an IndexError if executed from the Python side, but can lead corruption of the the interpreter when executed from the C side. (even when working full from the python side, there are some numpy methods not thread safe and can crash)

    @taldcroft

  17. inakleinbottle commented on Nov 13, 2025

    @inakleinbottle
    Contributor

    PyArray_Resize is occasionally useful in array creation routines where the total number of elements is not know at creation time, allowing the array to grow like a C++ vector. Constructing a new array and manually copying the data. I acknowledge that this is a little niche, but it does come up from time to time.

    This, of course, is only safe on arrays where there is single owner, but I think this is already a condition in the existing API.

  18. seberg commented on Nov 13, 2025

    @seberg
    Member

    FWIW, we were only mainly thinking about the Python side one right now. So the C-side version is actually a work-around in that case.
    In most cases concatenate(chunks) is a reasonable alternative really, but the reason I wrote is that I am not sure that it really is in almost all of these (yes rather niche) use-cases.

  19. mhvk commented on Nov 13, 2025

    @mhvk
    Contributor

    @inakleinbottle - PyArray_Resize is used similarly inside numpy itself (more obviously after I refactored ad-hoc equivalents to using it in #29925), so indeed that is not going to go anywhere any time soon.

  20. nickodell commented on Nov 22, 2025

    @nickodell

    Hello, I am working on removing all deprecated (or planned to be deprecated) functionality in a downstream project. Is there a version of NumPy that has all of these deprecations applied? I would like to see a warning for all cases where .strides, .dtype, and .shape are set on a NumPy array. Is installing the current main or nightly sufficient? Or would I need to make a version that is a merge of main, 29575, and 29536?

  21. eendebakpt commented on Nov 22, 2025

    @eendebakpt
    Contributor

    Or would I need to make a version that is a merge of main, 29575, and 29536?

    You would need to merge the open PRs and compile on your own system. The two open PRs will probably not be merged before development of numpy 2.5 starts.

  22. seberg commented on Apr 9, 2026

    @seberg
    Member

    I think these are actually now all in impressively enough.

    If we have to pull it out again (probably the dtype one most likely), I think we should open a new issue.

  23. seberg commented on Apr 9, 2026

    @seberg
    Member

    Deprecate ndarray.resize and thus PyArray_Resize.

    Ah, resize isn't fully deprecated, although on new Python versions very strict. We can open a new issue about it (I am personally a bit unsure about it, I don't like the function, but I am not sure it is worth doing more than documenting it as "don't normally use" which we did. The problem with these attributes is that they seem very reasonable to set without docs to the contrary).

  24. jorenham commented on Apr 9, 2026

    @jorenham
    Member

    Deprecate ndarray.resize and thus PyArray_Resize.

    Ah, resize isn't fully deprecated, although on new Python versions very strict.

    Oh, the stubs say it is:

    numpy/numpy/__init__.pyi

    Lines 2250 to 2255 in df35519

    @overload
    @deprecated("Resizing a NumPy array inplace has been deprecated in NumPy 2.5")
    def resize(self, new_shape: _ShapeLike, /, *, refcheck: py_bool = True) -> None: ...
    @overload
    @deprecated("Resizing a NumPy array inplace has been deprecated in NumPy 2.5")
    def resize(self, /, *new_shape: SupportsIndex, refcheck: py_bool = True) -> None: ...

    Should that be corrected?

  25. seberg commented on Apr 9, 2026

    @seberg
    Member

    🤷 no huge opinion from me. I have some reservation to just deleting it fully (hiding it away would make more sense to me, but not sure how useful it is). While I agree most should be avoided, I am not really sure it isn't useful and sensible enough in the cases where it is used...

    That said, I don't care if we have this type-only deprecation around indefinitely, although it is a bit misleading if it is only deprecated in typing (if just from the message) if we don't plan on following through soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    17 - Task39 - free-threadingPRs and issues related to support for free-threading CPython (a.k.a. no-GIL, PEP 703)

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions