Skip to content

ENH: ensure intent(inplace) does not mangle input array - #29929

Merged
seberg merged 5 commits into
numpy:mainfrom
mhvk:fortranobject-no-array-swap
Feb 9, 2026
Merged

seberg merged 5 commits into
numpy:mainfrom
mhvk:fortranobject-no-array-swap

Conversation

@mhvk

@mhvk mhvk commented Oct 11, 2025

Copy link
Copy Markdown
Contributor

While looking at trying to store data on instances, I ran into problems with f2py assuming one could just swap data between array instances. This is pretty bad form, but was probably necessary at the time, since we did not yet have WRITEBACKIFCOPY. But now we do. So, this PR uses it.

The commits are

  1. Make the required changes and test them with a new piece of fortran code;
  2. Adjust the tests that check the wrapper directly. This needed a small addition to wrap;
  3. Adjust the documentation;
  4. Changelog entry.

I'm quite pleased with how this turned out!

@mhvk

mhvk commented Oct 12, 2025

Copy link
Copy Markdown
Contributor Author

@charris, @HaoZeke - not sure you are the right ones, but most fixes for f2py recently seem to have been by you, so I was hoping you might be willing to have a look at this fairly involved PR, which makes intent(inplace) behave much more reasonably. Thanks!

presumably modified by the fortran routine, is copied back to the
input routine. This means one no longer has the risk that pre-existing
views or slices of the input array start pointing to unallocated
memory.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, but will keep it at a bit of a fly-by comment for now. This is scary (mostly the old code, it's scary enough that I suspect we should just change it... although I do wonder if we should maybe go towards an error)!
It seems to me that the swapping for outputs would lead to out=float_arr for integer array functions to just swap the float_arr to be an integer one!?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know, the old way meant that if you put in an integer array, your own instance would become a float array! With my new mechanism, you'd keep an integer array, with the copied-back float stuff truncated and possibly overflowing. So, definitely an aspect of "don't do that, then!".

I do think it is correct that if you put in a float32 to a fortran routine taking only float64, you just get reduced precision output (instead of your array becoming float64!). It is like the ufunc would do it (with the same mechanism, I think...).

Anyway, much of this predates having machinery to deal with these cases, and I find it quite remarkable how well f2py has held up -- I've quite happily used it myself when I started with python, and wanted not to lose all the fortran code I had written.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At first glance, I think this was a long standing problem that prevented full API compliance. Or maybe I am thinking of another one where data pointers were swapped ...

@mhvk

mhvk commented Nov 11, 2025

Copy link
Copy Markdown
Contributor Author

@HaoZeke - would you have time to look at this PR for f2py. I think it does solve a longstanding issue (and is needed for some of my other work... we really shouldn't be mangling arrays, swapping pointers to memory, etc.). I'm also fairly confident the code itself is right, but would want to be sure the changes I made to the documentation are useful/good.

@seberg

seberg commented Nov 12, 2025

Copy link
Copy Markdown
Member

We just brought it up, and I think the main reason this might fail I think is that before there was never any casts, out was basically ignored (i.e. replaced) if it doesn't fit perfectly.
But with this, we cast the result to the given out with its dtype.

That is clearly better, but will it break downstream code? We could also just disable inplace or just raise an error when the dtype doesn't match and a copy would be needed?

To me that last bit makes sense anyway, the user wanted in-place and didn't get it? That way we don't have to worry about changed behavior only about new errors.

@mattip

mattip commented Nov 12, 2025

Copy link
Copy Markdown
Member

We discussed this at a recent triage meeting and would prefer to just disallow this whole construct. Is that possible? There were some other opinions too.

@HaoZeke

HaoZeke commented Nov 12, 2025

Copy link
Copy Markdown
Member

Sorry for being AWOL I have penciled time in this weekend to review this properly and test some older projects.

@mhvk

mhvk commented Nov 12, 2025

Copy link
Copy Markdown
Contributor Author

Yes, seems certainly to make sense to error on bad cases, I should have considered that...

I'll wait with doing anything until @HaoZeke has had a chance to have a look, but FWIW, removing the option to have "inplace" would seem bad, in that it would somewhat gratuitously break code that worked perfectly well. However, refusing to run when the dtype changes seems reasonable, i.e., there would be a run-time failure if you put in, e.g., an integer array when a float was expected (compiled with "inplace").

The question then arises what we consider a "dtype change". Effectively, this is a casting question, and it seems one would definitely like to allow "equiv" (byte-order changes). The "safe" option is not relevant, since casts will happen in both directions. I think one would also want to allow "same_kind", as it seems OK to put in, e.g., a float32 array in float64 code (or vice versa), and it would probably not materially affect the result. But I think we would then error on anything requiring "unsafe". (Ideally, I guess it would be an option, but I do not know how to implement that...)

p.s. I do hope we can make a decision that does not, effectively, prevent changes to how ndarray organizes its associated memory, though! Ideally, those become proper implementation details...

@seberg seberg added the 56 - Needs Release Note. Needs an entry in doc/release/upcoming_changes label Nov 13, 2025
@seberg

seberg commented Nov 13, 2025

Copy link
Copy Markdown
Member

Yeah, we should move on, it's just would be nicer to know how much and how badly it might break things. It seemed like this isn't used much (thus why "do we need this" came up).

I tend towards it being OK to require exact (equivalent) dtype. I do think we do it in a few places for out= parameters (but would have to try to remember where).
UFuncs are a bit different, in that they at least use buffered casts, so there is no full copy with out=.

To me an "in-place" use-case should almost always be truly in-place anyway and it may actually surprise some users if it's allowed to not be, so I would tend towards the easy way out.

We could deprecate it even, but with f2py being transient and this apparently almost never used (and most use-cases probably unaffected), I would be happy to try to flip the switch for 2.5. (We should add a release note).

@mhvk

mhvk commented Dec 10, 2025

Copy link
Copy Markdown
Contributor Author

@HaoZeke, @seberg - I've updated my fix to intent(inplace) to ensure the input array is the same kind as the array needed by the fortran routine. I decided against enforcing strictly the same dtype, as I worried that greatly increased the likelihood of breaking existing code (in particular, for 32 and 64 bit float and int). Obviously, happy to change it if you think it is better (e.g., we could explicitly allow only 32 and 64 bit versions, not 8 and 16, which are much less likely to have been used in any fortran code, and much more likely to give wrong results).

Note that I think there are other pieces that might break. E.g., it doesn't look like the current code checks byte order.

I think it would be reasonable to deprecate the use of intent(inplace) overall, but probably best as follow-up.

@seberg seberg added the triage review Issue/PR to be discussed at the next triage meeting label Dec 10, 2025
@mhvk
mhvk force-pushed the fortranobject-no-array-swap branch from cba18c0 to 33bf19c Compare December 10, 2025 16:00
@seberg

seberg commented Dec 10, 2025

Copy link
Copy Markdown
Member

I would prefer the strict version since that will shout at you in a useful way, while the other one could in principle just do the wrong thing silently.
(The main issue I see is that I am not sure if downstream builds newer versions with newer NumPy versions, allowing for updating a downstream library in a bug-fix release, which is undesireable either way though)

@mhvk

mhvk commented Dec 10, 2025

Copy link
Copy Markdown
Contributor Author

I can certainly do that, but as I mentioned, I worry about breaking reasonable code (float32 code being fed float64 in particular). Let's see what @HaoZeke thinks...

@seberg seberg added triaged Issue/PR that was discussed in a triage meeting and removed triage review Issue/PR to be discussed at the next triage meeting labels Jan 21, 2026

@HaoZeke HaoZeke left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor doc nit aside, I think this makes sense. We'll likely see some pushback in terms of the performance loss but I think it's fine. Thanks @mhvk and sorry for the very long delay!

Comment on lines -118 to +120
* ``intent(inplace)`` array arguments will be changed *in situ* if the argument
has a different type than expected (see the ``intent(inplace)``
* ``intent(inplace)`` array arguments must be arrays. If these have
incompatible order or size, a converted copy is passed in, which is
copied back into the original array on exit (see the ``intent(inplace)``

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is of course a performance cost now, but given that it is more correct strictly speaking, I think that's fine. Might be worth highlighting that for performance parity the arrays should be pre-aligned.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I made a small addition to the more expansive definition that this piece of text links to, and also added a note to the changelog entry. Overall, it should be a minimal effect - most code presumably does something sufficiently interesting that going from one to two copies does not really matter...

@mhvk
mhvk force-pushed the fortranobject-no-array-swap branch from 33bf19c to b454fed Compare February 6, 2026 21:47
@mhvk

mhvk commented Feb 6, 2026

Copy link
Copy Markdown
Contributor Author

@seberg, @charris - OK to put this in now that @HaoZeke has approved?

@seberg seberg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I had a brief look, but all my comments were things that are weird all over F2Py, so no more comments...

FWIW, I am still very convinced that a clear error is the much better choice here rather than modifying results.

But if you two agree, and nobody else seems to have an opinion, we'll go this route (at least in a day or so).

(There is no performance gain for users in providing an output that needs a cast, just very mild convenience.
The main argument here is to not break existing code but we do so at the cost of potentially breaking some less code but in subtle ways.)

@seberg

seberg commented Feb 7, 2026

Copy link
Copy Markdown
Member

If others tend to agree to me, I would also be happy with following up to putting a VisibleDeprecationWarning on the copying path.

@mhvk

mhvk commented Feb 7, 2026

Copy link
Copy Markdown
Contributor Author

If others tend to agree to me, I would also be happy with following up to putting a VisibleDeprecationWarning on the copying path.

@HaoZeke - what do you think? It seems reasonable given that the documentation already recommends intent(in,out) -- indeed, that could be suggested as part of the warning.

@HaoZeke

HaoZeke commented Feb 7, 2026

Copy link
Copy Markdown
Member

If others tend to agree to me, I would also be happy with following up to putting a VisibleDeprecationWarning on the copying path.

@HaoZeke - what do you think? It seems reasonable given that the documentation already recommends intent(in,out) -- indeed, that could be suggested as part of the warning.

Yup that'd be very useful. That way we can also hope to get rid of it in a few releases..

@seberg
seberg merged commit 2ba781a into numpy:main Feb 9, 2026
77 checks passed
@mhvk
mhvk deleted the fortranobject-no-array-swap branch March 22, 2026 17:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

01 - Enhancement 56 - Needs Release Note. Needs an entry in doc/release/upcoming_changes component: numpy.f2py triaged Issue/PR that was discussed in a triage meeting

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants