Repository navigation
ENH: ensure intent(inplace) does not mangle input array - #29929
Conversation
e0cacc6 to
32ed872
Compare
| presumably modified by the fortran routine, is copied back to the | ||
| input routine. This means one no longer has the risk that pre-existing | ||
| views or slices of the input array start pointing to unallocated | ||
| memory. |
There was a problem hiding this comment.
Sorry, but will keep it at a bit of a fly-by comment for now. This is scary (mostly the old code, it's scary enough that I suspect we should just change it... although I do wonder if we should maybe go towards an error)!
It seems to me that the swapping for outputs would lead to out=float_arr for integer array functions to just swap the float_arr to be an integer one!?
There was a problem hiding this comment.
I know, the old way meant that if you put in an integer array, your own instance would become a float array! With my new mechanism, you'd keep an integer array, with the copied-back float stuff truncated and possibly overflowing. So, definitely an aspect of "don't do that, then!".
I do think it is correct that if you put in a float32 to a fortran routine taking only float64, you just get reduced precision output (instead of your array becoming float64!). It is like the ufunc would do it (with the same mechanism, I think...).
Anyway, much of this predates having machinery to deal with these cases, and I find it quite remarkable how well f2py has held up -- I've quite happily used it myself when I started with python, and wanted not to lose all the fortran code I had written.
There was a problem hiding this comment.
At first glance, I think this was a long standing problem that prevented full API compliance. Or maybe I am thinking of another one where data pointers were swapped ...
|
@HaoZeke - would you have time to look at this PR for |
|
We just brought it up, and I think the main reason this might fail I think is that before there was never any casts, That is clearly better, but will it break downstream code? We could also just disable To me that last bit makes sense anyway, the user wanted in-place and didn't get it? That way we don't have to worry about changed behavior only about new errors. |
|
We discussed this at a recent triage meeting and would prefer to just disallow this whole construct. Is that possible? There were some other opinions too. |
|
Sorry for being AWOL I have penciled time in this weekend to review this properly and test some older projects. |
|
Yes, seems certainly to make sense to error on bad cases, I should have considered that... I'll wait with doing anything until @HaoZeke has had a chance to have a look, but FWIW, removing the option to have "inplace" would seem bad, in that it would somewhat gratuitously break code that worked perfectly well. However, refusing to run when the dtype changes seems reasonable, i.e., there would be a run-time failure if you put in, e.g., an integer array when a float was expected (compiled with "inplace"). The question then arises what we consider a "dtype change". Effectively, this is a casting question, and it seems one would definitely like to allow "equiv" (byte-order changes). The "safe" option is not relevant, since casts will happen in both directions. I think one would also want to allow "same_kind", as it seems OK to put in, e.g., a p.s. I do hope we can make a decision that does not, effectively, prevent changes to how |
|
Yeah, we should move on, it's just would be nicer to know how much and how badly it might break things. It seemed like this isn't used much (thus why "do we need this" came up). I tend towards it being OK to require exact (equivalent) dtype. I do think we do it in a few places for To me an "in-place" use-case should almost always be truly in-place anyway and it may actually surprise some users if it's allowed to not be, so I would tend towards the easy way out. We could deprecate it even, but with f2py being transient and this apparently almost never used (and most use-cases probably unaffected), I would be happy to try to flip the switch for 2.5. (We should add a release note). |
|
@HaoZeke, @seberg - I've updated my fix to Note that I think there are other pieces that might break. E.g., it doesn't look like the current code checks byte order. I think it would be reasonable to deprecate the use of |
cba18c0 to
33bf19c
Compare
|
I would prefer the strict version since that will shout at you in a useful way, while the other one could in principle just do the wrong thing silently. |
|
I can certainly do that, but as I mentioned, I worry about breaking reasonable code (float32 code being fed float64 in particular). Let's see what @HaoZeke thinks... |
| * ``intent(inplace)`` array arguments will be changed *in situ* if the argument | ||
| has a different type than expected (see the ``intent(inplace)`` | ||
| * ``intent(inplace)`` array arguments must be arrays. If these have | ||
| incompatible order or size, a converted copy is passed in, which is | ||
| copied back into the original array on exit (see the ``intent(inplace)`` |
There was a problem hiding this comment.
There is of course a performance cost now, but given that it is more correct strictly speaking, I think that's fine. Might be worth highlighting that for performance parity the arrays should be pre-aligned.
There was a problem hiding this comment.
Thanks! I made a small addition to the more expansive definition that this piece of text links to, and also added a note to the changelog entry. Overall, it should be a minimal effect - most code presumably does something sufficiently interesting that going from one to two copies does not really matter...
33bf19c to
b454fed
Compare
seberg
left a comment
There was a problem hiding this comment.
Sure, I had a brief look, but all my comments were things that are weird all over F2Py, so no more comments...
FWIW, I am still very convinced that a clear error is the much better choice here rather than modifying results.
But if you two agree, and nobody else seems to have an opinion, we'll go this route (at least in a day or so).
(There is no performance gain for users in providing an output that needs a cast, just very mild convenience.
The main argument here is to not break existing code but we do so at the cost of potentially breaking some less code but in subtle ways.)
|
If others tend to agree to me, I would also be happy with following up to putting a |
@HaoZeke - what do you think? It seems reasonable given that the documentation already recommends |
Yup that'd be very useful. That way we can also hope to get rid of it in a few releases.. |
While looking at trying to store data on instances, I ran into problems with f2py assuming one could just swap data between array instances. This is pretty bad form, but was probably necessary at the time, since we did not yet have
WRITEBACKIFCOPY. But now we do. So, this PR uses it.The commits are
wrap;I'm quite pleased with how this turned out!