Repository navigation
Weird behavior of structured_to_unstructured on non-trivial dtypes #13333
Description
With the following helpers:
import numpy.lib.recfunctions as rfn
def subarray(dt, shape):
return np.dtype((dt, shape))
def structured(*dts):
return np.dtype([('x{}'.format(i), dt) for i, dt in enumerate(dts)])
def inspect(dt):
arr = np.zeros((), dt)
ret = rfn.structured_to_unstructured(arr)
print(ret.shape, ret.dtype)We can try a bunch of uses of structured_to_unstructured (added in #11526):
>>> inspect(structured(int, int))
(2,) int32 # obviously ok
>>> inspect(structured(int, structured(int, int)))
(3,) int32 # nested types are flattened, ok
>>> inspect(structured(int, subarray(int, 2)))
(3,) int32 # ok: 1 + 2
>>> inspect(structured(int, subarray(int, (2, 2))))
(5,) int32 # ok: 1 + 2*2Here's where things start to go bad:
>>> inspect(structured(subarray(structured(int, int), 3)))
(3,) [('x0', '<i4'), ('x1', '<i4')] # bug?>>> inspect(structured(subarray(subarray(int, 2), 2)))
(2, 2) int32 # bug>>> inspect(structured(int))
() int32 # bug(#13334)
>>> inspect(structured(int, subarray(subarray(int, 2), 2)))
TypeError: invalid type promotion>>> inspect(structured())
dts, counts, offsets = zip(*fields)
ValueError: not enough values to unpack (expected 3, got 0)A lot of this behavior looks undesirable to me. @ahaldane, which cases were actually intended to be supported? Rather than locking ourselve into some of these weird constucts, we might want to raise an error for anything non-trivial.
Activity
ahaldane commentedon Apr 22, 2019
Good catches, I don't think the "weird" cases were intended.
Looks like you already have the fix for the one-field bug in #13332.
For the subarray bug, there is evidently some wrong recursion code. The bug contradicts the docstring for
_get_fields_and_offsets.mattip commentedon May 16, 2019
Can we close this? #13332 and #13334 were merged
eric-wieser commentedon May 16, 2019
No, the other weird cases still apply
charris commentedon Aug 20, 2019
Pushed off to 1.17.2.
ahaldane commentedon Aug 20, 2019
Thanks for the ping, I'd forgotten this was hanging.
I've got a fix for the cases above, will make PR soon.
MAINT: fix behavior of structured_to_unstructured on non-trivial dtypes
MAINT: fix behavior of structured_to_unstructured on non-trivial dtypes
MAINT: fix behavior of structured_to_unstructured on non-trivial dtypes