BUG: recarray __repr__ gives inaccurate representation by ahaldane · Pull Request #5523 · numpy/numpy

ahaldane · 2015-01-29T18:52:20Z

This is a followup to #5483, with the goal of solving #3581,

In #5483, I solved the problem that a "recarray" and a "record array" (nomenclature defined in #5482) looked identical by making sure that a type's subclass was listed in the repr. However, there is still the problem that recarrays are represented using the function 'rec.array' even though this function technically creates record arrays, not recarrays.

So I have updated recarray.__repr__ to give accurate representations. Demonstration:

Setup:

>>> a = np.array([(1,'ABC'), (2, "DEF")], dtype=[('foo', int), ('bar', 'S4')])
>>> recordarr = np.rec.array(a)
>>> recarr = a.view(np.recarray)
>>> recordview = a.view(dtype((np.record, a.dtype)))

(Very) Old Behavior:

>>> recordarr
rec.array([(1, 'ABC'), (2, 'DEF')], 
      dtype=[('foo', '<i8'), ('bar', 'S4')])
>>> recarr
rec.array([(1, 'ABC'), (2, 'DEF')], 
      dtype=[('foo', '<i8'), ('bar', 'S4')])
>>> recordview
array([(1, 'ABC'), (2, 'DEF')], 
      dtype=[('foo', '<i8'), ('bar', 'S4')])

Behavior after #5483:

>>> recordarr
rec.array([(1, 'ABC'), (2, 'DEF')], 
      dtype=(numpy.record, [('foo', '<i8'), ('bar', 'S4')]))
>>> recarr
rec.array([(1, 'ABC'), (2, 'DEF')], 
      dtype=[('foo', '<i8'), ('bar', 'S4')])
>>> recordview
array([(1, 'ABC'), (2, 'DEF')], 
      dtype=(numpy.record, [('foo', '<i8'), ('bar', 'S4')]))

New Behavior:

>>> recordarr
rec.array([(1, 'ABC'), (2, 'DEF')], 
      dtype=[('foo', '<i8'), ('bar', '|S4')])
>>> recarr
array([(1, 'ABC'), (2, 'DEF')], 
      dtype=[('foo', '<i8'), ('bar', 'S4')]).view(numpy.recarray)
>>> recordview
array([(1, 'ABC'), (2, 'DEF')], 
      dtype=(numpy.record, [('foo', '<i8'), ('bar', 'S4')]))

Representing the recarr using array plus a view seems a little clunky, but there aren't many other options. The only other option I can think of is to use the recarray constructor, but that seems even clunkier since it has a totally different signature.

However, perhaps the clunkier description will be motivation for people to always use np.rec.array, and help emphasize that taking view(np.recarray) gives you a weird hybrid.

Also, I now get some extra | in the dtype since I use dtype.descr, hopefully that's acceptable.

I also fixed some missing parentheses in the record array docs.

charris · 2015-01-30T02:39:47Z

Should maybe be self.dtype.type is record.

charris · 2015-01-30T02:41:17Z

Well, that representation is certainly a bit strange, but, I suppose, accurate ;)

LGTM modulo possible nitpick.

In numpy#5483, I solved the problem that a "recarray" and a "record array" (nomenclature defined in numpy#5482) looked identical by making sure that a type's subclass was listed in the repr. However, recarrays are still represented using the function 'rec.array' even though this function technically creates record arrays, not recarrays. So I have updated recarray.__repr__. Setup: >>> a = np.array([(1,'ABC'), (2, "DEF")], dtype=[('foo', int), ('bar', 'S4')]) >>> recordarr = np.rec.array(a) >>> recarr = a.view(np.recarray) Behavior after numpy#5483: >>> recordarr rec.array([(1, 'ABC'), (2, 'DEF')], dtype=(numpy.record, [('foo', '<i8'), ('bar', 'S4')])) >>> recarr rec.array([(1, 'ABC'), (2, 'DEF')], dtype=[('foo', '<i8'), ('bar', 'S4')]) New Behavior: >>> recordarr rec.array([(1, 'ABC'), (2, 'DEF')], dtype=[('foo', '<i8'), ('bar', '|S4')]) >>> recarr array([(1, 'ABC'), (2, 'DEF')], dtype=[('foo', '<i8'), ('bar', 'S4')]).view(numpy.recarray)

ahaldane · 2015-01-30T19:24:59Z

Done.

Thanks for checking over everything!

I think this fixes #3581. At least, it explains why that behavior is expected.

BUG: recarray __repr__ gives inaccurate representation

charris · 2015-01-30T23:30:08Z

Merged, thanks @ahaldane .

juliantaylor · 2015-03-03T21:40:19Z

this breaks scipy/io/tests/test_idl.py

ERROR: test_idl.TestStructures.test_scalars_replicated_3d
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/scipy/io/tests/test_idl.py", line 190, in test_scalars_replicated_3d
    s = readsav(path.join(DATA_PATH, 'struct_scalars_replicated_3d.sav'), verbose=False)
  File "/usr/lib/python2.7/dist-packages/scipy/io/idl.py", line 813, in readsav
    replace, new = _replace_heap(r['data'], heap)
  File "/usr/lib/python2.7/dist-packages/scipy/io/idl.py", line 599, in _replace_heap
    for ir, record in enumerate(variable):
  File "/tmp/local/lib/python2.7/site-packages/numpy/core/records.py", line 481, in __getitem__
    return obj.view(dtype=(self.dtype.type, obj.dtype.descr))
  File "/tmp/local/lib/python2.7/site-packages/numpy/core/records.py", line 540, in view
    return ndarray.view(self, dtype)
  File "/tmp/local/lib/python2.7/site-packages/numpy/core/records.py", line 457, in __setattr__
    raise exctype(value)
TypeError: Cannot change data-type for object array.

ahaldane · 2015-03-03T22:07:47Z

I'll take a closer look tonight.

The purpose of the line that raises the error is to make accessing a structured field of an array with np.record dtype, also be returned as an np.record. I think the idea is correct and should be kept.

I don't think the bug is in this code, but rather I think this exposes the problem seen in a number of other issues, that you can't view object arrays. I happen to be trying to solve this problem in #5548 (see part 1 there for a list of similar problems with recarrays).

I haven't had enough time to work on that PR lately, and the scope of changes is getting a little too ambitious. However, I think the portion of that PR that fixes object array views is good to go. I could submit that as a smaller PR on its own, which I think would solve this and related issues.

rgommers · 2015-04-11T13:09:12Z

gh-5762 was opened for this and marked as a blocker for 1.10.

ahaldane · 2015-04-27T16:10:38Z

I don't think the bug is in this PR, exactly. Rather, this PR uses some views of the original array, and this PR merely exposes the fact that currently views into object arrays do not work properly.

This (gh-5762) would be solved by my other PR #5548, which allows views into object arrays. I just ran the test case in gh-5762 with that PR and it works fine.

ahaldane force-pushed the recarray_fixrepr branch 3 times, most recently from ed10d2b to d802e09 Compare January 29, 2015 19:54

charris reviewed Jan 30, 2015
View reviewed changes

Comment thread numpy/core/records.py Outdated

charris Jan 30, 2015

Copy link
Copy Markdown

Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should maybe be self.dtype.type is record.

ahaldane force-pushed the recarray_fixrepr branch from d802e09 to 73a74e9 Compare January 30, 2015 19:09

charris added a commit that referenced this pull request Jan 30, 2015

Merge pull request #5523 from ahaldane/recarray_fixrepr

ea52028

BUG: recarray __repr__ gives inaccurate representation

charris merged commit ea52028 into numpy:master Jan 30, 2015

juliantaylor mentioned this pull request Mar 3, 2015

test_idl fails with numpy master scipy/scipy#4589

Closed

rgommers added component: numpy._core 03 - Maintenance labels Apr 11, 2015

ahaldane mentioned this pull request Apr 27, 2015

TypeError: Cannot change data-type for object array #5762

Closed

ahaldane mentioned this pull request Jun 4, 2015

record array is made up of numpy.void, not numpy.record #3581

Closed

ahaldane deleted the recarray_fixrepr branch January 17, 2018 23:40

Navigation Menu

Repository navigation

BUG: recarray repr gives inaccurate representation - #5523

BUG: recarray repr gives inaccurate representation#5523
charris merged 1 commit into
numpy:masterfrom
ahaldane:recarray_fixrepr

ahaldane commented Jan 29, 2015

Uh oh!

charris Jan 30, 2015

Uh oh!

charris commented Jan 30, 2015

Uh oh!

ahaldane commented Jan 30, 2015

Uh oh!

charris commented Jan 30, 2015

Uh oh!

juliantaylor commented Mar 3, 2015

Uh oh!

ahaldane commented Mar 3, 2015

Uh oh!

rgommers commented Apr 11, 2015

Uh oh!

ahaldane commented Apr 27, 2015

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ahaldane commented Jan 29, 2015

Uh oh!

charris Jan 30, 2015

Choose a reason for hiding this comment

Uh oh!

charris commented Jan 30, 2015

Uh oh!

ahaldane commented Jan 30, 2015

Uh oh!

charris commented Jan 30, 2015

Uh oh!

juliantaylor commented Mar 3, 2015

Uh oh!

ahaldane commented Mar 3, 2015

Uh oh!

rgommers commented Apr 11, 2015

Uh oh!

ahaldane commented Apr 27, 2015

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants