Repository navigation
NumPy dtype weirdness breaks np.copy for vlen datasets (was: 64-bit problem (3rd party libs)) #217
Description
Original author: [email protected] (September 22, 2011 01:03:32)
Via email, referring to the libs at
The code to reproduce is below, the last two statements fail. Although it looks like numpy fails, I have tested multiple versions and once I start using fixed length strings as datatype it works fine.
The unique statement fails with:
Traceback (most recent call last):
File "...test2.py", line 23, in <module>
print np.unique(dset['text'])
File "...Python_x64\lib\site-packages\numpy\lib\arraysetops.py", line 160, in unique
ar = ar.flatten()
ValueError: low level cast function is for unequal type numbers
The copy statement with:
Traceback (most recent call last):
File "...test2.py", line 24, in <module>
print np.copy(dset)
File "...Python_x64\lib\site-packages\numpy\lib\function_base.py", line 818, in copy
return array(a, copy=True)
ValueError: low level cast function is for unequal type numbers
Another issue I can reproduce is that sparse selection of very big char datatypes causes the program to hang, see TEST_DTYPE_hangs.
Thanks for any help with that! And by the way: a great package for hdf5!!!!
Chris
import h5py
import numpy as np
hfile = '...DB_test.hdf5'
fd = h5py.File(hfile, 'w')
TEST_DTYPE = np.dtype([('text', h5py.special_dtype(vlen=str)),
('id','<i8')])
TEST_DTYPE_working = np.dtype([('text', 'a10000'),
('id','<i8')])
TEST_DTYPE_hangs = np.dtype([('text', 'a100000000'),
('id','<i8')])
testdata = np.array(
[('Text a',2),
('Text a',1),
('Text b',2)],
dtype=TEST_DTYPE)
create groups
grp = fd.create_group('/TEST')
create dataset
dset = grp.create_dataset('testdata', dtype=TEST_DTYPE, maxshape=(None,), data=testdata, chunks=True)
print len(dset)
print np.unique(dset['text'])
print np.copy(dset)
Activity
andrewcollette commentedon Dec 27, 2012
From [email protected] on November 03, 2011 00:34:09
I am getting similar tracebacks with numpy 1.6.1 but not numpy 1.6.0 on Ubuntu 10.04 64 bit in Python 2.6. A simple example program the demonstrates the error is the following:
import h5py
if 1:
fname = 'foo.hdf5'
f = h5py.File(fname)
dt = h5py.special_dtype(vlen=str)
if 1:
f2 = h5py.File(fname)
table = f2['bar'][...]
dt = h5py.special_dtype(vlen=str)
print 'table',table
table = table.astype(dt)
This produces the following traceback:
Traceback (most recent call last):
File "Desktop/vlen_error.py", line 20, in
table = table.astype(dt)
ValueError: low level cast function is for unequal type numbers
andrewcollette commentedon Dec 27, 2012
From [email protected] on November 08, 2012 14:36:14
Confirmed for 32bit as well, using numpy=1.7.0
This is a regression from h5py somewhere between 2.0.1 and 2.1.0 (former works with numpy 1.7.0)
andrewcollette commentedon Dec 27, 2012
From [email protected] on December 13, 2012 21:56:57
This seems to be a regression in NumPy (1.6.1 to 1.6.2/1.7); in the meantime I'll see if it can be worked around (perhaps by stripping the dtype metadata when returning an array).
andrewcollette commentedon Dec 27, 2012
From [email protected] on December 13, 2012 21:59:04
See also numpy/numpy#2798
andrewcollette commentedon Dec 27, 2012
From [email protected] on December 14, 2012 05:20:08
This appears to have been triggered in h5py by the fix for issue 202. When returning data with a special dtype, the data is now returned as plain object ('O') dtype. However, the dataset has an 'O' dtype with field information. NumPy then gets confused when converting from one to the other (this is the bug in 1.6.2 and 1.7).
Returning data with field names still intact breaks other things, however. For example, if I have an array of strings with field names, doing array[index] returns a 1-element array rather than the Python object, presumably to avoid losing the field information. You would have to do array[index]['fieldname']. This was the source of the original issue 202 (people would do group[reference_array[index]] and reference_array[index] would resolve to a NumPy array, not an Reference).
Unfortunately it seems we have to pick our poison here; either live with a broken np.copy for vlen string datasets until the NumPy folks fix their code, or reintroduce the indexing bug. I'm not in favor or regressing to the issue 202 behavior, so I lean towards the former.
I will leave this bug open for the moment in case people have other opinions or ideas.
andrewcollette commentedon Dec 29, 2012
NumPy PR 2798 has fixed this issue (confirmed in 1.7rc1).