Repository navigation
Saving large array fails with savez_compressed() but works with save() #5336
Description
I got this strange problem on a CentOS box, Python 2.6.6, Numpy 1.9.1:
[minhle@node069 ~]$ python
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.version.version
'1.9.1'
>>> a = np.ones((775890380,))
>>> np.savez_compressed('/home/minhle/scratch/test.npz', a)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/minhle/.local/lib/python2.6/site-packages/numpy/lib/npyio.py", line 560, in savez_compressed
_savez(file, args, kwds, True)
File "/home/minhle/.local/lib/python2.6/site-packages/numpy/lib/npyio.py", line 597, in _savez
format.write_array(fid, np.asanyarray(val))
File "/home/minhle/.local/lib/python2.6/site-packages/numpy/lib/format.py", line 562, in write_array
array.tofile(fp)
IOError: 775890380 requested and 233691638 written
>>> np.save('/home/minhle/scratch/test.npy', a)
>>> b = np.load('/home/minhle/scratch/test.npy')
>>> b[:10]
array([ 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
>>> quit()
[minhle@node069 ~]$ cat /etc/*-release
CentOS release 6.5 (Final)
Cluster Manager v5.2
slave
LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)
Activity
[-]Saving large array fails with savez_compressed() while works with save()[/-][+]Saving large array fails with savez_compressed() but works with save()[/+]omerlevy commentedon Mar 20, 2015
Got the same error... any solution in sight?
samuela commentedon Jul 10, 2015
I just got this error too but with
np.savezas opposed tonp.savez_compressed.samuela commentedon Jul 21, 2015
For future reference, using
np.saveinstead ofnp.savez[_compressed]seems to work as @omerlevy's example code indicates.So, is this an issue with
savez? Anyone who knows numpy well have any ideas what's happening here?pv commentedon Jul 21, 2015
savezneeds to store files temporarily on disk, and perhaps you run out of space on/tmp.Try setting
TMPDIR=$HOME/tmpor so.samuela commentedon Aug 14, 2015
@pv Yeah, changing
TMPDIRdid the trick for me. It may also be worth noting that I'm on an NFS filesystem. I'm not sure what sort of effects, if any, that might have. I'm guessing that I'm into subtleties in the way that /tmp is configured.samuela commentedon Aug 14, 2015
However, I think numpy should make this issue explicit in the documentation and provide some semblance of an intelligent, approachable error message.
pwolfram commentedon Dec 1, 2015
Agreed. I ran onto this issue on HPC resources too and I agree that it would be good if the error / warning message recommended checking to make sure
TMPDIRwas set properly with adequate space.yarikoptic commentedon Jan 27, 2016
Ran into the same problem (kudos to ngoldbaum@IRC for pointing me here). I understand that for temp storage TEMPDIR is usually the logical location. BUT when dealing with large datasets, there is a reason why we might want to savez to a different location and /tmp would never be big enough. So why not to follow any other downloader's behavior and use targetfilename + '_tempSMTHRANDOM' suffix instead. This has higher possibility to succeed since would use the target location/partition which should have adequate amount of storage. Also error message should include location of that temp file to give users sensible feedback
ENH: for savez create temporary file alongside with the target file
ENH: for savez create temporary file alongside with the target file
jinghuangzhu commentedon Jun 13, 2016
I have enough space on /tmp, but still ran into the IOError:
File "/data/anaconda/lib/python2.7/site-packages/numpy/lib/npyio.py", line 534, in savez
_savez(file, args, kwds, False)
File "/data/anaconda/lib/python2.7/site-packages/numpy/lib/npyio.py", line 597, in _savez
format.write_array(fid, np.asanyarray(val))
File "/data/anaconda/lib/python2.7/site-packages/numpy/lib/format.py", line 562, in write_array
array.tofile(fp)
IOError: 938283800 requested and 364613622 written
So is this a common problem with numpy.savez? still no fix? which means I have to use "save" instead.
Improved IOError checking to be more helpful if a filesystem bug appe…