Caffe cannot handle HDF5 files larger as large as 20GB? #2953
Description
I have a training database stored in the hdf5 format. However caffe immediately breaks down when it tries to train on it. Error-Message:
I0820 16:56:50.634572 15886 hdf5_data_layer.cpp:80] Loading list of HDF5 filenames from: /home/Databases/train.txt
I0820 16:56:50.634627 15886 hdf5_data_layer.cpp:94] Number of HDF5 files: 1
F0820 16:56:50.655230 15886 blob.cpp:101] Check failed: data_
*** Check failure stack trace: ***
@ 0x7f5f7eebcdaa (unknown)
@ 0x7f5f7eebcce4 (unknown)
@ 0x7f5f7eebc6e6 (unknown)
@ 0x7f5f7eebf687 (unknown)
@ 0x7f5f7f2b63ce caffe::Blob<>::mutable_cpu_data()
@ 0x7f5f7f20e85d caffe::hdf5_load_nd_dataset<>()
@ 0x7f5f7f2575ae caffe::HDF5DataLayer<>::LoadHDF5FileData()
@ 0x7f5f7f2563d8 caffe::HDF5DataLayer<>::LayerSetUp()
@ 0x7f5f7f2d0332 caffe::Net<>::Init()
@ 0x7f5f7f2d1df2 caffe::Net<>::Net()
@ 0x7f5f7f2ddec0 caffe::Solver<>::InitTrainNet()
@ 0x7f5f7f2defd3 caffe::Solver<>::Init()
@ 0x7f5f7f2df1a6 caffe::Solver<>::Solver()
@ 0x40c4b0 caffe::GetSolver<>()
@ 0x406481 train()
@ 0x404a21 main
@ 0x7f5f7e3cdec5 (unknown)
@ 0x404fcd (unknown)
@ (nil) (unknown)
When I split my training database into a smaller chunk (~13GB) everything works fine (all other parameters remained unchanged).
So I guess caffe has a problem with large HDF5 files?