This repository was archived by the owner on Jan 7, 2025. It is now read-only.
This repository was archived by the owner on Jan 7, 2025. It is now read-only.
Support HDF5 #224
Description
I'm breaking the discussion about HDF5 out from #197 ...
BVLC/caffe#2840 is a helpful summary. This is a more detailed comparison.
Are the benefits worth the effort?
HDF5 vs. LMDB backends
| HDF5 | LMDB | |
|---|---|---|
| Compression | Whole dataset | Each datum |
| Compression options | GZIP, LZF | [Most image formats](http://docs.opencv.org/modules/highgui/doc/reading_and_writing_images_and_video.html#Mat imread%28const string& filename, int flags%29) |
HDF5Data vs. Data layers
| HDF5Data | Data | |
|---|---|---|
| Data types | N-D tensors | "N<4"-D tensors |
| Transformations | shuffle | scale, mirror, crop, mean |
Other HDF5 pros/cons
- Pros
- Not tied to any particular Caffe-specific format, so it would be easier to integrate with other frameworks
- Can have many labels/blobs all in one file
- Can use multiple files in one HDF5Data layer
- Cons
- Doesn't have the memory-mapping optimizations that LMDB does