This repository was archived by the owner on Feb 8, 2023. It is now read-only.
This repository was archived by the owner on Feb 8, 2023. It is now read-only.
IPLD Importers - endeavor notes #144
Description
The "IPLD Data Importers" endeavor is the effort to create IPLD (or even unixfs) data structures out of various kinds of data.
Motivation:
The idea is that chunking POSIX files based on things like Rabin Fingerprinting is good but not great, and that many times we can be much smarter about the files, chunking them by taking into account the file's internal structure. Doing this for a few of the most important file formats may see huge dedup winnings (tar archives, VM images, containers, ...).
Examples:
- If importing a
.tararchive into ipfs, import it s.t. (a) each tar header is an object, and (b) each file is fed back into the importer(s). - If importing a video, maybe we can chunk intelligently around (a) container formats, (b) key frames, etc.
- If importing a VM image, maybe we can chunk s.t. we import the filesystem itself correctly.
Notes:
- IPLD Data Importing - Set of Importers js-ipfs#41
- https://github.com/ipfs/js-ipfs-unixfs-engine
- go-ipfs
- https://github.com/ipfs/go-ipfs/tree/master/importer
- https://github.com/ipfs/go-ipfs/tree/master/tar
- https://github.com/ipfs/go-ipfs/tree/master/unixfs
Diagram:
# meta importer - uses the other importers
+------+ +----------+ +------------+
| | | | | |
| | | Pick an | | IPLD graph |
| file | --------> | Importer | --------> | for file. |
| | | for file | | |
| | | | | |
+------+ +----------+ +------------+
# file type specific importers
+------+ +----------+ +------------+
| | | | | |
| | | TAR IPLD | | IPLD graph |
| .tar | --------> | IMPORTER | --------> | rep a .tar |
| | | | | (smartly) |
| | | | | |
+------+ +----------+ +------------+
+------+ +----------+ +------------+
| | | | | |
| | | MP4 IPLD | | IPLD graph |
| .mp4 | --------> | IMPORTER | --------> | rep a .mp4 |
| | | | | (smartly) |
| | | | | |
+------+ +----------+ +------------+
# Info N/A. Use "good enough" importer
+------+ +----------+ +------------+
| | | | | |
| unkn | | RABIN FP | | IPLD graph |
| file | --------> | CHUNKING | --------> | for file w |
| | | IMPORTER | | rabin chnk |
| | | | | |
+------+ +----------+ +------------+