Add metadata search index for more responsive FUSE #182
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Author
|
This could be improved even further by adding a read-cache into attic. As far as I could tell, the Cache class is currently used for write access only, right? As of yet, my patch still fetches chunks repeatedly if (and only if) multiple archives are loaded that have intersecting entries their metadata['items'] list... which isn't unlikely for real datasets. I wasn't sure though if this should be just slapped into the Cache class and used in |
This adds the construction of a metadata index during archive creation, which can be used to narrow down the location of particular entries within the items list. The FUSE mount uses this index to fetch only those chunks that are relevant to the specific operation instead of fetching the metadata of the entire archive. As a result, using FUSE mounts of large archives is consirably more reponsive. And more importantly, the performance isn't indirectly proportional to the the archive size anymore. Any bulk operations that require the full metadata tree anyways (such as running "find" on the entire archive) are not negatively impacted. For this to work, the filesystem traversal order had to be changed from depth-first to breadth-first, which introduces the new metadata version number 2. Any otherwise unrelated parts of the code and tests that relied on the previous behavior are adjusted accordingly.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds the construction of a metadata index during archive creation,
which can be used to narrow down the location of particular entries
within the items list. The FUSE mount uses this index to fetch only
those chunks that are relevant to the specific operation instead of
fetching the metadata of the entire archive.
As a result, using FUSE mounts of large archives is consirably more
reponsive. And more importantly, the performance isn't indirectly
proportional to the the archive size anymore. Any bulk operations that
require the full metadata tree anyways (such as running "find" on the
entire archive) are not negatively impacted.
For this to work, the filesystem traversal order had to be changed from
depth-first to breadth-first, which introduces the new metadata version
number 2. Any otherwise unrelated parts of the code and tests that
relied on the previous behavior are adjusted accordingly.