Mas i370 d30 sstmemory#371
Merged
Merged
Conversation
SQN checks are all background processes
SQN check in the penciller is used for journal (all object) folds, but mainly for journal compaction. Use this to trigger hibernation where SST files stay quiet after the compaction check.
martinsumner
marked this pull request as ready for review
March 11, 2022 11:10
Based on volume testing. Relatively speaking, far higher value to be gained from caches at higher levels (lower numbered levels). The cache at lower levels are proportionally much less efficient. so cache more at higher levels, where there is value, and less at lower levels where there is more cost relative to value.
Making significant change appears to not have had the expected positive improvement - so a more minimal change is proposed. The assumption is that the cache only really gets used for double reads in the write path (e.g. where the application reads before a write) - and so a large cache make minimal difference, but no cache still has a downside.
ThomasArts
reviewed
Mar 23, 2022
| % compact thta more advanced value, but this may cause | ||
| % confusion in snapshots. | ||
| current; | ||
| compare_to_sqn(Obj, SQN) -> |
Contributor
There was a problem hiding this comment.
Nice refactoring... gets more readable
ThomasArts
approved these changes
Mar 23, 2022
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On very large stores with lots of old, and largely untouched objects, too much memory is taken up by relatively inactive files.
The fetch_cache has an unpredictable size, and it is hard to reason or prove its effectiveness.
This change takes the existing caching level configuration, which prevents automatic fadvise of files below a certain level in the LSM tree, and also blocks the creation of the fetch_cache below a certain level.
There has been some observations of memory fragmentation as well. There is currently no hibernation of unused files. Unused files will still be used during all object folds and journal compaction because of SQN checking, so SQN checking is now used as a potential trigger for hibernation - if there is no activity in the 60s following. a SQN check the managing process for the SST file will hibernate.