Merged
Conversation
Add slow compression mode, but with the best compression ratio available Allows a nice size improvement over "better" mode, but at a significant CPU penalty. Typical speed is currently around 20MB/s Some speed examples: ``` github-june-2days-2019.json zskp 1 6273951764 699045015 11634 514.29 github-june-2days-2019.json zskp 2 6273951764 617881763 17756 336.96 github-june-2days-2019.json zskp 3 6273951764 537511906 35785 167.20 github-june-2days-2019.json zskp 4 6273951764 518319822 102880 58.16 silesia.tar zskp 1 211947520 73118028 713 283.09 silesia.tar zskp 2 211947520 67504318 1049 192.50 silesia.tar zskp 3 211947520 65102964 2466 81.93 silesia.tar zskp 4 211947520 62790088 8498 23.78 enwik8 zskp 1 100000000 39176328 408 233.17 enwik8 zskp 2 100000000 36036946 623 152.83 enwik8 zskp 3 100000000 33583681 1551 61.47 enwik8 zskp 4 100000000 31601631 4421 21.57 TS40.txt zskp 1 400000000 156408033 1857 205.42 TS40.txt zskp 2 400000000 144331263 2713 140.61 TS40.txt zskp 3 400000000 135435550 5997 63.61 TS40.txt zskp 4 400000000 127882512 18920 20.16 ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add slow compression mode, but with the best compression ratio available
Allows a nice size improvement over "better" mode, but at a significant CPU penalty.
Typical speed is currently around 25MB/s
Some speed examples:
So far fairly naiive implementation, but offers decent improvements.
Longer term it should probably have chaining replace some of the alternative scans. Some contribute very little.