I have some concern on changing the default to L4Z_LENGTH_PREFIXED because it is recently added and can potentially cause problem if segment creator gets the new version first.
Let's wait after 0.9 is released and add this in 0.10.0. We should also consider changing the default writer version to 4 which is much more efficient than 2

The V4 chunk writer automatically upgrades LZ4 to LZ4_LENGTH_PREFIXED so I could just change it LZ4 instead

Codecov Report

Merging #7797 (7d65dfb) into master (b809043) will decrease coverage by 6.50%.
The diff coverage is 50.00%.

@@             Coverage Diff              @@
##             master    #7797      +/-   ##
============================================
- Coverage     71.70%   65.20%   -6.51%     
- Complexity     4078     4083       +5     
============================================
  Files          1578     1533      -45     
  Lines         80777    78907    -1870     
  Branches      12001    11798     -203     
============================================
- Hits          57925    51449    -6476     
- Misses        18961    23797    +4836     
+ Partials       3891     3661     -230     

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b809043...7d65dfb. Read the comment docs.

Nice. When we had added support for ZSTD and LZ4 compression types in #6804, our perf testing also had similar results.

LZ4 gives nearly same (and even better sometimes) the compression ratio compared to SNAPPY and the time to compress is much faster.

ZSTD on the other hand as highest compression ratio (from our tests) but compression/decompression time was higher as well

Here are the results for reference (I think these were addd to JMH benchmarks as well) https://docs.google.com/document/d/1ie4JmVofk2o6ryCX7T4aD7GsRPKS651mroXhDk41wys/edit

LZ4 gives nearly same (and even better sometimes) the compression ratio compared to SNAPPY and the time to compress is much faster.

ZSTD on the other hand as highest compression ratio (from our tests) but compression/decompression time was higher as well

This is consistent with my experience, but it’s worth noting LZ4 has 16 compression levels which trade speed for ratio, and we default to level 1, whereas beyond level 3 a high compression mode is used - if ratio is really important the level could be exposed as a config parameter. But right now the motivation is faster decompression by default, rather than smaller file sizes.