Problem
When a string column has very low cardinality (e.g., 5,000 unique values across
14 million rows, with data sorted by that column), Lance v2.0/2.1 produces files
350x larger than Parquet with the same gzip compression. Lance v2.2 improves
this to ~5x, but the gap remains
Root Cause
The v2 encoder (lance-encoding/src/compression.rs) handles string columns via
build_variable_width_compressor() which uses FSST + Binary encoding per row —
there is no automatic cardinality detection and no column-wide dictionary+RLE for
string columns.
By contrast, build_fixed_width_compressor() for integer columns does have
RLE via try_rle_for_mini_block().
Parquet automatically detects low cardinality → builds a column-wide dictionary →
encodes dictionary indices with RLE/Bit-Packing → applies gzip. This is a 2-level
optimization (dict dedup + RLE) that Lance v2 lacks for strings.
Interestingly, Lance v1 (previous/encoder.rs) DID have this optimization:
check_dict_encoding() used HyperLogLog to estimate cardinality and auto-switched
to dictionary encoding for string columns with `< 100 unique values. This was
removed in the v2 rewrite.
Problem
When a string column has very low cardinality (e.g., 5,000 unique values across
14 million rows, with data sorted by that column), Lance v2.0/2.1 produces files
350x larger than Parquet with the same gzip compression. Lance v2.2 improves
this to ~5x, but the gap remains
Root Cause
The v2 encoder (
lance-encoding/src/compression.rs) handles string columns viabuild_variable_width_compressor()which uses FSST + Binary encoding per row —there is no automatic cardinality detection and no column-wide dictionary+RLE for
string columns.
By contrast,
build_fixed_width_compressor()for integer columns does haveRLE via
try_rle_for_mini_block().Parquet automatically detects low cardinality → builds a column-wide dictionary →
encodes dictionary indices with RLE/Bit-Packing → applies gzip. This is a 2-level
optimization (dict dedup + RLE) that Lance v2 lacks for strings.
Interestingly, Lance v1 (
previous/encoder.rs) DID have this optimization:check_dict_encoding()used HyperLogLog to estimate cardinality and auto-switchedto dictionary encoding for string columns with `< 100 unique values. This was
removed in the v2 rewrite.