You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Dictionary parquet reader is ~2-3x slower than Utf8View on low-cardinality string columns #11414
When reading a dictionary-encoded parquet string column back as its native Dictionary(Int32, Utf8) type, the arrow reader is slower than reading the same column as Utf8View across the low-to-medium cardinality range, even though Dict should strictly do less work (no gather, no per-row copy).
Measured on main, 1M rows, 16-byte strings, batch_size=8192, UNCOMPRESSED pages all RLE_DICTIONARY:
cardinality
Dict read
Utf8View read
View/Dict
10
629 µs
629 µs
~1.0×
100
582 µs
582 µs
~1.0×
1000
621 µs
621 µs
~1.0×
(Both paths read the same bytes; Dict should win by a wide margin.)
Profiling (sample) shows the Dict path spends ~40% of CPU in bounds validation at DictionaryBuffer::into_array finalization, plus notable time in per-batch keys-vector alloc/resize/truncate.
Describe the solution
Three targeted changes to the dict reader:
Track a per-batch key_upper_bound so finalization can skip the full O(n) bounds check when every observed key is in range.
Replace keys.resize(len, 0) -> decode -> truncate with reserve + spare_capacity_mut + set_len to avoid zeroing + reshrinking the key buffer every batch.
Reuse the keys backing Buffer across batches (via Buffer::into_vec when refcount drops to 1) instead of allocating a fresh Vec<K> per batch.
After all three, Dict at card=10 drops from 629 µs -> 218 µs (~2.9x), restoring the expected Dict < View ordering across the full matrix.
Additional context
Benchmark added as a separate PR (arrow_reader_string_encoding).
Related: DataFusion #24227 (RLE->Dictionary) regressions on low-cardinality string groupings may share the same root cause.
Activity
Parquet dict string encoding bench (apache#11417)
Rich-T-kid commentedon Oct 8, 2026
take
perf(parquet): skip per-batch key-bounds scan when RLE bit width prov… (