Skip to content

Dictionary parquet reader is ~2-3x slower than Utf8View on low-cardinality string columns #11414

Closed

Description

@Rich-T-kid

Describe the problem

When reading a dictionary-encoded parquet string column back as its native Dictionary(Int32, Utf8) type, the arrow reader is slower than reading the same column as Utf8View across the low-to-medium cardinality range, even though Dict should strictly do less work (no gather, no per-row copy).

Measured on main, 1M rows, 16-byte strings, batch_size=8192, UNCOMPRESSED pages all RLE_DICTIONARY:

cardinality Dict read Utf8View read View/Dict
10 629 µs 629 µs ~1.0×
100 582 µs 582 µs ~1.0×
1000 621 µs 621 µs ~1.0×

(Both paths read the same bytes; Dict should win by a wide margin.)

Profiling (sample) shows the Dict path spends ~40% of CPU in bounds validation at DictionaryBuffer::into_array finalization, plus notable time in per-batch keys-vector alloc/resize/truncate.

Describe the solution

Three targeted changes to the dict reader:

  1. Track a per-batch key_upper_bound so finalization can skip the full O(n) bounds check when every observed key is in range.
  2. Replace keys.resize(len, 0) -> decode -> truncate with reserve + spare_capacity_mut + set_len to avoid zeroing + reshrinking the key buffer every batch.
  3. Reuse the keys backing Buffer across batches (via Buffer::into_vec when refcount drops to 1) instead of allocating a fresh Vec<K> per batch.

After all three, Dict at card=10 drops from 629 µs -> 218 µs (~2.9x), restoring the expected Dict < View ordering across the full matrix.

Additional context

  • Benchmark added as a separate PR (arrow_reader_string_encoding).
  • Related: DataFusion #24227 (RLE->Dictionary) regressions on low-cardinality string groupings may share the same root cause.

Activity

  1. Rich-T-kid commented on Oct 8, 2026

    @Rich-T-kid
    ContributorAuthor

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions