Skip to content

optimize PLAIN string columns in parquet to dictionary array #10590

Description

@Rich-T-kid

Is your feature request related to a problem or challenge?

When reading a PLAIN-encoded string/binary column into a Dictionary Arrow type, the current sad path in ByteArrayDictionaryReader materializes string data twice:

  1. Page bytes -> OffsetBuffer (copy #/1)
  2. OffsetBuffer -> StringArray -> GenericByteDictionaryBuilder which hashes every value and copies unique ones into the dict values buffer (copy #/2)

each row also incurs a hash cost. Not sure if this is avoidable but worth flagging

Describe the solution you'd like

  1. add benchmarks and validate that the existing ones cover the changes this issues intends to make

  2. Do a single-pass decode that maintains the dedup HashTable during page scanning. Since PLAIN byte array encoding is [4-byte length][bytes], the bytes are already in memory when we scan them. We can hash before copying, check the table, and only write new entries into the dictionary values buffer — emitting key indices directly.

Describe alternatives you've considered

n/a

Additional context

Activity

  1. Rich-T-kid commented on Aug 7, 2026

    @Rich-T-kid
    ContributorAuthor

    take

  2. added a commit that references this issue on Aug 11, 2026
  3. added a commit that references this issue on Aug 22, 2026
    ebd3ecb
  4. added 3 commits that reference this issue on Aug 26, 2026
    3f6e0ca
    13b2eea
    32b0682
  5. added 3 commits that reference this issue on Sep 18, 2026
    884b571
    7541018
    38be0ca
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions