You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
optimize PLAIN string columns in parquet to dictionary array #10590
Is your feature request related to a problem or challenge?
When reading a PLAIN-encoded string/binary column into a Dictionary Arrow type, the current sad path in ByteArrayDictionaryReader materializes string data twice:
Page bytes -> OffsetBuffer (copy #/1)
OffsetBuffer -> StringArray -> GenericByteDictionaryBuilder which hashes every value and copies unique ones into the dict values buffer (copy #/2)
each row also incurs a hash cost. Not sure if this is avoidable but worth flagging
Describe the solution you'd like
add benchmarks and validate that the existing ones cover the changes this issues intends to make
Do a single-pass decode that maintains the dedup HashTable during page scanning. Since PLAIN byte array encoding is [4-byte length][bytes], the bytes are already in memory when we scan them. We can hash before copying, check the table, and only write new entries into the dictionary values buffer — emitting key indices directly.
Activity
Rich-T-kid commentedon Aug 7, 2026
take
parquet plain dict bench (#10591)
parquet plain dict bench (apache#10591)
Optimize PLAIN parquet string column to dictionary array (apache#10614)
Optimize PLAIN parquet string column to dictionary array (apache#10614)
Optimize PLAIN parquet string column to dictionary array (apache#10614)
Optimize PLAIN parquet string column to dictionary array (apache#10614)
Optimize PLAIN parquet string column to dictionary array (apache#10614)
Optimize PLAIN parquet string column to dictionary array (apache#10614)
Optimize PLAIN parquet string column to dictionary array (apache#10614)