Repository navigation
Allow users to define the cardinality threshold for dictionary reads #24113
Description
Is your feature request related to a problem or challenge?
No response
Describe the solution you'd like
Add a config option to set a cardinality threshold, so columns above it are read as regular strings instead of dictionaries.
for example
SET datafusion.parquet.dict_threshold = 0.2 # dataset must have less than 20% unqiue values
or
SET datafusion.parquet.dict_threshold = 200 # 200 unique values max
we can support both but im leaning towards supporting a percentage threshold.
Describe alternatives you've considered
No response
Activity
Rich-T-kid commentedon Aug 5, 2026
take
Rich-T-kid commentedon Aug 13, 2026
important in this context : apache/arrow-rs#8608 (comment)
Rich-T-kid commentedon Nov 12, 2026
After extensive investigation and benchmarking, I think we should close this without adding a standalone NDV threshold.
NDV by itself is not a reliable predictor of whether dictionary reads will help. Even at medium or high cardinality, dictionaries can be beneficial when data pages remain dictionary encoded. Conversely, when a column chunk mixes
RLE_DICTIONARYand plain pages, the Arrow reader may need to expand and rebuild the dictionary during encoding transitions, which can erase the benefit or cause regressions. More context on that reader behavior is tracked in apache/arrow-rs#12195.I think the narrower encoding-aware direction is worth pursuing, but it belongs as part of the broader optimization work in #24117 rather than as a user-configured NDV threshold. Closing this specific issue in favor of that direction.
cc @asolimando