Repository navigation
[Parquet] Encoding configuration should be easier and more automated #38441
Description
Describe the enhancement requested
Right now, configuring good encoding values is difficult for users. There is nothing to help them make those decisions, and the defaults are a bit simplistic (try RLE_DICTIONARY then fall back on PLAIN, IIUC). If they want to override encodings, they have to do so on a column-by-column basis (which probably becomes very cumbersome if there hundreds of columns).
Ideally, there should be a way for users to get an automatic selection of encodings, based on their data or at least their data types (and also the selected Parquet version), that provides a good compromise between disk footprint and decoding speed.
(in Python, think pq.write_table(..., column_encoding="auto"))
Perhaps it would be also nice for users to pass per-datatype preferences, rather than per-column.
Component(s)
C++, Parquet
Activity
pitrou commentedon Oct 24, 2023
cc @jorisvandenbossche @mapleFU @wgtmac
pitrou commentedon Oct 24, 2023
Also @mrocklin, perhaps you have specific API suggestions here.
jorisvandenbossche commentedon Oct 24, 2023
At the very least, we should also improve the documention around the different encoding options. Currently we just have this table listing them: https://arrow.apache.org/docs/dev/cpp/parquet.html#encodings (and this brief section in the Python user guide: https://arrow.apache.org/docs/dev/python/parquet.html#compression-encoding-and-file-compatibility)
For example mention for which data types they are relevant (eg BYTE_STREAM_SPLIT is only for floats, DELTA_BINARY_PACKED for integers), or if they are useful for specific characteristcs (eg many repeated values, sorted data, ..), etc.
mapleFU commentedon Oct 25, 2023
https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/papers/btrblocks.pdf
I remember here is a policy here. Though I think this paper has something weird, we can be able to using it.
However, hacking the encoder would be tricky...Currently the encoder depends on
dict_encoder_or etc, it would be a bit tricky to having a Adaptive encoder within it. Besides, different user would has different requirements. We need to balance the file-size ( related-to IO ) and the decompression/decoding speed ( CPU -bound)wgtmac commentedon Oct 25, 2023
That paper looks a little bit academic to me as it would significantly affect the write time. In many common cases, a simple sampling process (e.g. on first N input values if not skewed) would help determine a proper encoding.
pitrou commentedon Oct 25, 2023
Yes, and perhaps the encoding decisions can be remembered for the entire row batch (or the entire file?).
mapleFU commentedon Oct 25, 2023
Another problem is that, for DataPageV1, we need to both considering compression and encoding(DataPage V2 can choose not to compression if compression ratio is low). This make deciding the adaptive encoding more complex.
pitrou commentedon Oct 25, 2023
We can decide not to support DataPageV2 for this feature if it makes things easier.
mapleFU commentedon Oct 25, 2023
Yeah, I mean, usally user would pickup a compression. For datapagev1, it doesn't support a
is_compressionflag. So, all the page should be compressed ( whenever the compression ratio is)The funny thing is that, if we meet some data, and have
PLAINDELTAas candidate. DELTA might be better if not compression, but considering compression, PLAIN is better.Sometimes, the compression size even grows larger for some input. DataPageV2 can reject to compress for these types, but DataPageV1 cannot.
wgtmac commentedon Oct 26, 2023
I think it is better to decouple encoding and compression, though some compression codecs may favor different encoding schemes when data pattern varies. So it would be easier if we just focus on "best" encoding that we can decide from sample data regardless of follow-up compression. Then probably we don't have to bother with DataPage version at the moment.
pitrou commentedon Oct 26, 2023
Well, for example, the only point of BYTE_STREAM_SPLIT is that the encoded output compresses better. But if you don't compress, it's exactly the same size as the input.
So I'm afraid the strategy should take compression into account somehow, if some kind of sampling is used.
That said, the strategy can also not use sampling at all, and instead rely on simple heuristics (which ones?).
All this probably means is that:
I can draft an API when I find the time, which would be in a couple weeks probably (feel free to ping me :-)).
mapleFU commentedon Oct 26, 2023
How could you sampling and doing this?🤔 I've implement a similiar one but I must say it isn't suitable for all cases and the interface would be tricky(for example, handling
dictis tricky). Also, whenNULL FIRSTis setting, we cannot sampling any data.mapleFU commentedon Oct 26, 2023
Also, would it related to: #33885 ?
pitrou commentedon Oct 26, 2023
Slighty related, but also independent IMHO.
wgtmac commentedon Oct 26, 2023
Parquet is usually used as a low-level library in the database or data warehouse, which has a better knowledge of the actual data pattern. They can decide the best combination of encoding and compression for any column based on the data already stored. Should we provide a simple tool or something to evaluate each page of every column from an input parquet file and decides the best encoding & compression for them?
pitrou commentedon Oct 26, 2023
Such a tool can easily be developed as a separate project. Here we're talking about allowing users to get better defaults out of Parquet C++.
thisisnic commentedon Feb 18, 2026
This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.
daniel-x commentedon Apr 15, 2026
I'm interested to work on this issue, as I see high potential in it for reduced file sizes, which is a strong reason for choosing parquet as a file format for long term storage - and we would improve io speed at the same time. I personally work with data where slightly more detailed encoding defaults would reduce my zstd compressed file sizes by about 20%. Currently, specifying encoding per-column through the Python API requires a fair bit of manual configuration, which can be a barrier for new users.
Here is how I think about scoping a first step:
I believe this will address many of the common cases.
Technically, encoding can be decoupled from subsequent compression, but it can cause larger file sizes and worse io speed, as demonstrated by the benchmarks in issue #49715. Hence, I think encoding should take compression into account in order to get best results, even though it increases code complexity.
I'd be happy to put together an initial PR if this direction sounds reasonable. Happy to hear if there are considerations I'm missing.
daniel-x commentedon May 10, 2026
Please let me know if there is interest by other developers in me implementing something and/or if I should use another medium for the discussion, like the mailing list or something else.
kou commentedon May 10, 2026
Sorry for no response in a month.
Could you also try
[email protected]https://arrow.apache.org/community/#mailing-lists for visibility?