You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Separate parquet::basic::Compression into two enums #9863
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The Compression enum in the parquet crate currently serves two purposes. First it is part of the Parquet thrift schema, and is used for informational purposes in the metadata. In this role, it is a stand-in for the actual Thrift CompressionCodec.
Its second purpose is as a configuration parameter to control the writing of Parquet pages. As such, it is desirable for it to carry extra information to fine tune the compression codecs in use. But as more information is added, the memory burden on the metadata increases.
Describe the solution you'd like
I propose splitting this enum into two. Compression can move to file::properties for use in configuration. A new CompressionCodec that can make use of the thrift macros would be created in basic.
Describe alternatives you've considered
To keep breaking changes to a minimum, Compression could be simply left in basic. Also ColumnChunkMetaData::compression and ColumnChunkMetaDataBuilder::set_compression could keep their current signatures and simply convert internally.
Additional context
This would be a breaking API change, but I think it make adding new features to the writer easier while simplifying the parsing and representation of the Parquet metadata.
I thought of this while evaluating #9807 and #9367
#9864 addresses the first part of this issue, but does so in a non-breaking way. Moving the Compression enum to file::properties can wait until doing so can be bundled with other breaking changes to that enum.
Activity
etseidl commentedon May 1, 2026
#9864 addresses the first part of this issue, but does so in a non-breaking way. Moving the
Compressionenum tofile::propertiescan wait until doing so can be bundled with other breaking changes to that enum.Add `CompressionCodec` Thrift enum for Parquet metadata (#9864)
Add `CompressionCodec` Thrift enum for Parquet metadata (apache#9864)