Skip to content

Separate parquet::basic::Compression into two enums #9863

Open

Description

@etseidl

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The Compression enum in the parquet crate currently serves two purposes. First it is part of the Parquet thrift schema, and is used for informational purposes in the metadata. In this role, it is a stand-in for the actual Thrift CompressionCodec.

Its second purpose is as a configuration parameter to control the writing of Parquet pages. As such, it is desirable for it to carry extra information to fine tune the compression codecs in use. But as more information is added, the memory burden on the metadata increases.

Describe the solution you'd like
I propose splitting this enum into two. Compression can move to file::properties for use in configuration. A new CompressionCodec that can make use of the thrift macros would be created in basic.

Describe alternatives you've considered
To keep breaking changes to a minimum, Compression could be simply left in basic. Also ColumnChunkMetaData::compression and ColumnChunkMetaDataBuilder::set_compression could keep their current signatures and simply convert internally.

Additional context
This would be a breaking API change, but I think it make adding new features to the writer easier while simplifying the parsing and representation of the Parquet metadata.

I thought of this while evaluating #9807 and #9367

Activity

  1. etseidl commented on May 1, 2026

    @etseidl
    ContributorAuthor

    #9864 addresses the first part of this issue, but does so in a non-breaking way. Moving the Compression enum to file::properties can wait until doing so can be bundled with other breaking changes to that enum.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions