Skip to content

Parquet Writer Ignores "max statistics size" specification in WriterProperties #2033

@alamb

Description

@alamb

Describe the bug

WriterProperties::max_statistics_size is ignored in the writer

https://docs.rs/parquet/17.0.0/parquet/file/properties/struct.WriterProperties.html#method.max_statistics_size

To Reproduce
Set the stats size to 1 (byte) and notice that statistics are still happily created

Expected behavior
The statistics size limit should be respected (and documented more carefully -- like is it the total size of all statistics? If the limit is exceeded will partial statistics have been written, etc). Looking at the java or C++ parquet writer for inspiration is likely a good idea.

Additional context

In #2022 @tustvold fixed the writer to respect the "do/don't compute stats", however the "max size of computed statistics" setting is still ignored

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugparquetChanges to the parquet crate

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions