Repository navigation
Open
Labels
area:Icebergarea:writerNative Parquet writerNative Parquet writerenhancementNew feature or requestNew feature or request
Description
What is the problem the feature request solves?
Each of these settings makes a write fall back, but each has a direct parquet-rs equivalent:
parquet.enable.dictionary=falseas a table or write property. iceberg-java reads it inParquet.WriteBuilderand turns dictionary encoding off for every column. Native:WriterProperties::set_dictionary_enabled(false), skipping theDictionaryChooser.parquet.columnindex.truncate.length, as a table property or in the writer's Hadoop configuration. iceberg-java'sParquetWriterreads it (default 64) for the column index. Native:set_column_index_truncate_length.- A zstd
write.parquet.compression-levelof0, which zstd defines as its default level, 3. parquet-rs accepts 1 to 22, so translate 0 to 3 on the JVM side. Negative levels wait on Parquet: accept zstd compression level 0 and negative levels inZstdLevelarrow-rs#11545. - Non-default
write.parquet.row-group-check-min-record-countandrow-group-check-max-record-count. iceberg-java uses them for its own row-group size checks and passes them to parquet-mr as the minimum and maximum row counts between page size checks. The one native dependency isFIRST_PAGE_SIZE_CHECK_ROWSiniceberg_dictionary.rs, which hard-codes the default minimum of 100 for the first page, the page that decides dictionary encoding; pass the table's value instead. Row-group and page boundaries are already accepted divergences. write.parquet.stats-enabled.column.<col>(Iceberg 1.10+). iceberg-java turns off parquet-mr statistics for the column. Native:set_column_statistics_enabled(path, EnabledStatistics::None). Check that the committed manifest metrics still match, including the float and double NaN counts and bounds the native writer carries over.write.parquet.page-version=v2(Iceberg 1.11). Native:WriterVersion::PARQUET_2_0. parquet-rs and parquet-mr choose the same v2 fallback encodings (RLEfor booleans,DELTA_BINARY_PACKEDfor int32 and int64,DELTA_BYTE_ARRAYfor binary and fixed-length binary,PLAINfor float and double), and under v2 both dictionary-encode fixed-length binary, soDictionaryChooserhas to consider fixed-length binary columns too.
Describe the potential solution
One setting at a time: vet the key in CometIcebergNativeWrite, translate it in IcebergWriteProtoTranslation and build_writer_properties, and add a test that writes through both writers and compares rows and readable_metrics, plus, for the encoding settings, each column chunk's encodings. Add a CometIcebergWriteDetectionSuite case, and update the user guide's eligibility table and its "Plans for each fallback" lists in the same change.
Additional context
Metadata
Metadata
Assignees
Labels
area:Icebergarea:writerNative Parquet writerNative Parquet writerenhancementNew feature or requestNew feature or request
Activity