Skip to content

Native Iceberg writes: translate the Parquet settings that have a parquet-rs equivalent #6954

Open

Description

@andygrove

What is the problem the feature request solves?

Each of these settings makes a write fall back, but each has a direct parquet-rs equivalent:

  • parquet.enable.dictionary=false as a table or write property. iceberg-java reads it in Parquet.WriteBuilder and turns dictionary encoding off for every column. Native: WriterProperties::set_dictionary_enabled(false), skipping the DictionaryChooser.
  • parquet.columnindex.truncate.length, as a table property or in the writer's Hadoop configuration. iceberg-java's ParquetWriter reads it (default 64) for the column index. Native: set_column_index_truncate_length.
  • A zstd write.parquet.compression-level of 0, which zstd defines as its default level, 3. parquet-rs accepts 1 to 22, so translate 0 to 3 on the JVM side. Negative levels wait on Parquet: accept zstd compression level 0 and negative levels in ZstdLevel arrow-rs#11545.
  • Non-default write.parquet.row-group-check-min-record-count and row-group-check-max-record-count. iceberg-java uses them for its own row-group size checks and passes them to parquet-mr as the minimum and maximum row counts between page size checks. The one native dependency is FIRST_PAGE_SIZE_CHECK_ROWS in iceberg_dictionary.rs, which hard-codes the default minimum of 100 for the first page, the page that decides dictionary encoding; pass the table's value instead. Row-group and page boundaries are already accepted divergences.
  • write.parquet.stats-enabled.column.<col> (Iceberg 1.10+). iceberg-java turns off parquet-mr statistics for the column. Native: set_column_statistics_enabled(path, EnabledStatistics::None). Check that the committed manifest metrics still match, including the float and double NaN counts and bounds the native writer carries over.
  • write.parquet.page-version=v2 (Iceberg 1.11). Native: WriterVersion::PARQUET_2_0. parquet-rs and parquet-mr choose the same v2 fallback encodings (RLE for booleans, DELTA_BINARY_PACKED for int32 and int64, DELTA_BYTE_ARRAY for binary and fixed-length binary, PLAIN for float and double), and under v2 both dictionary-encode fixed-length binary, so DictionaryChooser has to consider fixed-length binary columns too.

Describe the potential solution

One setting at a time: vet the key in CometIcebergNativeWrite, translate it in IcebergWriteProtoTranslation and build_writer_properties, and add a test that writes through both writers and compares rows and readable_metrics, plus, for the encoding settings, each column chunk's encodings. Add a CometIcebergWriteDetectionSuite case, and update the user guide's eligibility table and its "Plans for each fallback" lists in the same change.

Additional context

Part of #5643 and the native write epic #5649.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions