Skip to content

Adopt a pin-bump policy for the iceberg-rust dependency #5645

Open

Description

@andygrove

What is the problem the feature request solves?

native/Cargo.toml pins iceberg and iceberg-storage-opendal to a git revision (3d84c81353b1b23b6e4ae8eea8f8a021cc6927a7) rather than a release. The pin is what the native Iceberg scan and writer are tested against, but nothing decides when it moves, and #5636 is the first bug caused by it drifting behind an upstream fix (partition-path escaping, apache/iceberg-rust#2875). With the writer now producing files that iceberg-java reads back, a stale pin can turn an upstream bug fix into a Comet correctness bug.

Describe the potential solution

Write down and follow a policy, for example:

  • Bump the pin at least once per Comet release cycle, and immediately when an upstream fix affects file bytes, manifest metadata, or partition layout.
  • Prefer tagged iceberg-rust releases over arbitrary revisions once a release contains everything Comet needs, so the version is visible in Cargo.lock and in the docs.
  • Each bump runs the Iceberg suites on all four Spark/Iceberg profiles plus the manifest-metrics parity tests, since those are what catch behavior changes in the writer.
  • Track "waiting on upstream" items (see the eligibility-restrictions issue) so a bump can also lift restrictions.
  • Note the bump in the release notes, since users reading Comet-written tables with iceberg-java care which writer semantics they got.

Additional context

Part of the native Iceberg writes epic, #5649. Related: #5636.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions