[EPIC] Deletion vector read support (deletion-vector-v1 Puffin blobs) #2792

Is your feature request related to a problem or challenge?

Iceberg format version 3 stores positional deletes as deletion vectors instead of position delete files. A deletion vector is a roaring bitmap of deleted row positions, written as a deletion-vector-v1 Puffin blob and referenced from a delete manifest entry by content_offset and content_size_in_bytes.

iceberg-rust does not apply them on read. CachingDeleteFileLoader::load_file_for_task branches only on DataContentType (PositionDeletes / EqualityDeletes / Data), and the PositionDeletes arm unconditionally calls parquet_to_batch_stream(...) on the file path (crates/iceberg/src/arrow/caching_delete_file_loader.rs). A V3 deletion vector arrives as PositionDeletes content but points at a Puffin file, so the read fails when the loader parses that file as Parquet. The DV branch is stubbed: DeleteFileContext carries // TODO: Delete Vector loader from Puffin files (caching_delete_file_loader.rs:54). Any V3 table with positional deletes is therefore unreadable.

This issue tracks the read path under the V3 epic #2411, scoped to land independently of the DV write path.

What already exists on main

• DataFile parses the V3 manifest fields referenced_data_file, content_offset, content_size_in_bytes (crates/iceberg/src/spec/manifest/data_file.rs).
• DeleteVector (a RoaringTreemap wrapper) and the ParsedDeleteFileContext::DelVecs loader variant exist (crates/iceberg/src/delete_vector.rs, caching_delete_file_loader.rs).
• Positional deletes already flow DeleteVector -> DeleteFilter -> ArrowReader, so DV application reuses that path unchanged.
• PuffinReader, PuffinWriter, and the DELETION_VECTOR_V1 blob-type constant exist (crates/iceberg/src/puffin/).

Describe the solution you'd like

Implement DV read as a series of small PRs that depend only on each other, not on the write path (#2678 / #2203). DV writing is out of scope here; it belongs with #2678 under #2411.

Status: all tasks below are implemented on the dv-read branch and validated end to end against Spark / Iceberg-Java-written deletion vectors via a draft DataFusion Comet PR (apache/datafusion-comet#4887), which reads real V3 merge-on-read tables through this code and confirms the deleted rows match Spark. They are now being peeled into small, independently reviewable PRs, tracked per task below. The codec PR supersedes #2414, which has been idle since review feedback.

Task list

• Task 1: DV blob codec (decode). Parse a deletion-vector-v1 blob into deleted positions. No scan or reader API changes, so it merges first. PR: feat(delete-vector): [1/N] decode deletion-vector-v1 puffin blobs #2866 (supersedes feat(delete-vector): parse deletion-vector-v1 puffin blob - read support #2414).
• Task 2: Carry DV coordinates on the scan task. Add content_offset, content_size_in_bytes, and referenced_data_file to FileScanTaskDeleteFile (crates/iceberg/src/scan/task.rs), populated from DataFile during scan planning. This matches Iceberg-Java, where a DV is a DeleteFile (content=POSITION_DELETES, format=PUFFIN) attached to the task via FileScanTask.deletes(). PR: feat(scan): [2/N] carry deletion-vector coordinates on FileScanTaskDeleteFile #2868.
• Task 3: Loader branch, apply rules, end-to-end read. In load_file_for_task, route entries with content_offset.is_some() to a DV path: read the blob at content_offset / content_size_in_bytes via PuffinReader, decode with Task 1, and return ParsedDeleteFileContext::DelVecs. Apply the spec rules below (DeleteFileIndex supersede logic as in feat(scan): apply V3 deletion vectors on read (stacked on #2678) #2681, decoupled from write). Validate that the decoded cardinality equals record_count and the blob length equals content_size_in_bytes. Add an end-to-end test reading a V3 table whose deletes are DVs. PR: feat(scan): [3/N] read and apply V3 deletion vectors #3035.

Two read-path rules from the table spec must hold. A DV applies to a data file when the file path equals the DV's referenced_data_file, the file's data sequence number is <= the DV's, and the partitions match. When a DV applies to a data file, readers ignore any position delete files that would otherwise match it, because the DV subsumes them. At most one DV exists per data file per snapshot.

References

• Iceberg table spec format/spec.md, "Deletion Vectors": semantics, at-most-one-per-data-file, DV-subsumes-position-deletes, apply-scope rules, and the referenced_data_file / content_offset / content_size_in_bytes fields (required for DVs, must match the Puffin footer).
• Iceberg Puffin spec format/puffin-spec.md: the deletion-vector-v1 blob byte layout.
• Iceberg-Java: core/.../deletes/BitmapPositionDeleteIndex.java (framing, length, magic, CRC) and core/.../deletes/RoaringPositionBitmap.java (portable 64-bit layout).

Willingness to contribute

I can contribute to this feature independently.