Skip to content

fix(reader): keep rows with a null equality-delete column value - #2781

Merged
blackmwk merged 3 commits into
apache:mainfrom
mbutrovich:comet-eqdel-repro
Jul 8, 2026
Merged

blackmwk merged 3 commits into
apache:mainfrom
mbutrovich:comet-eqdel-repro

Conversation

@mbutrovich

Copy link
Copy Markdown
Collaborator

Which issue does this PR close?

What changes are included in this PR?

Equality deletes are applied by building a predicate that keeps rows not matched by any delete row. For each equality column the keep term was col != v, produced by negating the equality match. Applied as a row filter, null != v evaluates to null, so a data row whose equality column is null was dropped. Per the spec (Equality Delete Files), a null value matches only a null delete value, so such rows should be kept.

This builds the keep predicate directly in parse_equality_deletes_record_batch_stream: col IS NULL OR col != v for a non-null delete value, and col IS NOT NULL for a null delete value. Rows with a null value in an equality column are no longer deleted.

Are these changes tested?

Yes, unit tests in caching_delete_file_loader covering the predicate construction:

  • test_equality_delete_predicate_preserves_null_rows: a non-null delete value on a nullable column produces (col IS NULL) OR (col != v).
  • test_equality_delete_predicate_matches_null_delete_value: a null delete value produces col IS NOT NULL.
  • test_equality_delete_predicate_multiple_columns: per-column keep terms are OR-ed.
  • test_equality_delete_predicate_multiple_delete_rows: per-row keep predicates are AND-ed.
  • test_delete_file_loader_parse_equality_deletes (existing, updated): the combined case with multiple columns, a required column, and null delete values.

@mbutrovich

mbutrovich commented Jul 7, 2026 •

Copy link
Copy Markdown
Collaborator Author

The tl;dr from the issue is that I found this while adding Iceberg 1.11 support to DataFusion Comet, and ran the new TestSparkReaderDeletes.testEqualityDeleteWithSchemaEvolution test there.

@mbutrovich mbutrovich self-assigned this Jul 7, 2026

@blackmwk blackmwk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mbutrovich for this fix!

@blackmwk
blackmwk merged commit 22c256e into apache:main Jul 8, 2026
21 checks passed
@mbutrovich
mbutrovich deleted the comet-eqdel-repro branch July 8, 2026 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Equality delete on a nullable column deletes rows whose value in that column is null

2 participants