[Bug] Negated predicates match NULL rows, so delete_by_predicate deletes them #1143

Search before asking

• I searched in the issues and found nothing similar.

Please describe the bug 🐞

Predicate::negate wraps most predicates in Predicate::Not, and Not is evaluated as plain !inner. So if the column is NULL, NOT(p = 'a') comes out true. In Java, equal(p, 'a').negate() becomes NotEqual(p, a), which is false for NULL.

So reads return the NULL rows, and TableUpdate::delete_by_predicate deletes them. Using not_equal directly works fine.

Repro on main (9568f39) with the fixture from table_predicate_delete_test.rs (id 3 has p = NULL):

let pred = Predicate::negate(pb.equal("p", Datum::String("a".into()))?);
let msgs = table.new_write_builder().new_update()?.delete_by_predicate(Some(pred)).await?;
// after commit: remaining ids [0, 1], expected [0, 1, 3]

The deletes fail the same way on the partition-only path and on the DV path. I checked Java with paimon-bundle 2.1.0: NOT(p='a'), NOT(v=10 AND p='b') and NOT(v=10) all return false for the NULL row.

DataFusion SQL isn't affected because NOT is pushed down with requires_residual, and the Python binding rejects not. Go Predicate.Not() and the C API both go through negate, though.

Solution

Do what Java does in negate: flip the leaf op (=/<>, </>=, IS NULL/IS NOT NULL, IN/NOT IN, ...) and push through AND/OR with De Morgan. For ops Java can't negate (starts_with etc.), Not would need NULL-aware evaluation, or deletes should reject it.

Are you willing to submit a PR?

• I'm willing to submit a PR!