ORC-1898: When column is all null, NULL_SAFE_EQUALS pushdown doesn't get evaluated correctly

jayhan94 · dongjoon-hyun · commit 467a293bdc8b · 2025-05-20T17:19:15.000-07:00
### What changes were proposed in this pull request? When all values in column `col_0` are `NULL`s within a row group, and we attempt to apply the predicate pushdown `col_0 <=> 'xxx'`, the `evaluatePredicateProto` function returns `TruthValue.NULL`. In this case, we can directly determine the result based on the literal value: if the literal is `NULL`, return `TruthValue.YES`, otherwise, return `TruthValue.NO`. ### Why are the changes needed? See [SPARK-52032](https://issues.apache.org/jira/projects/SPARK/issues/SPARK-52032). When we pushdown the NULL_SAFE_EQUALS predicate, all values of the column are `NULL`. The `evaluatePredicateProto` returns `TruthValue.NULL`, whose `isNeeded` returns false so that the whole row group is skipped by `SargApplier.pickRowGroups`, which actually is incorrect. ### How was this patch tested? There already exists unit test -- `TestOrcTimezonePPD.testTimestampAllNulls` ### Was this patch authored or co-authored using generative AI tooling? Co-authored using generative AI tooling. Closes #2223 from jayhan94/fix_null_safe_equals_pred_push. Authored-by: Jay Han <feynmanhan94@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
diff --git a/java/core/src/java/org/apache/orc/impl/RecordReaderImpl.java b/java/core/src/java/org/apache/orc/impl/RecordReaderImpl.java
@@ -763,6 +763,13 @@ static TruthValue evaluatePredicateRange(PredicateLeaf predicate,
     if (!range.hasValues()) {
       if (predicate.getOperator() == PredicateLeaf.Operator.IS_NULL) {
         return TruthValue.YES;
+      } else if (predicate.getOperator() == PredicateLeaf.Operator.NULL_SAFE_EQUALS) {
+        Object literal = predicate.getLiteral();
+        if (literal == null) {
+          return TruthValue.YES;
+        } else {
+          return TruthValue.NO;
+        }
       } else {
         return TruthValue.NULL;
       }
diff --git a/java/core/src/test/org/apache/orc/TestOrcTimezonePPD.java b/java/core/src/test/org/apache/orc/TestOrcTimezonePPD.java
@@ -387,7 +387,7 @@ public void testTimestampAllNulls(String writerTimeZone, String readerTimeZone)
     PredicateLeaf pred = createPredicateLeaf(
       PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.TIMESTAMP, "x",
       Timestamp.valueOf("2007-08-01 00:00:00.0"), null);
-    assertEquals(SearchArgument.TruthValue.NULL, RecordReaderImpl.evaluatePredicate(colStats[1], pred, bf));
+    assertEquals(SearchArgument.TruthValue.NO, RecordReaderImpl.evaluatePredicate(colStats[1], pred, bf));
 
     pred = createPredicateLeaf(PredicateLeaf.Operator.IS_NULL, PredicateLeaf.Type.TIMESTAMP, "x", null, null);
     assertEquals(SearchArgument.TruthValue.YES, RecordReaderImpl.evaluatePredicate(colStats[1], pred, bf));
