Add ExistenceJoin support to Comet native execution

What is the problem the feature request solves?

Comet does not support ExistenceJoin, causing incorrect results for correlated IN subqueries combined with OR on Spark 4.0. Adding native ExistenceJoin support would allow Comet to handle these plans end-to-end and eliminate the mixed Spark/Comet execution that produces wrong results.

The following query returns incorrect results when Comet is enabled on Spark 4.0:

CREATE TEMPORARY VIEW t1(c1, c2) AS VALUES (0, 1), (1, 2);
CREATE TEMPORARY VIEW t2(c1, c2) AS VALUES (0, 2), (0, 3);

SELECT * FROM t1 WHERE
  c1 IN (SELECT count(*) + 1 FROM t2 WHERE t2.c1 = t1.c1) OR
  c2 IN (SELECT count(*) - 1 FROM t2 WHERE t2.c1 = t1.c1);

Expected: (0, 1) and (1, 2) (both rows match via different OR branches) Actual with Comet: (1, 2) only (row (0, 1) is dropped)

The same query with a single IN (no OR) produces correct results.

Describe the potential solution

No response

Additional context

This only reproduces on Spark 4.0 because:

in-count-bug.sql test only exists in Spark 4.0
Spark 4.0 uses a new decorrelation path (decorrelateInnerQueryEnabledForExistsIn = true by default) that produces DomainJoin + ExistenceJoin plan structures
Spark 3.5 uses the old decorrelation which has the "count bug" (different expected behavior)

  spark.sql("CREATE TEMP VIEW t1 AS SELECT * FROM VALUES (1,5),(2,-1),(3,0) AS t1(id,v)")
  spark.sql("CREATE TEMP VIEW t2 AS SELECT * FROM VALUES (1),(3) AS t2(id)")
  spark.sql("SELECT * FROM t1 WHERE v > 0 OR EXISTS (SELECT 1 FROM t2 WHERE t2.id = t1.id)")
       .explain(true) 

== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- Project [id#41, v#42]
   +- Filter ((v#42 > 0) OR exists#50)
      +- BroadcastHashJoin [id#41], [id#45], ExistenceJoin(exists#50), BuildRight, false
         :- LocalTableScan [id#41, v#42]
         +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [plan_id=123]
            +- Project [id#45]
               +- LocalTableScan [1#47, id#45]

In DataFusion potentially we can use LeftMark/RightMark joins, however we need to check if those follow full three-valued mark semantics

Navigation Menu

Repository navigation

Add ExistenceJoin support to Comet native execution #3881

What is the problem the feature request solves?

Describe the potential solution

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add ExistenceJoin support to Comet native execution #3881

Description

What is the problem the feature request solves?

Describe the potential solution

Additional context

Activity

comphead commented on May 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions