Query parsed to LogicalJoin changed to LogicalFilter causing ClassCastException then CannotPlanException in newer versions

We are trying to upgrade Apache Calcite version in our project (Apache Beam https://github.com/apache/beam/pull/35588). Found a potential breaking change.

Prior to Calcite <=1.33, the following query works for the test

"select * from CUSTOMER "
+ " where exists ( "
+ " select * from ORDERS "
+ " where o_custkey = c_custkey )";

 
However, since Calcite 1.34 (to 1.38), the following error is observed
 

java.lang.RuntimeException: Error while applying rule FilterToCalcRule, args [rel#55:LogicalFilter.NONE(input=RelSubset#54,condition=EXISTS(
Unknown macro: { LogicalFilter(condition=[=($1, $cor0.c_custkey)])   BeamIOSourceRel(table=[[beam, ORDERS]]) }
),variablesSet=[$cor0])]

at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:250)

at org.apache.calcite.plan.volcano.IterativeRuleDriver.drive(IterativeRuleDriver.java:59)

at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:523)

at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:317)

at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:385)

at org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:210)

Caused by: java.lang.ClassCastException: class org.apache.calcite.rex.RexSubQuery cannot be cast to class org.apache.calcite.rex.RexLocalRef (org.apache.calcite.rex.RexSubQuery and org.apache.calcite.rex.RexLocalRef are in unnamed module of loader 'app')

at org.apache.calcite.rex.RexProgramBuilder.registerInput(RexProgramBuilder.java:304)

It is appeared due to CALCITE-6874, which is fixed in Calcite 1.39. However the fix does not fix our use case In 1.39/1.40, a different error is seen:

org.apache.beam.sdk.extensions.sql.impl.SqlConversionException: Unable to convert query select * from CUSTOMER  where exists (  select * from ORDERS  where o_custkey = c_custkey )

at app//org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:214)

at app//org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:116)

 

Caused by: org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are not enough rules to produce a node with desired properties: convention=BEAM_LOGICAL.

Missing conversion is LogicalFilter[convention: NONE -> BEAM_LOGICAL]

at app//org.apache.calcite.plan.volcano.RelSubset$CheapestPlanReplacer.visit(RelSubset.java:718)

at app//org.apache.calcite.plan.volcano.RelSubset.buildCheapestPlan(RelSubset.java:391)

at app//org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:535)

at app//org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:353)

at app//org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:385)

at app//org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:210)

... 48 more

Taking a closer look, In Calcite<=1.33 the query is parsed (planner.parse)

as

LogicalProject(c_custkey=[$0], c_acctbal=[$1], c_city=[$2])

  LogicalProject(c_custkey=[$0], c_acctbal=[$1], c_city=[$2], o_custkey=[CAST($3):INTEGER], $f1=[CAST($4):BOOLEAN])

    LogicalJoin(condition=[=($0, $3)], joinType=[inner])

      BeamIOSourceRel(table=[[beam, CUSTOMER]])

      LogicalAggregate(group=[

Unknown macro: {0}

], agg#0=[MIN($1)])

        LogicalProject(o_custkey=[$1], $f0=[true])

          BeamIOSourceRel(table=[[beam, ORDERS]])

Then send to planner.rel(). After Calcite 1.34, the query parsed differently

LogicalProject(c_custkey=[$0], c_acctbal=[$1], c_city=[$2]) LogicalFilter(condition=[EXISTS(

Unknown macro: { LogicalFilter(condition=[=($1, $cor0.c_custkey)]) BeamIOSourceRel(table=[[beam, ORDERS]]) }

)], variablesSet=[[$cor0]]) BeamIOSourceRel(table=[[beam, CUSTOMER]])

such that a LogicalFilter no longer converted to LogicalJoin. Then later on planner.rel() fails.

What might have caused this change (between Calcite 1.33 -> 1.34). This is currently blocking our upgrade. Is it possible to mitigate this without implementing a custom "LogicalFilter[convention: NONE -> BEAM_LOGICAL]" rule such that Calcite still knows to apply FilterToJoinRule?