Repository navigation
fix(rel): UNWIND of maps feeding two inline-map node patterns fails in optimize_projections #2084
Description
Defect
A read that unwinds a list of maps and uses two of the map's fields as inline node properties in one MATCH fails at planning time. It returns no rows and raises an optimizer error.
UNWIND [{a: 1, b: 2}] AS e
MATCH (a:N {id: e.a}), (b:N {id: e.b})
RETURN a.name, b.nameThe error:
Optimizer rule 'optimize_projections' failed ... No field named var_1.name.
Valid fields are __datafusion_extracted_1, var_1.id, var_2.id
With RETURN count(*), the same pattern fails with No field named var_2.node_id.
- It reproduces on in-memory and durable projects, on main
3b8eb2b3cand at47b2adf8c(before perf(rel): project only the node properties a query reads #2041), so it is pre-existing. - A single pattern (
MATCH (a:N {id: e.a})) passes, and so does a plain two-MATCHcount(*). __datafusion_extracted_1points at a DataFusion extraction or common-subexpression rule interacting with the two correlated inline-map scans.- A second shape, two separate MATCH clauses keyed off the unwound map, was reported while perf(rel): search a bound-end variable-length expansion from the bound end (#1984) #2083 was being implemented. Its exact text is still being confirmed.
Found during #1982. The introducing commit is being bisected, and the result will be added here.
Acceptance criteria
- The query above returns the correct rows (and
count(*)) on in-memory and durable projects, at 1 and 4 partitions. - The same holds for the two-clause form
UNWIND ... AS e MATCH (a:N {id: e.a}) MATCH (b:N {id: e.b}), and for the write form that creates a relationship betweenaandb. - A regression test for each shape fails before the fix.
- The root cause is fixed in our lowering or optimizer configuration, not by disabling
optimize_projectionsglobally. If the cause is upstream DataFusion, it is fixed in the vendored crate as for fix(exec): backport the DataFusion spill-pool deadlock fix (apache/datafusion#24891) into a vendored datafusion-physical-plan 54.1.0 (Part of #1985) #2046, with an upstream link.
Activity
DecisionNerd commentedon Oct 14, 2026
Root cause (pinned with a per-rule optimizer observer by another session on 2026-10-14):
The fault is in DataFusion 53/54's
extract_leaf_expressionsandpush_down_leaf_projectionsrule pair, not in GraphForge's demand code and not #2041. The lowered plan is correct and carriesvar_1.name,var_2.nameandnode_id.get_field(var_0,'a') = var_1.idandget_field(var_0,'b') = var_2.id, wherevar_0is the unwound map.extract_leaf_expressionsaliases bothget_fieldcalls as__datafusion_extracted_N.push_down_leaf_projectionsmoves both extraction projections onto the Unwind side.optimize_projectionsthen fails.Removing either rule makes optimization succeed.
count(*)andCREATEforms fail nondeterministically (4–7 of 8–12 runs on the same binary), which points at hash or iteration order inside the pushdown rule.0f5bf2227(2026-07-31), the oldest commit that compiles after the crate rename, so it has likely failed since the DataFusion 53 adoption. No GraphForge commit introduced it.Fix in progress on
fix/2084-unwind-inline-map-patterns: project every non-column join key into a named column on its input before the join, so no join key is a field-extraction expression. Both DataFusion rules stay enabled. The tests must hold under the nondeterminism, so each form is repeated or the absence of aget_fieldjoin key is asserted.test(api): cover two inline-map node patterns keyed off one unwound m…
fix(rel): project expression join keys to named columns before leaf e…
chore(deps): vendor datafusion-optimizer 54.1.0 unchanged (Part of #2084
chore(deps): wire the vendored datafusion-optimizer through [patch.cr…
fix(rel): recompute a join's schema when the optimizer rewrites its i…