Skip to content

Commit 234c2ce

Browse files
adriangbclaude
andcommitted
feat: release the output of a materialized CTE when the last scan finishes
The buffered output of a materialized CTE (the in-memory batches, their memory reservations and the spill files) was held until the `TaskContext` of the run and all replay streams were dropped. The output of a run now counts its readers. It is released when each partition of each bound scan was executed in the run, and each of these executions has finished or was dropped. The count is of the partitions that can still be executed, so the output is not released between two scans that are executed one after the other. A scan partition that is never executed keeps the output until the run ends, as before. A scan partition that is executed after the release computes the body again, so it never reads an empty output. A second execution of a plan with the same `TaskContext` is such a case. `RecursiveQueryExec` executes its recursive term one time for each iteration. It holds the outputs of the CTEs that the recursive term reads and does not declare, so that they are computed one time for the whole query. Closes apache#26233. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 71f28c3 commit 234c2ce

3 files changed

Lines changed: 542 additions & 54 deletions

File tree

‎datafusion/core/tests/sql/select.rs‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -593,3 +593,41 @@ async fn test_recursive_cte_batch_schema_stable_with_order_by_limit() -> Result<
593593
}
594594
Ok(())
595595
}
596+
597+
/// The output of a materialized CTE is released when its last scan finishes.
598+
/// The plan and the `TaskContext` of the run are still alive.
599+
#[tokio::test]
600+
async fn materialized_cte_output_is_released_after_the_last_scan() -> Result<()> {
601+
use futures::StreamExt;
602+
603+
// sqlparser only parses `[NOT] MATERIALIZED` for the PostgreSQL dialect.
604+
let config = SessionConfig::new()
605+
.with_target_partitions(4)
606+
.set_str("datafusion.sql_parser.dialect", "PostgreSQL");
607+
let ctx = SessionContext::new_with_config(config);
608+
// The CTE is read by the two inputs of a join and by a scalar subquery.
609+
let plan = ctx
610+
.sql(
611+
"WITH c AS MATERIALIZED (SELECT value AS v FROM generate_series(1, 20000)) \
612+
SELECT count(*) FROM c a JOIN c b ON a.v = b.v \
613+
WHERE a.v <= (SELECT max(v) FROM c)",
614+
)
615+
.await?
616+
.create_physical_plan()
617+
.await?;
618+
let displayed = displayable(plan.as_ref()).indent(true).to_string();
619+
assert_eq!(displayed.matches("MaterializedCteScanExec").count(), 3);
620+
621+
let task_ctx = ctx.task_ctx();
622+
let pool = Arc::clone(task_ctx.memory_pool());
623+
let mut stream =
624+
datafusion::physical_plan::execute_stream(Arc::clone(&plan), task_ctx.clone())?;
625+
let mut rows = 0;
626+
while let Some(batch) = stream.next().await.transpose()? {
627+
rows += batch.column(0).as_primitive::<Int64Type>().value(0);
628+
}
629+
assert_eq!(rows, 20000);
630+
assert_eq!(pool.reserved(), 0);
631+
drop((stream, plan, task_ctx));
632+
Ok(())
633+
}

0 commit comments

Comments
 (0)