Skip to content

Commit 2273e33

Browse files
adriangbclaude
andcommitted
feat: report statistics, ordering and partitioning of materialized CTE scans
A `MaterializedCteScanExec` is a leaf, and the CTE body is a child of `MaterializedCteExec`, so the scan reported unknown statistics, no ordering and unknown partitioning. `JoinSelection` could then build a hash table on a large table and not on a small CTE, and the continuation could sort or repartition rows that the body already sorted or partitioned. The planner now copies the properties of the body to the scans in `optimize_physical_plan`: - Statistics: each scan gets the statistics of the body of the initial plan, before the optimizer runs. - Ordering and hash partitioning: after the plan is optimized, each scan of the initial plan gets the partition count, the output ordering and the hash partitioning of the optimized body, and the plan is optimized again. The second plan is used only if each body in it has the properties that its scans report. If not, the first plan is used, in which the scans report no ordering and no partitioning. A scan with the properties of the body replays one body partition per partition, and fails if the partition counts are not the same. Closes apache#26230. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 1a43b6c commit 2273e33

4 files changed

Lines changed: 1101 additions & 86 deletions

File tree

‎datafusion/core/src/datasource/view_test.rs‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -527,4 +527,58 @@ mod tests {
527527

528528
Ok(())
529529
}
530+
531+
/// The plan of a view that is not inlined is optimized two times: by the
532+
/// planning run of the view, and as a part of the outer plan. The scans
533+
/// of its CTE get the partitioning of the CTE body in both runs.
534+
#[tokio::test]
535+
async fn query_not_inlined_view_with_partitioned_materialized_cte() -> Result<()> {
536+
let session_ctx = SessionContext::new_with_config(
537+
SessionConfig::new()
538+
.with_target_partitions(4)
539+
.set_str("datafusion.sql_parser.dialect", "PostgreSQL"),
540+
);
541+
session_ctx
542+
.sql("CREATE TABLE t AS VALUES (1, 10), (1, 20), (2, 30)")
543+
.await?
544+
.collect()
545+
.await?;
546+
let plan = session_ctx
547+
.sql(
548+
"WITH r AS MATERIALIZED \
549+
(SELECT column1 AS g, sum(column2) AS s FROM t GROUP BY column1) \
550+
SELECT g, max(s) AS s FROM r GROUP BY g",
551+
)
552+
.await?
553+
.into_optimized_plan()?;
554+
let view = crate::datasource::ViewTable::new(plan, None);
555+
session_ctx.register_table("w", std::sync::Arc::new(NotInlinedView(view)))?;
556+
557+
let df = session_ctx
558+
.sql("SELECT x.g, x.s + y.s AS s FROM w x JOIN w y ON x.g = y.g ORDER BY g")
559+
.await?;
560+
let physical_plan = df.clone().create_physical_plan().await?;
561+
let displayed = crate::physical_plan::displayable(physical_plan.as_ref())
562+
.indent(true)
563+
.to_string();
564+
assert_eq!(
565+
displayed
566+
.matches("MaterializedCteScanExec: name=r, partitioning=Hash([g@0], 4)")
567+
.count(),
568+
2,
569+
"{displayed}"
570+
);
571+
572+
let results = df.collect().await?;
573+
insta::assert_snapshot!(batches_to_string(&results),@r"
574+
+---+----+
575+
| g | s |
576+
+---+----+
577+
| 1 | 60 |
578+
| 2 | 60 |
579+
+---+----+
580+
");
581+
582+
Ok(())
583+
}
530584
}

0 commit comments

Comments
 (0)