Skip to content

Commit 61a82b2

Browse files
committed
feat: report the ordering and partitioning of materialized CTE scans
A `MaterializedCteScanExec` reported no ordering and unknown partitioning, so the continuation sorted or repartitioned rows that the CTE body already sorted or partitioned. `optimize_physical_plan` now optimizes each CTE body on its own first, as the child of a `MaterializedCteExec` with an empty continuation. A body that reads another CTE is optimized after that CTE. Each scan then gets the partition count, the output ordering and the hash partitioning of its optimized body, and the full plan is optimized one time. The result is used only if each body in it still has the properties that its scans report. If not, the plan with statistics only is optimized and used. A scan with the properties of the body replays one body partition per partition, and returns an internal error if the partition counts are not the same at execution. Closes apache#26424.
1 parent db48d4e commit 61a82b2

4 files changed

Lines changed: 1025 additions & 62 deletions

File tree

‎datafusion/core/src/datasource/view_test.rs‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -527,4 +527,58 @@ mod tests {
527527

528528
Ok(())
529529
}
530+
531+
/// The plan of a view that is not inlined is optimized two times: by the
532+
/// planning run of the view, and as a part of the outer plan. The scans
533+
/// of its CTE get the partitioning of the CTE body in both runs.
534+
#[tokio::test]
535+
async fn query_not_inlined_view_with_partitioned_materialized_cte() -> Result<()> {
536+
let session_ctx = SessionContext::new_with_config(
537+
SessionConfig::new()
538+
.with_target_partitions(4)
539+
.set_str("datafusion.sql_parser.dialect", "PostgreSQL"),
540+
);
541+
session_ctx
542+
.sql("CREATE TABLE t AS VALUES (1, 10), (1, 20), (2, 30)")
543+
.await?
544+
.collect()
545+
.await?;
546+
let plan = session_ctx
547+
.sql(
548+
"WITH r AS MATERIALIZED \
549+
(SELECT column1 AS g, sum(column2) AS s FROM t GROUP BY column1) \
550+
SELECT g, max(s) AS s FROM r GROUP BY g",
551+
)
552+
.await?
553+
.into_optimized_plan()?;
554+
let view = crate::datasource::ViewTable::new(plan, None);
555+
session_ctx.register_table("w", std::sync::Arc::new(NotInlinedView(view)))?;
556+
557+
let df = session_ctx
558+
.sql("SELECT x.g, x.s + y.s AS s FROM w x JOIN w y ON x.g = y.g ORDER BY g")
559+
.await?;
560+
let physical_plan = df.clone().create_physical_plan().await?;
561+
let displayed = crate::physical_plan::displayable(physical_plan.as_ref())
562+
.indent(true)
563+
.to_string();
564+
assert_eq!(
565+
displayed
566+
.matches("MaterializedCteScanExec: name=r, partitioning=Hash([g@0], 4)")
567+
.count(),
568+
2,
569+
"{displayed}"
570+
);
571+
572+
let results = df.collect().await?;
573+
insta::assert_snapshot!(batches_to_string(&results),@r"
574+
+---+----+
575+
| g | s |
576+
+---+----+
577+
| 1 | 60 |
578+
| 2 | 60 |
579+
+---+----+
580+
");
581+
582+
Ok(())
583+
}
530584
}

0 commit comments

Comments
 (0)