Skip to content

Commit 3310fd5

Browse files
authored
fix: report LowerEqual cardinality effect for CoalescePartitionsExec with fetch (#26051)
## Which issue does this PR close? - Closes #26050. ## Rationale for this change `CoalescePartitionsExec::cardinality_effect` returns `CardinalityEffect::Equal` even when `fetch` is set, although the operator can then produce fewer rows than its input. Code that relies on `cardinality_effect` gets the wrong answer, for example `PassthroughStatisticsProvider` reports the input row count and drops the fetch. ## What changes are included in this PR? - `CoalescePartitionsExec::cardinality_effect` returns `CardinalityEffect::LowerEqual` when `fetch` is set, as `SortExec` and `SortPreservingMergeExec` already do. ## What is the testing strategy for this PR? - New unit test for `cardinality_effect` with and without `fetch`. - New unit test showing that `PassthroughStatisticsProvider` no longer drops the fetch (`Exact(10)` instead of `Exact(1000)`). - `sqllogictest` and the core `physical_optimizer` integration tests pass with no plan changes. ## Are there any user-facing changes? No. ---- Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding.
1 parent 7833125 commit 3310fd5

2 files changed

Lines changed: 38 additions & 1 deletion

File tree

‎datafusion/physical-plan/src/coalesce_partitions.rs‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -304,7 +304,11 @@ impl ExecutionPlan for CoalescePartitionsExec {
304304
}
305305

306306
fn cardinality_effect(&self) -> CardinalityEffect {
307-
CardinalityEffect::Equal
307+
if self.fetch.is_none() {
308+
CardinalityEffect::Equal
309+
} else {
310+
CardinalityEffect::LowerEqual
311+
}
308312
}
309313

310314
/// Tries to swap `projection` with its input, which is known to be a
@@ -698,4 +702,21 @@ mod tests {
698702

699703
Ok(())
700704
}
705+
706+
#[test]
707+
fn test_cardinality_effect_with_fetch() {
708+
let input = test::scan_partitioned(4);
709+
710+
let coalesce = CoalescePartitionsExec::new(Arc::clone(&input));
711+
assert!(matches!(
712+
coalesce.cardinality_effect(),
713+
CardinalityEffect::Equal
714+
));
715+
716+
let coalesce = CoalescePartitionsExec::new(input).with_fetch(Some(10));
717+
assert!(matches!(
718+
coalesce.cardinality_effect(),
719+
CardinalityEffect::LowerEqual
720+
));
721+
}
701722
}

‎datafusion/physical-plan/src/operator_statistics/mod.rs‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1986,6 +1986,22 @@ mod tests {
19861986
Ok(())
19871987
}
19881988

1989+
#[test]
1990+
fn test_passthrough_skips_coalesce_partitions_with_fetch() -> Result<()> {
1991+
use crate::coalesce_partitions::CoalescePartitionsExec;
1992+
1993+
let registry = StatisticsRegistry::with_providers(vec![Arc::new(
1994+
PassthroughStatisticsProvider,
1995+
)]);
1996+
let source = make_source(1000);
1997+
let coalesce: Arc<dyn ExecutionPlan> =
1998+
Arc::new(CoalescePartitionsExec::new(source).with_fetch(Some(10)));
1999+
2000+
let stats = compute(&registry, coalesce.as_ref())?;
2001+
assert_eq!(stats.base.num_rows, Precision::Exact(10));
2002+
Ok(())
2003+
}
2004+
19892005
#[test]
19902006
fn test_chain_priority() -> Result<()> {
19912007
let mut registry = StatisticsRegistry::new();

0 commit comments

Comments
 (0)