Skip to content

Commit 984f55e

Browse files
schenksjclaude
andcommitted
contrib(delta) P7f: 4.1.0.diff hunk for DeltaSuite partition-pruning test
Patch `DeltaSuite "query with predicates should skip partitions"` to also recognise `CometDeltaNativeScanExec` in its `executedPlan.collect { case f: FileSourceScanExec => f }`. Comet's planner replaces the underlying `FileSourceScanExec` with the contrib's native exec; without this hunk the collect returns an empty list and the test fails at `assert(fileScans.size == 1)`. The contrib's exec exposes the same `numFiles` metric (aliased to `total_files` over the post-prunePartitions task list), so once the test sees the exec the existing assertion on `numFiles == 1` validates that driver-side partition pruning is actually filtering AddFiles. Closes tasks apache#124 (partition pushdown validated) and apache#125 (data-stats pushdown validated via `DataSkippingDeltaV1Suite "loading data from Delta to parquet should skip data"`). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent abd9d95 commit 984f55e

1 file changed

Lines changed: 15 additions & 0 deletions

File tree

‎contrib/delta/dev/diffs/4.1.0.diff‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -215,3 +215,18 @@ index 93c7da9..755fae9 100644
215215
case _ => None
216216
}
217217
}
218+
diff --git a/spark/src/test/scala/org/apache/spark/sql/delta/DeltaSuite.scala b/spark/src/test/scala/org/apache/spark/sql/delta/DeltaSuite.scala
219+
--- a/spark/src/test/scala/org/apache/spark/sql/delta/DeltaSuite.scala
220+
+++ b/spark/src/test/scala/org/apache/spark/sql/delta/DeltaSuite.scala
221+
@@ -118,6 +118,11 @@
222+
val query = spark.read.format("delta").load(testPath).where("part = 1")
223+
val fileScans = query.queryExecution.executedPlan.collect {
224+
case f: FileSourceScanExec => f
225+
+ // Comet regression: Comet's planner replaces `FileSourceScanExec` with
226+
+ // `CometDeltaNativeScanExec`. Its `metrics` map aliases `numFiles` to the
227+
+ // post-pruning task count so this assertion exercises the same partition-skip
228+
+ // semantics from the Comet path.
229+
+ case c: org.apache.spark.sql.comet.CometDeltaNativeScanExec => c
230+
}
231+
232+
// Force the query to read files and generate metrics

0 commit comments

Comments
 (0)