Skip to content

Commit 7b6e780

Browse files
schenksjclaude
andcommitted
fix(delta): add synthesizedFilePartitions + patch DeltaSinkSuite/IdentityColumnAdmission
Two more regression failures fixed: #7. `IdentityColumnAdmissionScalaSuite.streaming` -- This is an UPSTREAM Delta test bug, NOT a Comet bug. Reproduces with Comet entirely disabled (plugin commented out). The test calls `MemoryStream.addData(1 to 10)` AFTER `start()` on a query with `Trigger.AvailableNow`. AvailableNow processes only data present at trigger time and exits before the late-arriving data can be consumed; the expected `StreamingQueryException` is never thrown. Patch the test diff to pre-populate the MemoryStream BEFORE `start()`. Worth filing upstream against delta-io/delta. #8. `DeltaSinkIdColumnMappingSuite "partitioned writing and batch reading - column mapping id mode"` -- The test inspects `executedPlan.collect[DataSourceScanExec]` and reads `inputRDDs.head.asInstanceOf[FileScanRDD].filePartitions`. Comet replaces the scan with `CometDeltaNativeScanExec` which uses `CometExecRDD`, not `FileScanRDD`. Add a public method `synthesizedFilePartitions` on `CometDeltaNativeScanExec` that builds an equivalent `Seq[FilePartition]` from the scan's task list (one PartitionedFile per task, with partition_values cast from the proto using `DeltaReflection.castPartitionString`). Patch the helper in the test diff to fall back to that accessor. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent 533893c commit 7b6e780

2 files changed

Lines changed: 78 additions & 0 deletions

File tree

‎dev/diffs/delta/3.3.2.diff‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -324,3 +324,44 @@ diff --git a/spark/src/test/scala-spark-3.5/shims/MergeIntoMetricsShims.scala b/
324324
+ // MergeIntoMetricsBase.scala explicitly anticipates 1 or 2 across versions.
325325
+ val DELETE_WITH_DUPLICATE_NUM_TARGET_FILES_ADDED_NON_PARTITIONED_NO_CDF = 2
326326
}
327+
328+
diff --git a/spark/src/test/scala/org/apache/spark/sql/delta/IdentityColumnAdmissionSuite.scala b/spark/src/test/scala/org/apache/spark/sql/delta/IdentityColumnAdmissionSuite.scala
329+
--- a/spark/src/test/scala/org/apache/spark/sql/delta/IdentityColumnAdmissionSuite.scala
330+
+++ b/spark/src/test/scala/org/apache/spark/sql/delta/IdentityColumnAdmissionSuite.scala
331+
@@ -271,6 +271,11 @@ trait IdentityColumnAdmissionSuiteBase
332+
withTempDir { checkpointDir =>
333+
val ex = intercept[StreamingQueryException] {
334+
val stream = MemoryStream[Int]
335+
+ // Comet regression: pre-populate MemoryStream BEFORE start. With
336+
+ // `Trigger.AvailableNow`, only data present at trigger time is processed;
337+
+ // data added after `start()` is not picked up before the query exits.
338+
+ // Without this the query exits with no rows and no expected exception.
339+
+ stream.addData(1 to 10)
340+
val q = stream
341+
.toDF
342+
.map(_ => Tuple2(1L, 1))
343+
@@ -281,7 +286,6 @@ trait IdentityColumnAdmissionSuiteBase
344+
.option("checkpointLocation", checkpointDir.getCanonicalPath)
345+
.trigger(Trigger.AvailableNow)
346+
.start(path)
347+
- stream.addData(1 to 10)
348+
q.processAllAvailable()
349+
q.stop()
350+
}
351+
352+
diff --git a/spark/src/test/scala/org/apache/spark/sql/delta/DeltaSinkSuite.scala b/spark/src/test/scala/org/apache/spark/sql/delta/DeltaSinkSuite.scala
353+
--- a/spark/src/test/scala/org/apache/spark/sql/delta/DeltaSinkSuite.scala
354+
+++ b/spark/src/test/scala/org/apache/spark/sql/delta/DeltaSinkSuite.scala
355+
@@ -259,6 +259,12 @@ class DeltaSinkSuite extends StreamTest with DeltaSQLCommandTest with DeltaSinkS
356+
val filePartitions = df.queryExecution.executedPlan.collect {
357+
case scan: DataSourceScanExec if scan.inputRDDs().head.isInstanceOf[FileScanRDD] =>
358+
scan.inputRDDs().head.asInstanceOf[FileScanRDD].filePartitions
359+
+ // Comet regression: Comet's planner replaces `DataSourceScanExec` with
360+
+ // `CometDeltaNativeScanExec`. The Comet exec exposes a synthesised
361+
+ // `FilePartition` list derived from its task list, with the same per-file
362+
+ // partition_values; the assertions below work the same way.
363+
+ case c: org.apache.spark.sql.comet.CometDeltaNativeScanExec =>
364+
+ c.synthesizedFilePartitions
365+
}.flatten
366+
if (filePartitions.isEmpty) {
367+
fail(s"No FileScan in query\n${df.queryExecution}")

‎spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,43 @@ case class CometDeltaNativeScanExec(
8282
@transient private lazy val allTasks: Seq[OperatorOuterClass.DeltaScanTask] =
8383
OperatorOuterClass.DeltaScanTaskList.parseFrom(taskListBytes).getTasksList.asScala.toSeq
8484

85+
/**
86+
* Synthesise a `Seq[FilePartition]` from this scan's tasks, with each task becoming one
87+
* `PartitionedFile` carrying its partition values as an `InternalRow`. Delta tests (e.g.
88+
* `DeltaSinkSuite`) inspect `executedPlan.collect[DataSourceScanExec]` and read
89+
* `inputRDDs.head.asInstanceOf[FileScanRDD].filePartitions` to verify partition pruning; those
90+
* tests find nothing under Comet because we replace the scan with this exec. The test diff in
91+
* `dev/diffs/delta/<version>.diff` patches the helper to fall back to this accessor, so the
92+
* same partition-pruning assertions pass against Comet's scan.
93+
*/
94+
def synthesizedFilePartitions: Seq[org.apache.spark.sql.execution.datasources.FilePartition] = {
95+
if (allTasks.isEmpty) return Nil
96+
val sessionTz = java.time.ZoneId.of(SQLConf.get.sessionLocalTimeZone)
97+
val files = allTasks.zipWithIndex.map { case (task, _) =>
98+
val pvRow = InternalRow.fromSeq(partitionSchema.fields.toSeq.map { f =>
99+
val proto = task.getPartitionValuesList.asScala.find(_.getName == f.name)
100+
val s = if (proto.exists(_.hasValue)) Some(proto.get.getValue) else None
101+
org.apache.comet.delta.DeltaReflection.castPartitionString(s, f.dataType, sessionTz)
102+
})
103+
val sparkPath =
104+
org.apache.spark.paths.SparkPath.fromUrlString(task.getFilePath)
105+
org.apache.spark.sql.execution.datasources.PartitionedFile(
106+
partitionValues = pvRow,
107+
filePath = sparkPath,
108+
start = if (task.hasByteRangeStart) task.getByteRangeStart else 0L,
109+
length = {
110+
if (task.hasByteRangeStart && task.hasByteRangeEnd) {
111+
task.getByteRangeEnd - task.getByteRangeStart
112+
} else task.getFileSize
113+
},
114+
modificationTime = 0L,
115+
fileSize = task.getFileSize)
116+
}
117+
files.zipWithIndex.map { case (pf, i) =>
118+
org.apache.spark.sql.execution.datasources.FilePartition(i, Array(pf))
119+
}
120+
}
121+
85122
/**
86123
* Build per-partition bytes from the current DPP-pruned task list. DPP filters that are still
87124
* `SubqueryAdaptiveBroadcastExec` placeholders at planning time materialise lazily once AQE

0 commit comments

Comments
 (0)