Skip to content

Commit a83ecb3

Browse files
schenksjclaude
andcommitted
feat(delta): keep input_file_name() native via one-task-per-partition
Closes the input_file_name() / input_file_block_* fallback (apache#75 design A). Previously CometScanRule declined the entire Delta scan whenever the plan referenced these expressions, falling back to vanilla Spark+Delta. CometExecRDD.setInputFileForDeltaScan already sets InputFileBlockHolder, but only with the FIRST task's path -- correct only when 1 task per partition. We now: - Detect input_file_name() / input_file_block_* in the plan and tag the relation's options with comet.delta.needsInputFileName=true. - In CometDeltaNativeScan.convert: skip byte-range splitting in splitTasks when the tag is set. - In CometDeltaNativeScanExec.packTasks: emit one task per group when the constructor's oneTaskPerPartition flag is set. With 1 file per partition, setInputFileForDeltaScan attributes every row correctly. input_file_name() stays a JVM-evaluated expression (no native serde) in any Project above the Comet exec frontier. The propagation goes through relation.options rather than a ThreadLocal because CometScanRule (sets the flag) and CometExecRule (calls convert / createExec) run as separate plan-transform passes; a ThreadLocal would be cleared before convert/createExec ran. Verified: - DeletionVectorsSuite 293/293 pass (was passing via fallback; now via native one-task-per-partition path). - DeleteSQLSuite + UpdateSQLSuite 133/133 pass (these are the canonical input_file_name() consumers via findTouchedFiles). - Today's run produced zero "Native Delta scan is not compatible with input_file_name" fallback messages. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent 8588afe commit a83ecb3

3 files changed

Lines changed: 70 additions & 41 deletions

File tree

‎spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala‎

Lines changed: 34 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -349,37 +349,32 @@ case class CometScanRule(session: SparkSession)
349349
// at execution time (DPP expressions are filtered out of the
350350
// planning-time InterpretedPredicate and applied by Spark post-scan).
351351
if (DeltaReflection.isDeltaFileFormat(r.fileFormat)) {
352-
// Decline when the plan references `input_file_name()` /
353-
// `input_file_block_*`. Those expressions read from
354-
// `InputFileBlockHolder`, a thread-local set per file. Comet's
355-
// `CometExecRDD.setInputFileForDeltaScan` *does* populate it but
356-
// only with the FIRST task's path -- correct only if the partition
357-
// holds exactly one task. Both `splitTasks` (byte-range chunking)
358-
// and `CometDeltaNativeScanExec.packTasks` (bin-packing) can put
359-
// multiple tasks per partition, so the path would be wrong for
360-
// rows from other files. Breaks Delta UPDATE which finds touched
361-
// files via `select(input_file_name()).distinct()`.
362-
//
363-
// TODO(#75 design A): instead of declining the whole scan, propagate
364-
// a `needsInputFileName` flag into CometDeltaNativeScan.convert ->
365-
// splitTasks (skip byte-range chunking) and CometDeltaNativeScanExec
366-
// (skip packTasks). With 1 task per partition, setInputFileForDeltaScan
367-
// sets the correct path and `input_file_name()` evaluated JVM-side
368-
// (no native serde exists, so it stays in a Project layer above
369-
// Comet) returns the right value.
370-
if (plan.exists(node =>
371-
node.expressions.exists(_.exists {
372-
case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength =>
373-
true
374-
case _ => false
375-
}))) {
376-
return withInfo(
377-
scanExec,
378-
"Native Delta scan is not compatible with input_file_name, " +
379-
"input_file_block_start, or input_file_block_length")
380-
}
381-
return nativeDeltaScan(session, scanExec, r, hadoopConfOrNull = null)
382-
.getOrElse(scanExec)
352+
// #75 design A: when the plan references `input_file_name()` /
353+
// `input_file_block_*`, we stay on native but signal CometDeltaNativeScan
354+
// to disable byte-range chunking AND task bin-packing so each Spark
355+
// partition holds exactly one file. `CometExecRDD.setInputFileForDeltaScan`
356+
// then sets `InputFileBlockHolder` to that file's path; Spark evaluates
357+
// input_file_name() JVM-side (no native serde) in any Project above the
358+
// Comet exec frontier, against the per-partition thread-local. Without
359+
// this gating, byte-range chunks or bin-packed task groups would mis-
360+
// attribute rows to the FIRST task's path.
361+
val needsInputFileName = plan.exists(node =>
362+
node.expressions.exists(_.exists {
363+
case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength =>
364+
true
365+
case _ => false
366+
}))
367+
// Tag the relation so CometDeltaNativeScan.convert (which runs in a LATER
368+
// CometExecRule pass) can read the signal. Thread-locals cleared by this
369+
// rule would not survive across passes; embedding in options ensures it
370+
// travels with the scan node.
371+
val scanForDelta = if (needsInputFileName) {
372+
val taggedOptions = r.options + (CometScanRule.NeedsInputFileNameOption -> "true")
373+
val taggedRelation = r.copy(options = taggedOptions)(r.sparkSession)
374+
scanExec.copy(relation = taggedRelation)
375+
} else scanExec
376+
return nativeDeltaScan(session, scanForDelta, scanForDelta.relation,
377+
hadoopConfOrNull = null).getOrElse(scanForDelta)
383378
}
384379
// DPP fallback for non-Delta scans (DataFusion/Iceberg-compat paths
385380
// don't support DPP natively).
@@ -1478,6 +1473,14 @@ case class CometScanTypeChecker(scanImpl: String) extends DataTypeSupport with C
14781473

14791474
object CometScanRule extends Logging {
14801475

1476+
/**
1477+
* Synthetic option key set by CometScanRule on a Delta HadoopFsRelation when the surrounding
1478+
* plan references `input_file_name()` / `input_file_block_*`. Read by
1479+
* `CometDeltaNativeScan.convert` / `createExec` (a later rule pass) to disable byte-range
1480+
* chunking + task bin-packing so each Spark partition holds exactly one file. See #75 design A.
1481+
*/
1482+
val NeedsInputFileNameOption: String = "comet.delta.needsInputFileName"
1483+
14811484
/**
14821485
* Validating object store configs can cause requests to be made to S3 APIs (such as when
14831486
* resolving the region for a bucket). We use a cache to reduce the number of S3 calls.

‎spark/src/main/scala/org/apache/comet/serde/operator/CometDeltaNativeScan.scala‎

Lines changed: 23 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,20 @@ object CometDeltaNativeScan extends CometOperatorSerde[CometScanExec] with Loggi
6161
// during planning so a simple ThreadLocal is safe.
6262
private val lastTaskListBytes = new ThreadLocal[Array[Byte]]()
6363

64+
// #75 design A: when the surrounding plan references `input_file_name()` /
65+
// `input_file_block_*`, CometScanRule tags the relation's options with
66+
// `CometScanRule.NeedsInputFileNameOption`. We read it here to (a) skip
67+
// byte-range splitting in splitTasks and (b) emit `oneTaskPerPartition = true`
68+
// on the CometDeltaNativeScanExec so packTasks keeps each task in its own
69+
// partition. With 1 task per partition, `CometExecRDD.setInputFileForDeltaScan`
70+
// sets InputFileBlockHolder to the correct path and Spark's JVM-side
71+
// input_file_name() evaluation (no native serde exists) returns the right
72+
// value.
73+
private def scanNeedsInputFileName(scan: CometScanExec): Boolean =
74+
scan.relation.options
75+
.get(org.apache.comet.rules.CometScanRule.NeedsInputFileNameOption)
76+
.contains("true")
77+
6478
override def enabledConfig: Option[ConfigEntry[Boolean]] = Some(
6579
CometConf.COMET_DELTA_NATIVE_ENABLED)
6680

@@ -638,10 +652,12 @@ object CometDeltaNativeScan extends CometOperatorSerde[CometScanExec] with Loggi
638652
scan: CometScanExec,
639653
tasks: Seq[OperatorOuterClass.DeltaScanTask]): Seq[OperatorOuterClass.DeltaScanTask] = {
640654
if (tasks.isEmpty) return tasks
641-
// TODO(#75 design A): when the plan needs input_file_name() (signal threaded down
642-
// from CometScanRule), bail out here so each task remains 1:1 with a file. The
643-
// executor-side `CometDeltaNativeScanExec.packTasks` also needs to honour the
644-
// same flag.
655+
// #75 design A: when the plan needs input_file_name(), keep each task 1:1 with
656+
// a file so `setInputFileForDeltaScan` (which reads only the first task) sets
657+
// the correct path. Without this, byte-range chunking would create multiple
658+
// tasks for one file -- still same path -- BUT combined with packTasks below
659+
// could end up with multiple FILES per partition.
660+
if (scanNeedsInputFileName(scan)) return tasks
645661
val sizes = tasks.map(_.getFileSize)
646662
val msb = maxSplitBytes(scan, sizes)
647663
if (msb <= 0) return tasks
@@ -778,6 +794,7 @@ object CometDeltaNativeScan extends CometOperatorSerde[CometScanExec] with Loggi
778794
} finally {
779795
lastTaskListBytes.remove()
780796
}
797+
val oneTaskPerPartition = scanNeedsInputFileName(op)
781798

782799
val dppFilters = op.partitionFilters.filter(
783800
_.exists(_.isInstanceOf[org.apache.spark.sql.catalyst.expressions.PlanExpression[_]]))
@@ -791,6 +808,7 @@ object CometDeltaNativeScan extends CometOperatorSerde[CometScanExec] with Loggi
791808
tableRoot,
792809
tlBytes,
793810
dppFilters,
794-
partitionSchema)
811+
partitionSchema,
812+
oneTaskPerPartition = oneTaskPerPartition)
795813
}
796814
}

‎spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,14 @@ case class CometDeltaNativeScanExec(
6060
tableRoot: String,
6161
@transient taskListBytes: Array[Byte],
6262
@transient dppFilters: Seq[Expression] = Seq.empty,
63-
partitionSchema: StructType = new StructType())
63+
partitionSchema: StructType = new StructType(),
64+
/**
65+
* #75 design A: when true, `packTasks` emits one group (= one partition) per task so
66+
* `CometExecRDD.setInputFileForDeltaScan` correctly sets `InputFileBlockHolder` to the
67+
* partition's only file path. Set by `CometDeltaNativeScan.createExec` when the surrounding
68+
* plan references `input_file_name()` / `input_file_block_*`.
69+
*/
70+
oneTaskPerPartition: Boolean = false)
6471
extends CometLeafExec {
6572

6673
override val supportsColumnar: Boolean = true
@@ -141,12 +148,13 @@ case class CometDeltaNativeScanExec(
141148
}
142149
}
143150

144-
// TODO(#75 design A): when input_file_name() is needed (signal threaded from
145-
// CometScanRule via convert), return one-task-per-group so setInputFileForDeltaScan
146-
// correctly attributes the path. The CometScanRule TODO at the early-decline gate
147-
// documents the full design.
151+
// #75 design A: when input_file_name() is needed (signal threaded from CometScanRule
152+
// via CometDeltaNativeScan.createExec into `oneTaskPerPartition`), short-circuit
153+
// packing so each task gets its own partition. `setInputFileForDeltaScan` reads
154+
// task[0]'s path; with 1 task per partition that path correctly attributes every row.
148155
private def packTasks(tasks: Seq[OperatorOuterClass.DeltaScanTask])
149156
: Seq[Seq[OperatorOuterClass.DeltaScanTask]] = {
157+
if (oneTaskPerPartition) return tasks.map(t => Seq(t))
150158
val conf = originalPlan.relation.sparkSession.sessionState.conf
151159
val openCostInBytes = conf.filesOpenCostInBytes
152160
val maxPartitionBytes = conf.filesMaxPartitionBytes

0 commit comments

Comments
 (0)