Skip to content

Commit 8588afe

Browse files
schenksjclaude
andcommitted
docs(delta): inline TODOs for apache#75 input_file_name native support design A
The early-decline gate in CometScanRule, splitTasks in CometDeltaNativeScan, and packTasks in CometDeltaNativeScanExec are the three touch points that need a coordinated change to support input_file_name() natively without falling back to vanilla Spark+Delta. Each gets a pointer to the others so the next session can implement the flag-threading end-to-end in one pass. Behaviour unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent 779eeef commit 8588afe

3 files changed

Lines changed: 24 additions & 4 deletions

File tree

‎spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -351,10 +351,22 @@ case class CometScanRule(session: SparkSession)
351351
if (DeltaReflection.isDeltaFileFormat(r.fileFormat)) {
352352
// Decline when the plan references `input_file_name()` /
353353
// `input_file_block_*`. Those expressions read from
354-
// `InputFileBlockHolder`, a thread-local that Spark's `FileScanRDD`
355-
// sets per file. Comet's `CometExecRDD` doesn't populate it, so the
356-
// expressions return empty, breaking commands like Delta UPDATE
357-
// which find touched files via `select(input_file_name()).distinct()`.
354+
// `InputFileBlockHolder`, a thread-local set per file. Comet's
355+
// `CometExecRDD.setInputFileForDeltaScan` *does* populate it but
356+
// only with the FIRST task's path -- correct only if the partition
357+
// holds exactly one task. Both `splitTasks` (byte-range chunking)
358+
// and `CometDeltaNativeScanExec.packTasks` (bin-packing) can put
359+
// multiple tasks per partition, so the path would be wrong for
360+
// rows from other files. Breaks Delta UPDATE which finds touched
361+
// files via `select(input_file_name()).distinct()`.
362+
//
363+
// TODO(#75 design A): instead of declining the whole scan, propagate
364+
// a `needsInputFileName` flag into CometDeltaNativeScan.convert ->
365+
// splitTasks (skip byte-range chunking) and CometDeltaNativeScanExec
366+
// (skip packTasks). With 1 task per partition, setInputFileForDeltaScan
367+
// sets the correct path and `input_file_name()` evaluated JVM-side
368+
// (no native serde exists, so it stays in a Project layer above
369+
// Comet) returns the right value.
358370
if (plan.exists(node =>
359371
node.expressions.exists(_.exists {
360372
case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength =>

‎spark/src/main/scala/org/apache/comet/serde/operator/CometDeltaNativeScan.scala‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -638,6 +638,10 @@ object CometDeltaNativeScan extends CometOperatorSerde[CometScanExec] with Loggi
638638
scan: CometScanExec,
639639
tasks: Seq[OperatorOuterClass.DeltaScanTask]): Seq[OperatorOuterClass.DeltaScanTask] = {
640640
if (tasks.isEmpty) return tasks
641+
// TODO(#75 design A): when the plan needs input_file_name() (signal threaded down
642+
// from CometScanRule), bail out here so each task remains 1:1 with a file. The
643+
// executor-side `CometDeltaNativeScanExec.packTasks` also needs to honour the
644+
// same flag.
641645
val sizes = tasks.map(_.getFileSize)
642646
val msb = maxSplitBytes(scan, sizes)
643647
if (msb <= 0) return tasks

‎spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,10 @@ case class CometDeltaNativeScanExec(
141141
}
142142
}
143143

144+
// TODO(#75 design A): when input_file_name() is needed (signal threaded from
145+
// CometScanRule via convert), return one-task-per-group so setInputFileForDeltaScan
146+
// correctly attributes the path. The CometScanRule TODO at the early-decline gate
147+
// documents the full design.
144148
private def packTasks(tasks: Seq[OperatorOuterClass.DeltaScanTask])
145149
: Seq[Seq[OperatorOuterClass.DeltaScanTask]] = {
146150
val conf = originalPlan.relation.sparkSession.sessionState.conf

0 commit comments

Comments
 (0)