Skip to content

Commit 16780c0

Browse files
schenksjclaude
andcommitted
feat(delta): DML rewrites synthesize via kernel-enumerate + path-filter (apache#76 Stage B option a)
MERGE / UPDATE / DELETE rewrites (TahoeBatchFileIndex) now get kernel's per-file transforms -- so they synthesize in-worker like regular reads, no DeltaSyntheticColumnsExec. The driver enumerates the pinned snapshot via kernel (with the synthesize read schema: row_index/row_id metadata columns) and FILTERS the result to the touched AddFile path set (touched files are a subset of the snapshot). Matching is exact string equality on the resolved file_path -- both sides resolve identically (table_root + rel, or pass-through for scheme'd paths). Safety: if ANY touched file isn't found in kernel's enumeration -- or enumeration throws -- decline to the legacy buildTaskListFromAddFiles + DeltaSyntheticColumnsExec path (never wrong files). Validated: the only declines in the full suite are 2 OptimizeJob-internal edge cases (CM-id materialised row_commit_version, which kernel genuinely can't express; and a file-not- found race during concurrent OPTIMIZE) -- both correctly fall back. - DeltaReflection.isDmlRewriteFileIndex: TahoeBatchFileIndex only (CDC family excluded -- it references files outside the snapshot, kept on the legacy path). - Empty predicate for the DML enumeration: the AddFile set is the authoritative selection; kernel stats pruning could drop a touched file and force a needless decline. Full contrib Delta suite: 148 passed, 0 failed (synthesize on for regular + DML reads). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1 parent ffdda0f commit 16780c0

2 files changed

Lines changed: 91 additions & 13 deletions

File tree

‎contrib/delta/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala‎

Lines changed: 80 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -645,12 +645,14 @@ object CometDeltaNativeScan extends CometOperatorSerde[CometDeltaScanMarker] wit
645645

646646
// Stage B/C: produce synthetic columns inside DeltaKernelScanExec (kernel metadata columns +
647647
// engine-side DV-invert / per-file constants), retiring the stacked DeltaSyntheticColumnsExec.
648-
// Only on the kernel-enumeration read path (NOT the AddFiles/subset path), which ships kernel's
649-
// per-file transform (needed for row_id GenerateRowId + partition injection). Config-gated while
650-
// validated.
651-
val synthesizeInWorker: Boolean =
652-
DeltaConf.COMET_DELTA_SYNTHESIZE_IN_WORKER_ENABLED.get(scan.conf) &&
653-
!DeltaReflection.isSubsetFileIndex(relation.location)
648+
// Needs kernel's per-file transform (row_id GenerateRowId + partition injection), which only the
649+
// kernel-enumeration path ships. Regular reads always take it; DML rewrites (TahoeBatchFileIndex)
650+
// get it via kernel-enumerate + path-filter (set below, only if every touched file matched).
651+
// Config-gated while validated.
652+
val synthesizeConfig: Boolean =
653+
DeltaConf.COMET_DELTA_SYNTHESIZE_IN_WORKER_ENABLED.get(scan.conf)
654+
var synthesizeInWorker: Boolean =
655+
synthesizeConfig && !DeltaReflection.isSubsetFileIndex(relation.location)
654656

655657
// Column name list for resolving BoundReference indices to kernel column
656658
// names. Must match the order of scan.output because exprToProto binds
@@ -719,13 +721,78 @@ object CometDeltaNativeScan extends CometOperatorSerde[CometDeltaScanMarker] wit
719721
// executor = decode). If a DV file is missing/corrupt the executor
720722
// surfaces a `SparkException` -- same observable behaviour as before,
721723
// just at execution rather than planning.
722-
buildTaskListFromAddFiles(
723-
tableRoot,
724-
snapshotVersion,
725-
addFiles,
726-
nativeOp = null,
727-
columnNames,
728-
physicalToLogicalPartitionNames = physToLogical).toByteArray
724+
// Option (a): DML rewrites (TahoeBatchFileIndex -- touched files are a subset of the
725+
// pinned snapshot) get kernel's per-file transforms by enumerating the snapshot and
726+
// filtering to the touched AddFile paths, so they synthesize in-worker like regular
727+
// reads. Decline to the legacy AddFiles path if ANY touched file isn't in kernel's
728+
// enumeration (safe: the legacy DeltaSyntheticColumnsExec path still works -- never wrong
729+
// files). CDC indexes (files outside the snapshot) are excluded by isDmlRewriteFileIndex.
730+
val dmlSynthBytes: Option[Array[Byte]] =
731+
if (synthesizeConfig && DeltaReflection.isDmlRewriteFileIndex(relation.location)) {
732+
try {
733+
val annotated = scan.deltaMetadata.analyzedSchema.orElse(
734+
DeltaReflection.extractSnapshotSchema(relation))
735+
val projJson = CometDeltaNativeScan.synthesizeReadSchemaJson(
736+
annotated,
737+
scan.requiredSchema,
738+
relation.partitionSchema)
739+
// Empty predicate: the touched AddFile set is the authoritative selection; kernel
740+
// stats pruning could drop a touched file and force a needless decline.
741+
val kernelTaskList = DeltaScanTaskList.parseFrom(
742+
nativeLib.planDeltaScan(
743+
tableRoot,
744+
snapshotVersion,
745+
storageOptions,
746+
Array.emptyByteArray,
747+
columnNames,
748+
projJson))
749+
// Resolve touched AddFile paths the SAME way the native side resolves kernel paths
750+
// (table_root + rel, or pass-through for scheme'd paths), then match by file_path.
751+
val sep = if (tableRoot.endsWith("/")) "" else "/"
752+
val touched: Set[String] = addFiles.map { af =>
753+
if (af.path.contains(":/")) af.path else tableRoot + sep + af.path
754+
}.toSet
755+
val matched = kernelTaskList.getTasksList.asScala
756+
.filter(t => touched.contains(t.getFilePath))
757+
if (touched.nonEmpty && matched.size == touched.size) {
758+
Some(
759+
DeltaScanTaskList
760+
.newBuilder()
761+
.setSnapshotVersion(kernelTaskList.getSnapshotVersion)
762+
.setTableRoot(kernelTaskList.getTableRoot)
763+
.addAllUnsupportedFeatures(kernelTaskList.getUnsupportedFeaturesList)
764+
.setPhysicalSchema(kernelTaskList.getPhysicalSchema)
765+
.setLogicalSchema(kernelTaskList.getLogicalSchema)
766+
.addAllTasks(matched.asJava)
767+
.build()
768+
.toByteArray)
769+
} else {
770+
None
771+
}
772+
} catch {
773+
case scala.util.control.NonFatal(e) =>
774+
logWarning(
775+
s"CometDeltaNativeScan: DML kernel-enumerate for $tableRoot failed; " +
776+
s"using legacy AddFiles path",
777+
e)
778+
None
779+
}
780+
} else {
781+
None
782+
}
783+
dmlSynthBytes match {
784+
case Some(bytes) =>
785+
synthesizeInWorker = true
786+
bytes
787+
case None =>
788+
buildTaskListFromAddFiles(
789+
tableRoot,
790+
snapshotVersion,
791+
addFiles,
792+
nativeOp = null,
793+
columnNames,
794+
physicalToLogicalPartitionNames = physToLogical).toByteArray
795+
}
729796
case None =>
730797
// Reflection failed; fall back conservatively.
731798
import org.apache.comet.CometSparkSessionExtensions.withFallbackReason

‎contrib/delta/src/main/scala/org/apache/comet/contrib/delta/DeltaReflection.scala‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -492,6 +492,17 @@ object DeltaReflection extends Logging {
492492
cls.contains("TahoeChangeFileIndex")
493493
}
494494

495+
/**
496+
* A subset FileIndex whose touched `AddFile`s are a SUBSET of the pinned snapshot's active files
497+
* (MERGE / UPDATE / DELETE post-join rewrites, and streaming appends): `TahoeBatchFileIndex`. These
498+
* can be served by kernel enumeration FILTERED to the touched paths (every touched file is in the
499+
* snapshot, so kernel has its transform). The CDC family (`CdcAddFileIndex` / `TahoeRemoveFileIndex`
500+
* / `TahoeChangeFileIndex`) is NOT here -- it references removed / `_change_data` files outside the
501+
* snapshot, so kernel enumeration can't supply their transforms; those keep the legacy path.
502+
*/
503+
def isDmlRewriteFileIndex(location: Any): Boolean =
504+
location.getClass.getName.contains("TahoeBatchFileIndex")
505+
495506
/**
496507
* Detect whether the FileIndex carries a non-empty `rowIndexFilters` map. Delta
497508
* uses this to flag CDC "delete events" / "insert events" reads where the DV

0 commit comments

Comments
 (0)