Repository navigation
Commit 7b6e780
fix(delta): add synthesizedFilePartitions + patch DeltaSinkSuite/IdentityColumnAdmission
Two more regression failures fixed:
#7. `IdentityColumnAdmissionScalaSuite.streaming` -- This is an UPSTREAM
Delta test bug, NOT a Comet bug. Reproduces with Comet entirely
disabled (plugin commented out). The test calls
`MemoryStream.addData(1 to 10)` AFTER `start()` on a query with
`Trigger.AvailableNow`. AvailableNow processes only data present
at trigger time and exits before the late-arriving data can be
consumed; the expected `StreamingQueryException` is never thrown.
Patch the test diff to pre-populate the MemoryStream BEFORE
`start()`. Worth filing upstream against delta-io/delta.
#8. `DeltaSinkIdColumnMappingSuite "partitioned writing and batch
reading - column mapping id mode"` -- The test inspects
`executedPlan.collect[DataSourceScanExec]` and reads
`inputRDDs.head.asInstanceOf[FileScanRDD].filePartitions`. Comet
replaces the scan with `CometDeltaNativeScanExec` which uses
`CometExecRDD`, not `FileScanRDD`. Add a public method
`synthesizedFilePartitions` on `CometDeltaNativeScanExec` that
builds an equivalent `Seq[FilePartition]` from the scan's task
list (one PartitionedFile per task, with partition_values cast
from the proto using `DeltaReflection.castPartitionString`).
Patch the helper in the test diff to fall back to that accessor.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>1 parent 533893c commit 7b6e780
2 files changed
Lines changed: 78 additions & 0 deletions
File tree
- dev/diffs/delta
- spark/src/main/scala/org/apache/spark/sql/comet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
324 | 324 | | |
325 | 325 | | |
326 | 326 | | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
82 | 82 | | |
83 | 83 | | |
84 | 84 | | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
85 | 122 | | |
86 | 123 | | |
87 | 124 | | |
| |||
0 commit comments