Repository navigation
Commit 29361d6
fix: distinguish CometDeltaNativeScan instances across snapshot versions
CometDeltaNativeScanExec's equality and findAllPlanData key were both
tableRoot-only, so two scans of the same Delta path at different
time-travel versions (e.g. v0 and v1 in a self-join) collided:
- findAllPlanData's `.toMap` silently dropped one of the two entries,
leaving only one scan's file list available to the injector
- Spark's ReuseExchangeAndSubquery rule considered the two exchanges
equal via the scan's equals/hashCode, replacing v1's exchange with
ReusedExchange of v0's — so both sides of a full-outer join on the
same key read v0's data and "unmatched" rows (keys 5..9) vanished
Introduce `CometDeltaNativeScanExec.computeSourceKey(op)` derived from
the DeltaScanCommon proto (table root, snapshot version, schemas,
filters, projection, column mappings) — mirrors CometNativeScanExec's
sourceKey pattern. Use it:
- as the key in commonByKey / perPartitionByKey maps
- as the key in findAllPlanData results
- as the lookup key in DeltaPlanDataInjector.getKey
- in equals/hashCode so two scans at different versions are not equal
Surfaced by running Delta's own DeltaTimeTravelSuite under the Comet
regression diff:
`scans on different versions of same table are executed correctly`
was producing 0 rows where `a.key IS NULL` (should be 5). All 24 tests
in that suite now pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>1 parent 3e4b6a0 commit 29361d6
2 files changed
Lines changed: 51 additions & 8 deletions
File tree
- spark/src/main/scala/org/apache/spark/sql/comet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
144 | 144 | | |
145 | 145 | | |
146 | 146 | | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
147 | 159 | | |
148 | 160 | | |
149 | 161 | | |
| |||
191 | 203 | | |
192 | 204 | | |
193 | 205 | | |
194 | | - | |
195 | | - | |
| 206 | + | |
| 207 | + | |
196 | 208 | | |
197 | 209 | | |
198 | 210 | | |
| |||
244 | 256 | | |
245 | 257 | | |
246 | 258 | | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
247 | 263 | | |
248 | 264 | | |
249 | | - | |
| 265 | + | |
| 266 | + | |
250 | 267 | | |
251 | 268 | | |
252 | 269 | | |
253 | 270 | | |
254 | | - | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
255 | 296 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
253 | 253 | | |
254 | 254 | | |
255 | 255 | | |
256 | | - | |
| 256 | + | |
257 | 257 | | |
258 | 258 | | |
259 | 259 | | |
| |||
721 | 721 | | |
722 | 722 | | |
723 | 723 | | |
724 | | - | |
| 724 | + | |
| 725 | + | |
| 726 | + | |
725 | 727 | | |
726 | 728 | | |
727 | 729 | | |
728 | | - | |
729 | | - | |
| 730 | + | |
| 731 | + | |
730 | 732 | | |
731 | 733 | | |
732 | 734 | | |
| |||
0 commit comments