Repository navigation
Commit 1e23881
committed
fix: recognize Dataset.rdd plans whose deserializer Spark removed
The guard for code reading Dataset.rdd looked for the DeserializeToObjectExec
that Dataset.rdd puts at the root of the plan. When the Dataset ends in a
typed operation such as map, Spark's EliminateSerialization drops that
deserializer together with the operation's serializer, so the root is the
operation itself, or a typed filter over it. The output of an earlier typed
operation was then still converted, and map(f).filter(...).map(g).rdd.take(1)
ran f on rows that Spark never reaches. The guard now takes any root that
produces objects, under a filter or a project, as code reading Dataset.rdd.
A Dataset's own plan ends in rows, so no other plan has such a root.1 parent bd36c6b commit 1e23881
2 files changed
Lines changed: 21 additions & 7 deletions
File tree
- spark/src
- main/scala/org/apache/comet/rules
- test/scala/org/apache/comet/exec
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1340 | 1340 | | |
1341 | 1341 | | |
1342 | 1342 | | |
1343 | | - | |
1344 | | - | |
1345 | | - | |
1346 | | - | |
1347 | | - | |
| 1343 | + | |
| 1344 | + | |
| 1345 | + | |
| 1346 | + | |
| 1347 | + | |
| 1348 | + | |
| 1349 | + | |
| 1350 | + | |
| 1351 | + | |
| 1352 | + | |
1348 | 1353 | | |
1349 | | - | |
| 1354 | + | |
1350 | 1355 | | |
1351 | 1356 | | |
1352 | 1357 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
317 | 317 | | |
318 | 318 | | |
319 | 319 | | |
320 | | - | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
321 | 330 | | |
322 | 331 | | |
323 | 332 | | |
| |||
0 commit comments