Repository navigation
Commit 891008e
authored
fix: report SinglePartition from a native Iceberg scan that replaces one (#7424)
* fix: report SinglePartition from a native Iceberg scan that replaces one
On Spark 3.4, BatchScanExec reports SinglePartition for a scan of one
input partition, and Spark plans no exchange above it. The native Iceberg
scan reported UnknownPartitioning(0). Since the aggregates declare their
required distribution (#7231), EnsureRequirements, which AQE runs again
over a DPP subquery's converted plan, inserted an exchange inside a native
block that was already serialized, and the native scan lost its tasks.
Report SinglePartition when the replaced scan does.
* test: run the SPARK-34637 Iceberg DPP test on Spark 3.4 again1 parent c7b2cfd commit 891008e
2 files changed
Lines changed: 20 additions & 9 deletions
File tree
- spark/src
- main/scala/org/apache/spark/sql/comet
- test/scala/org/apache/comet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | | - | |
| 29 | + | |
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| |||
135 | 135 | | |
136 | 136 | | |
137 | 137 | | |
138 | | - | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
139 | 150 | | |
140 | 151 | | |
141 | 152 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6743 | 6743 | | |
6744 | 6744 | | |
6745 | 6745 | | |
6746 | | - | |
6747 | | - | |
6748 | 6746 | | |
6749 | 6747 | | |
6750 | 6748 | | |
| |||
6780 | 6778 | | |
6781 | 6779 | | |
6782 | 6780 | | |
6783 | | - | |
6784 | | - | |
| 6781 | + | |
| 6782 | + | |
| 6783 | + | |
| 6784 | + | |
| 6785 | + | |
| 6786 | + | |
| 6787 | + | |
6785 | 6788 | | |
6786 | | - | |
6787 | | - | |
6788 | | - | |
6789 | 6789 | | |
6790 | 6790 | | |
6791 | 6791 | | |
| |||
0 commit comments