Repository navigation
Commit dc35e97
feat(delta): partitioned-table support for native Delta scan
Reads partitioned Delta tables natively: the projection now uses the scan's full output
schema (data + partition columns, in output order), and delta-kernel-rs injects partition
values during the scan. tryDeltaNativeScan no longer bails on partitioned tables, but stays
correct by bailing when the scan carries partition filters (Spark consumes those during file
pruning with no Filter node above, so a full kernel read would return pruned-away rows) or
data filters.
CometDeltaNativeReadSuite gains two tests: a full read of a table partitioned by an int and a
string column (values match Spark via checkSparkAnswer), and a partition-filter query that must
fall back to Spark.
Validated end to end on Spark 4.1 + Delta 4.1.0 with a delta-feature native build (5/5 tests
pass).
Part of apache#174.
Co-authored-by: Copilot <[email protected]>1 parent c03ccf7 commit dc35e97
2 files changed
Lines changed: 77 additions & 7 deletions
File tree
- spark/src
- main/scala/org/apache/comet/rules
- test/scala/org/apache/comet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
197 | 197 | | |
198 | 198 | | |
199 | 199 | | |
200 | | - | |
201 | | - | |
202 | | - | |
203 | | - | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
204 | 206 | | |
205 | 207 | | |
206 | 208 | | |
| |||
212 | 214 | | |
213 | 215 | | |
214 | 216 | | |
215 | | - | |
| 217 | + | |
| 218 | + | |
216 | 219 | | |
217 | 220 | | |
218 | 221 | | |
219 | | - | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
220 | 225 | | |
221 | | - | |
| 226 | + | |
222 | 227 | | |
223 | 228 | | |
224 | 229 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
98 | 98 | | |
99 | 99 | | |
100 | 100 | | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
101 | 166 | | |
102 | 167 | | |
103 | 168 | | |
| |||
0 commit comments