Repository navigation
Commit c7bc49e
committed
refactor: dedup Spark-columnar-to-Arrow copy and tidy cache serializer
Extract the element-wise Spark `ColumnVector` -> Arrow copy loop that
`columnarBatchToArrowBatch` had duplicated from `SparkColumnarArrowReader` into
a shared `CometArrowConverters.writeColumns`, so the zero-column rowCount
workaround lives in one place.
Also:
- hoist the Arrow schema out of the per-batch path
- replace the `(batch, ownsBatch)` tuple and `.toList` with a `serializeBatch`
helper that takes the single element eagerly
- move `isArrowBacked` to `Utils`, next to the `getBatchFieldVectors`
precondition it describes
- drop `toStructType` in favour of the existing `Utils.fromAttributes`
- document at `supportsColumnarInput` why its schema-only answer means the
conversion in `encodeBatches` is load bearing
- route the new tests through the existing `withNativeCache` helper and assert
both of them really cached in Comet's format1 parent 9dce0d6 commit c7bc49e
5 files changed
Lines changed: 137 additions & 151 deletions
File tree
- spark/src
- main/scala/org/apache/spark/sql/comet
- execution/arrow
- util
- test/scala/org/apache/comet/exec
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
70 | 70 | | |
71 | 71 | | |
72 | 72 | | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | 73 | | |
81 | 74 | | |
82 | 75 | | |
| |||
181 | 174 | | |
182 | 175 | | |
183 | 176 | | |
184 | | - | |
185 | | - | |
186 | | - | |
187 | | - | |
188 | | - | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
189 | 180 | | |
190 | 181 | | |
191 | 182 | | |
192 | | - | |
| 183 | + | |
| 184 | + | |
193 | 185 | | |
194 | | - | |
| 186 | + | |
195 | 187 | | |
196 | 188 | | |
197 | | - | |
198 | | - | |
199 | | - | |
200 | | - | |
201 | | - | |
202 | | - | |
203 | | - | |
204 | | - | |
205 | | - | |
206 | | - | |
207 | | - | |
208 | | - | |
209 | | - | |
210 | | - | |
211 | | - | |
212 | | - | |
213 | | - | |
214 | | - | |
215 | | - | |
216 | | - | |
217 | | - | |
218 | | - | |
219 | | - | |
220 | | - | |
221 | | - | |
222 | | - | |
223 | | - | |
224 | | - | |
225 | | - | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
226 | 196 | | |
227 | 197 | | |
228 | 198 | | |
229 | 199 | | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
230 | 211 | | |
231 | 212 | | |
232 | 213 | | |
| |||
285 | 266 | | |
286 | 267 | | |
287 | 268 | | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
288 | 276 | | |
289 | 277 | | |
290 | 278 | | |
| |||
356 | 344 | | |
357 | 345 | | |
358 | 346 | | |
359 | | - | |
| 347 | + | |
360 | 348 | | |
361 | 349 | | |
362 | 350 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
31 | | - | |
| 31 | + | |
32 | 32 | | |
33 | 33 | | |
34 | | - | |
35 | | - | |
36 | | - | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
37 | 38 | | |
38 | | - | |
39 | | - | |
40 | | - | |
41 | | - | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
42 | 44 | | |
43 | 45 | | |
44 | 46 | | |
| |||
76 | 78 | | |
77 | 79 | | |
78 | 80 | | |
79 | | - | |
80 | | - | |
81 | | - | |
| 81 | + | |
82 | 82 | | |
83 | | - | |
84 | | - | |
85 | | - | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
86 | 86 | | |
87 | | - | |
| 87 | + | |
| 88 | + | |
88 | 89 | | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | | - | |
93 | | - | |
| 90 | + | |
| 91 | + | |
94 | 92 | | |
95 | | - | |
96 | 93 | | |
97 | 94 | | |
98 | 95 | | |
99 | | - | |
| 96 | + | |
100 | 97 | | |
101 | 98 | | |
102 | 99 | | |
| |||
109 | 106 | | |
110 | 107 | | |
111 | 108 | | |
112 | | - | |
113 | 109 | | |
114 | 110 | | |
115 | | - | |
116 | | - | |
117 | | - | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
118 | 127 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
25 | | - | |
| 25 | + | |
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | | - | |
31 | | - | |
| 30 | + | |
| 31 | + | |
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| |||
76 | 76 | | |
77 | 77 | | |
78 | 78 | | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
91 | 84 | | |
92 | 85 | | |
93 | | - | |
94 | | - | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | | - | |
99 | 86 | | |
100 | 87 | | |
101 | 88 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
397 | 397 | | |
398 | 398 | | |
399 | 399 | | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
400 | 409 | | |
401 | 410 | | |
402 | 411 | | |
| |||
0 commit comments