Repository navigation
Commit 09ebec3
committed
[SPARK-58918][SQL][TEST][FOLLOWUP] Cover deeply nested types in the Arrow cache column-pruning read
### What changes were proposed in this pull request?
Add a test to `ArrowCachedBatchSerializerSuite` covering deeply nested types in the Arrow cache column-pruning read path added by #58177 (SPARK-58918): `array<struct<...>>`, `struct<struct<struct<...>>>`, and `map<int, array<int>>`, each selected alone, mixed with primitives, reordered, and with other columns pruned in between, under both the row and vectorized read paths.
### Why are the changes needed?
The pruning read copies each selected top-level column's entire buffer subtree, so `readProjectedRecordBatch`'s buffer-span arithmetic (`fieldBufferCount` / `fieldNodeCount`) recurses through nested types to arbitrary depth. The original test only exercised one level of nesting (`array<int>`, `struct` of primitives), so an off-by-one in the recursive span over a deeper subtree -- which would shift the buffers of a following selected column and produce wrong values -- was not covered. The new test fails if that recursion is broken (verified locally by making the buffer count ignore children).
### Does this PR introduce _any_ user-facing change?
No, test-only.
### How was this patch tested?
The new test; full `ArrowCachedBatchSerializerSuite` passes.
### Was this patch authored or co-authored using generative AI tooling?
Yes, this pull request and its description were written by Claude Code.
Closes #58215 from viirya/arrow-cache-nested-projection-test.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Liang-Chi Hsieh <[email protected]>1 parent a6108a5 commit 09ebec3
1 file changed
Lines changed: 56 additions & 0 deletions
File tree
- sql/core/src/test/scala/org/apache/spark/sql/execution/columnar
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
304 | 304 | | |
305 | 305 | | |
306 | 306 | | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
307 | 363 | | |
308 | 364 | | |
309 | 365 | | |
| |||
0 commit comments