Skip to content

Commit 09ebec3

Browse files
committed
[SPARK-58918][SQL][TEST][FOLLOWUP] Cover deeply nested types in the Arrow cache column-pruning read
### What changes were proposed in this pull request? Add a test to `ArrowCachedBatchSerializerSuite` covering deeply nested types in the Arrow cache column-pruning read path added by #58177 (SPARK-58918): `array<struct<...>>`, `struct<struct<struct<...>>>`, and `map<int, array<int>>`, each selected alone, mixed with primitives, reordered, and with other columns pruned in between, under both the row and vectorized read paths. ### Why are the changes needed? The pruning read copies each selected top-level column's entire buffer subtree, so `readProjectedRecordBatch`'s buffer-span arithmetic (`fieldBufferCount` / `fieldNodeCount`) recurses through nested types to arbitrary depth. The original test only exercised one level of nesting (`array<int>`, `struct` of primitives), so an off-by-one in the recursive span over a deeper subtree -- which would shift the buffers of a following selected column and produce wrong values -- was not covered. The new test fails if that recursion is broken (verified locally by making the buffer count ignore children). ### Does this PR introduce _any_ user-facing change? No, test-only. ### How was this patch tested? The new test; full `ArrowCachedBatchSerializerSuite` passes. ### Was this patch authored or co-authored using generative AI tooling? Yes, this pull request and its description were written by Claude Code. Closes #58215 from viirya/arrow-cache-nested-projection-test. Authored-by: Liang-Chi Hsieh <[email protected]> Signed-off-by: Liang-Chi Hsieh <[email protected]>
1 parent a6108a5 commit 09ebec3

1 file changed

Lines changed: 56 additions & 0 deletions

File tree

‎sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializerSuite.scala‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -304,6 +304,62 @@ class ArrowCachedBatchSerializerSuite extends QueryTest with SharedSparkSession
304304
}
305305
}
306306

307+
test("column projection prunes deeply nested columns on load") {
308+
// The buffer-span arithmetic must skip (and, when selected, copy) a column's ENTIRE buffer
309+
// subtree, at arbitrary nesting depth. Exercise array<struct>, struct<struct<struct>>, and
310+
// map<int, array<int>>, each selected while pruning neighbours whose own subtrees have varying
311+
// buffer counts, under both read paths, so an off-by-one in the recursive span would surface
312+
// as wrong values in a following column.
313+
Seq(false, true).foreach { vectorized =>
314+
withSQLConf(SQLConf.CACHE_VECTORIZED_READER_ENABLED.key -> vectorized.toString) {
315+
val schema = new StructType()
316+
.add("id", IntegerType)
317+
.add("arrOfStruct", ArrayType(new StructType().add("x", LongType).add("y", StringType)))
318+
.add("s", StringType)
319+
.add("deepStruct", new StructType()
320+
.add("l1", new StructType()
321+
.add("l2", new StructType().add("v", LongType).add("t", StringType))))
322+
.add("mapOfArray", MapType(IntegerType, ArrayType(IntegerType)))
323+
val rows = (1 to 30).map { i =>
324+
Row(i, Seq(Row(i.toLong, s"a$i"), Row((i + 1).toLong, s"b$i")), s"s$i",
325+
Row(Row(Row(i.toLong * 10, s"deep$i"))), Map(i -> Seq(i, i + 1, i + 2)))
326+
}
327+
val df = spark.createDataFrame(
328+
spark.sparkContext.parallelize(rows, 1), schema).cache()
329+
try {
330+
def expected(cols: Seq[String]): Seq[Row] = (1 to 30).map { i =>
331+
Row.fromSeq(cols.map {
332+
case "id" => i
333+
case "arrOfStruct" => Seq(Row(i.toLong, s"a$i"), Row((i + 1).toLong, s"b$i"))
334+
case "s" => s"s$i"
335+
case "deepStruct" => Row(Row(Row(i.toLong * 10, s"deep$i")))
336+
case "mapOfArray" => Map(i -> Seq(i, i + 1, i + 2))
337+
})
338+
}
339+
// Each nested column selected alone: its whole subtree must be copied, nothing else.
340+
checkAnswer(df.select("arrOfStruct"), expected(Seq("arrOfStruct")))
341+
checkAnswer(df.select("deepStruct"), expected(Seq("deepStruct")))
342+
checkAnswer(df.select("mapOfArray"), expected(Seq("mapOfArray")))
343+
// A primitive after a pruned deep column: the skip must span the whole deep subtree.
344+
checkAnswer(df.select("id", "s"), expected(Seq("id", "s")))
345+
// Reordered mix of nested and primitive columns, pruning others in between.
346+
checkAnswer(
347+
df.select("mapOfArray", "id", "deepStruct"),
348+
expected(Seq("mapOfArray", "id", "deepStruct")))
349+
checkAnswer(
350+
df.select("deepStruct", "arrOfStruct"),
351+
expected(Seq("deepStruct", "arrOfStruct")))
352+
// Full projection round-trips.
353+
checkAnswer(
354+
df.select("id", "arrOfStruct", "s", "deepStruct", "mapOfArray"),
355+
expected(Seq("id", "arrOfStruct", "s", "deepStruct", "mapOfArray")))
356+
} finally {
357+
df.unpersist()
358+
}
359+
}
360+
}
361+
}
362+
307363
test("caching with multiple batches") {
308364
withSQLConf(SQLConf.ARROW_EXECUTION_MAX_RECORDS_PER_BATCH.key -> "10") {
309365
val df = (1 to 50).map(i => (i, s"str$i")).toDF("a", "b")

0 commit comments

Comments
 (0)