Skip to content

Commit 66c63bd

Browse files
authored
fix: keep the values of sliced batches when coalescing a broadcast (#6900)
* fix: keep the values of sliced batches when coalescing a broadcast A slice of a native string, binary, list or map array reaches the JVM with offsets that start past zero, with the rows before the slice still at the front of its data. Arrow Java writes those bytes into the IPC stream, and the VectorSchemaRootAppender that coalesces broadcast batches assumes offsets start at zero, so it prepended them to the first value of each slice. serializeBatches now writes such a batch through a slice of its root, which rebases the offsets to zero and leaves the earlier bytes out. * Pass Long arguments explicitly in the UtilsSuite slice helper The strict-warnings build rejects the implicit Int to Long widening in the `ArrowFieldNode` and `ArrowBuf.slice` calls, so convert the arguments explicitly. * Document that sliced variable-width arrays reach the JVM with offsets past zero `zero_offsets` handles `ArrayData::offset`, but a sliced string, binary, list or map array still reaches the JVM with offsets that start past zero and the rows before the slice at the front of its data. Say so in the "Array Offsets" section, and that `serializeBatches` rebases broadcast batches because `VectorSchemaRootAppender` needs offsets that start at zero. * Close the rebased slice when writing a broadcast batch fails `splitAndTransfer` takes its own references on the buffers the slice shares with the batch, so a write that failed, as it does when a cancelled broadcast interrupts the task, left them pinned after the batch closed. Close the slice in a `finally`, and test that the slice's allocator drains to zero after an interrupted write.
1 parent a230ae5 commit 66c63bd

4 files changed

Lines changed: 181 additions & 6 deletions

File tree

‎docs/source/contributor-guide/ffi.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -241,6 +241,16 @@ level and shares every other buffer. `move_to_spark` applies it to executed batc
241241
or sliced booleans reach the JVM misaligned
242242
([#6288](https://github.com/apache/datafusion-comet/issues/6288)).
243243

244+
`zero_offsets` leaves the offsets of a sliced string, binary, list or map array as they are. arrow-rs slices those
245+
arrays by moving the start of their offsets, so the JVM imports them with offsets that start past zero and with the
246+
rows before the slice still at the front of the data buffer, or of the child vector for a list or map. Reading values
247+
through the offsets is correct, but Arrow Java 18.3's `VectorSchemaRootAppender` assumes offsets that start at zero and
248+
folds those rows into the first value it appends, and `VectorUnloader` writes them into an IPC stream. So
249+
`Utils.serializeBatches` rebases a broadcast batch's offsets with `VectorSchemaRoot.slice` before
250+
`coalesceBroadcastBatches` appends it ([#6779](https://github.com/apache/datafusion-comet/issues/6779)), and code that
251+
appends imported vectors has to rebase them too. The in-memory cache serializer still writes the earlier rows
252+
([#6995](https://github.com/apache/datafusion-comet/issues/6995)).
253+
244254
### Ownership and Lifecycle
245255

246256
Native allocates the data, and the JVM references it without copying:

‎spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala‎

Lines changed: 35 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ import scala.jdk.CollectionConverters._
2727

2828
import org.apache.arrow.c.CDataDictionaryProvider
2929
import org.apache.arrow.vector._
30-
import org.apache.arrow.vector.complex.{ListVector, MapVector, StructVector}
30+
import org.apache.arrow.vector.complex.{BaseRepeatedValueVector, ListVector, MapVector, StructVector}
3131
import org.apache.arrow.vector.dictionary.{Dictionary, DictionaryProvider}
3232
import org.apache.arrow.vector.dictionary.DictionaryProvider.MapDictionaryProvider
3333
import org.apache.arrow.vector.ipc.{ArrowStreamReader, ArrowStreamWriter}
@@ -267,11 +267,22 @@ object Utils extends CometTypeShim with Logging {
267267
}
268268
val provider = batchProviderOpt.getOrElse(dictionaryProvider)
269269

270-
val writer = new ArrowStreamWriter(root, provider, Channels.newChannel(out))
271-
writer.start()
272-
writer.writeBatch()
273-
root.clear()
274-
writer.close()
270+
// VectorSchemaRootAppender, which coalesces broadcast batches, assumes offsets that start
271+
// at zero, and Arrow Java writes the bytes before the first offset too. A slice of the root
272+
// rebases the offsets to zero and leaves those bytes out.
273+
val written =
274+
if (fieldVectors.forall(offsetsStartAtZero)) root else root.slice(0, root.getRowCount)
275+
try {
276+
val writer = new ArrowStreamWriter(written, provider, Channels.newChannel(out))
277+
writer.start()
278+
writer.writeBatch()
279+
root.clear()
280+
writer.close()
281+
} finally {
282+
// The slice holds its own references on the batch's buffers, so it is closed even when the
283+
// write fails, as it does when a cancelled broadcast interrupts the task.
284+
if (written ne root) written.close()
285+
}
275286

276287
if (out.size() > 0) {
277288
(batch.numRows().toLong, cbbos.toChunkedByteBuffer)
@@ -281,6 +292,23 @@ object Utils extends CometTypeShim with Logging {
281292
}
282293
}
283294

295+
/**
296+
* Whether every offset buffer in `vector`, its children included, starts at zero.
297+
*
298+
* arrow-rs slices a string, binary, list or map array by moving the start of its offsets, and
299+
* the C Data import keeps them, so a vector imported from a sliced native array has offsets
300+
* that start past zero. The rows before the slice stay at the front of its data buffer, or of
301+
* its child vector for a list or map.
302+
*/
303+
private def offsetsStartAtZero(vector: FieldVector): Boolean = {
304+
val firstOffset = vector match {
305+
case v: BaseVariableWidthVector if v.getValueCount > 0 => v.getOffsetBuffer.getInt(0)
306+
case v: BaseRepeatedValueVector if v.getValueCount > 0 => v.getOffsetBuffer.getInt(0)
307+
case _ => 0
308+
}
309+
firstOffset == 0 && vector.getChildrenFromFields.asScala.forall(offsetsStartAtZero)
310+
}
311+
284312
/**
285313
* The classes that carry the output of [[serializeBatches]] out of Comet, for Kryo registration
286314
* by [[org.apache.comet.CometKryoRegistrator]].
@@ -380,6 +408,7 @@ object Utils extends CometTypeShim with Logging {
380408
targetRoot.allocateNew()
381409
}
382410
try {
411+
// Appending needs offsets that start at zero, which serializeBatches writes.
383412
VectorSchemaRootAppender.append(targetRoot, sourceRoot)
384413
} catch {
385414
case e: IllegalArgumentException =>

‎spark/src/test/scala/org/apache/comet/exec/CometJoinSuite.scala‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,8 @@
1919

2020
package org.apache.comet.exec
2121

22+
import java.nio.charset.StandardCharsets
23+
2224
import org.scalactic.source.Position
2325
import org.scalatest.Tag
2426

@@ -1716,6 +1718,34 @@ class CometJoinSuite extends CometTestBase {
17161718
}
17171719
}
17181720

1721+
test("Broadcast coalescing keeps the values of sliced batches") {
1722+
// A grouped aggregate emits its groups as batch-size slices of one batch, so every slice after
1723+
// the first reaches the JVM with offsets that start past zero. Coalescing those slices used to
1724+
// prepend the bytes of the rows before each slice to its first value (#6779).
1725+
withSQLConf(
1726+
CometConf.COMET_BATCH_SIZE.key -> "100",
1727+
SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
1728+
SQLConf.SHUFFLE_PARTITIONS.key -> "2") {
1729+
val rows = (0 until 1000).map { i =>
1730+
val s = if (i % 7 == 0) null else s"value-$i"
1731+
val bytes = if (s == null) null else s.getBytes(StandardCharsets.UTF_8)
1732+
val array = if (i % 11 == 0) null else Seq(s, s"element-$i")
1733+
(i, s, bytes, array, (s, i))
1734+
}
1735+
withParquetTable(rows, "t") {
1736+
val (_, cometPlan) = checkSparkAnswerAndOperator(
1737+
sql("""
1738+
|SELECT /*+ BROADCAST(d) */ t._1, d._2, d._3, d._4, d._5
1739+
|FROM t JOIN (SELECT DISTINCT * FROM t) d ON t._1 = d._1
1740+
|""".stripMargin),
1741+
Seq(classOf[CometBroadcastExchangeExec], classOf[CometBroadcastHashJoinExec]))
1742+
1743+
val broadcast = collect(cometPlan) { case b: CometBroadcastExchangeExec => b }.head
1744+
assert(broadcast.metrics("numCoalescedRows").value == rows.length)
1745+
}
1746+
}
1747+
}
1748+
17191749
// Reproducer for SPARK-43113: full outer SMJ with a join filter that references
17201750
// a nullable column should not match when the filter evaluates to NULL.
17211751
test("SPARK-43113: Full outer SMJ with NULL in join filter") {

‎spark/src/test/scala/org/apache/spark/sql/comet/util/UtilsSuite.scala‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,20 @@
1919

2020
package org.apache.spark.sql.comet.util
2121

22+
import java.nio.channels.ClosedByInterruptException
23+
import java.nio.charset.StandardCharsets
24+
25+
import scala.jdk.CollectionConverters._
26+
2227
import org.apache.arrow.c.CDataDictionaryProvider
28+
import org.apache.arrow.memory.BufferAllocator
29+
import org.apache.arrow.vector.VarCharVector
30+
import org.apache.arrow.vector.ipc.message.ArrowFieldNode
2331
import org.apache.spark.sql.CometTestBase
2432
import org.apache.spark.sql.execution.vectorized.ConstantColumnVector
2533
import org.apache.spark.sql.types.{IntegerType, StringType, StructField, StructType, TimestampType}
2634
import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
35+
import org.apache.spark.util.io.ChunkedByteBuffer
2736

2837
import org.apache.comet.CometArrowAllocator
2938
import org.apache.comet.vector.CometVector
@@ -58,6 +67,64 @@ class UtilsSuite extends CometTestBase {
5867
assert(decoded.map(_.numRows()).sum == expected)
5968
}
6069

70+
test("serializeBatches rebases offsets that start past zero") {
71+
// A slice of a native string array reaches the JVM with its original offsets, which start
72+
// past zero, and with the bytes of the rows before the slice at the front of its data buffer.
73+
// Coalescing such batches for a broadcast used to fold those bytes into each batch's first
74+
// value (#6779).
75+
val values = (0 until 24).map(i => if (i % 5 == 0) null else s"value-$i")
76+
val skips = Seq(8, 16)
77+
78+
def serialize(vectors: Seq[VarCharVector]): Seq[ChunkedByteBuffer] = {
79+
val batches = vectors.map { v =>
80+
val column = CometVector.getVector(v, new CDataDictionaryProvider)
81+
new ColumnarBatch(Array[ColumnVector](column), v.getValueCount)
82+
}
83+
try Utils.serializeBatches(batches.iterator).map(_._2).toList
84+
finally batches.foreach(_.close())
85+
}
86+
87+
val sliced = serialize(skips.map(slicedStrings(values, _)))
88+
val (coalesced, batchCount, _) = Utils.coalesceBroadcastBatches(sliced.iterator)
89+
assert(batchCount == skips.length)
90+
val it = Utils.decodeBatches(coalesced.head, "test")
91+
val out = it.next()
92+
val got = (0 until out.numRows()).map { i =>
93+
if (out.column(0).isNullAt(i)) null else out.column(0).getUTF8String(i).toString
94+
}
95+
assert(!it.hasNext)
96+
assert(got == skips.flatMap(values.drop))
97+
98+
// The bytes before a slice are not written either: it takes as many bytes as the same values
99+
// in a vector of their own.
100+
val fresh = serialize(skips.map(skip => strings(values.drop(skip))))
101+
assert(sliced.map(_.size) == fresh.map(_.size))
102+
}
103+
104+
test("serializeBatches closes the slice it writes when the write fails") {
105+
// A cancelled broadcast interrupts its task, so the next write to the interruptible channel
106+
// serializeBatches writes through fails. The slice holds its own references on the batch's
107+
// buffers, and they would outlive the batch unless the slice is closed on that path too.
108+
val allocator = CometArrowAllocator.newChildAllocator("sliced-batch", 0, Long.MaxValue)
109+
try {
110+
val vector = slicedStrings((0 until 24).map(i => s"value-$i"), 8, allocator)
111+
val column = CometVector.getVector(vector, new CDataDictionaryProvider)
112+
val batch = new ColumnarBatch(Array[ColumnVector](column), vector.getValueCount)
113+
Thread.currentThread().interrupt()
114+
try {
115+
intercept[ClosedByInterruptException] {
116+
Utils.serializeBatches(Iterator(batch)).next()
117+
}
118+
} finally {
119+
Thread.interrupted()
120+
batch.close()
121+
}
122+
assert(allocator.getAllocatedMemory == 0)
123+
} finally {
124+
allocator.close()
125+
}
126+
}
127+
61128
test("serializeBatches materializes ConstantColumnVector columns") {
62129
// Spark wraps file-source partition columns and other per-batch constants in
63130
// ConstantColumnVector. When such a batch reaches Comet's serialization/export path
@@ -197,4 +264,43 @@ class UtilsSuite extends CometTestBase {
197264
}
198265
}
199266
}
267+
268+
private def strings(
269+
values: Seq[String],
270+
allocator: BufferAllocator = CometArrowAllocator): VarCharVector = {
271+
val vector = new VarCharVector("s", allocator)
272+
vector.allocateNew()
273+
values.zipWithIndex.foreach {
274+
case (null, i) => vector.setNull(i)
275+
case (value, i) => vector.setSafe(i, value.getBytes(StandardCharsets.UTF_8))
276+
}
277+
vector.setValueCount(values.length)
278+
vector
279+
}
280+
281+
/**
282+
* `values.drop(skip)` in the shape the C Data import gives a slice of a native string array:
283+
* the offsets are the original ones from `skip` on, and the data buffer still starts with the
284+
* bytes of the first `skip` values. `skip` is a multiple of 8, so the validity bitmap slices on
285+
* a byte boundary.
286+
*/
287+
private def slicedStrings(
288+
values: Seq[String],
289+
skip: Int,
290+
allocator: BufferAllocator = CometArrowAllocator): VarCharVector = {
291+
val full = strings(values, allocator)
292+
try {
293+
val length = values.length - skip
294+
val sliced = new VarCharVector("s", allocator)
295+
sliced.loadFieldBuffers(
296+
new ArrowFieldNode(length.toLong, values.drop(skip).count(_ == null).toLong),
297+
Seq(
298+
full.getValidityBuffer.slice(skip / 8L, (length + 7) / 8L),
299+
full.getOffsetBuffer.slice(skip * 4L, (length + 1) * 4L),
300+
full.getDataBuffer).asJava)
301+
sliced
302+
} finally {
303+
full.close()
304+
}
305+
}
200306
}

0 commit comments

Comments
 (0)