Skip to content

Commit 5c1e79c

Browse files
committed
test: check the dictionary path's recorded sizes, and state the size rationale once
1 parent a371d64 commit 5c1e79c

4 files changed

Lines changed: 29 additions & 30 deletions

File tree

‎spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala‎

Lines changed: 10 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -54,9 +54,7 @@ import org.apache.comet.vector.NativeUtil
5454
* payload.
5555
*
5656
* `sizeInBytes` is not the payload's size. It is inherited from `SimpleMetricsCachedBatch`, which
57-
* sums the per-column sizes in `stats`, and those are decoded sizes. Spark's planner reads it as
58-
* the size of a materialized cached relation, as it does for Spark's own cache formats, which
59-
* also report decoded sizes. See `statsRow`.
57+
* sums the decoded per-column sizes in `stats`. `statsRow` explains why they are decoded sizes.
6058
*/
6159
private case class CometCachedBatch(
6260
override val numRows: Int,
@@ -354,13 +352,14 @@ class ArrowCachedBatchSerializer extends SimpleMetricsCachedBatchSerializer {
354352
values(base + 1) = upper(c)
355353
values(base + 2) = nulls(c)
356354
values(base + 3) = numRows
357-
// The column's decoded size: the plain length of its own Arrow buffers before compression,
358-
// which is what Spark's own Arrow cache format records here. SimpleMetricsCachedBatch sums
359-
// these into sizeInBytes, which Spark's planner reads as the size of a materialized cached
360-
// relation, for the broadcast threshold and the shuffled hash join build side among others.
361-
// The compressed payload can be several times smaller, and a relation reported at that size
362-
// would be planned differently from the same relation in Spark's cache format, for example
363-
// broadcast where Spark's cache would have it shuffled.
355+
// The column's decoded size: the plain length of its own Arrow buffers before compression.
356+
// SimpleMetricsCachedBatch sums these into sizeInBytes, which Spark's planner reads as the
357+
// size of a materialized cached relation, for the broadcast threshold and the shuffled hash
358+
// join build side among others. Spark's DefaultCachedBatchSerializer also records each
359+
// column's uncompressed size here (ColumnStats.sizeInBytes). The compressed payload can be
360+
// several times smaller, and a relation reported at that size would be planned differently
361+
// from the same relation in Spark's cache format, for example broadcast where Spark's cache
362+
// would have it shuffled.
364363
values(base + 4) = columnSizes(c)
365364
c += 1
366365
}
@@ -407,7 +406,7 @@ class ArrowCachedBatchSerializer extends SimpleMetricsCachedBatchSerializer {
407406

408407
batches.map { batch =>
409408
// Bounds and null counts are read from the input batch before it is serialized, and the row
410-
// is only assembled once the per-column sizes the message reports are known.
409+
// is only assembled once serialize has measured the per-column sizes.
411410
val (lower, upper, nulls) = gatherColumnStats(batch, attrs, orderings)
412411
val numRows = batch.numRows()
413412

‎spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CachedBatchIpc.scala‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -186,10 +186,10 @@ private[comet] object CachedBatchIpc {
186186
* batch arrives at whatever size the plan above produced. Chunks are appended rather than grown
187187
* and recopied, so the write also never holds the payload twice.
188188
*
189-
* Returns the message and the decoded size of each top-level column, which the caller records
190-
* in the statistics row. Each size is measured on the batch before compression, from the plain
191-
* lengths of the column's own buffers, which is what `getBufferSize` reports for a vector and
192-
* what Spark's own Arrow cache format records.
189+
* Returns the message and the decoded size of each top-level column. Each size is measured on
190+
* the batch before compression, from the plain lengths of the column's own buffers, which is
191+
* what `getBufferSize` reports for a vector. The caller records the sizes in the statistics
192+
* row; `ArrowCachedBatchSerializer.statsRow` explains why they are decoded sizes.
193193
*
194194
* Dictionary-encoded columns are decoded to their plain form first. A payload with no Schema
195195
* message cannot describe a dictionary encoding, and the schema the reader rebuilds from Spark

‎spark/src/test/scala/org/apache/comet/exec/CometInMemoryCacheSuite.scala‎

Lines changed: 14 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,6 @@ package org.apache.comet.exec
2222
import java.{util => ju}
2323
import java.nio.charset.StandardCharsets
2424

25-
import scala.collection.mutable
2625
import scala.jdk.CollectionConverters._
2726

2827
import org.apache.arrow.compression.ZstdCompressionCodec
@@ -1998,12 +1997,12 @@ class CometInMemoryCacheSuite extends CometTestBase {
19981997
}
19991998

20001999
test("Comet in-memory cache records per-column decoded sizes in its statistics") {
2001-
// SimpleMetricsCachedBatch reserves a fifth field per column for its size and sums those into
2002-
// the batch's sizeInBytes, which is what Spark's planner reads as the size of a materialized
2003-
// cached relation. Spark's own formats record a column's decoded size there, so each field is
2004-
// compared with its column decoded back out of the payload, not with what the column occupies
2005-
// compressed. Run over the nested relation as well: a nested column's size is the sum of its
2006-
// whole subtree, so this is also where a size attributed to the wrong column surfaces.
2000+
// Each column's size field must be its decoded size (see ArrowCachedBatchSerializer.statsRow),
2001+
// so each field is compared with its column decoded back out of the payload, not with what the
2002+
// column occupies compressed. Run over the nested relation as well: a nested column's size is
2003+
// the sum of its whole subtree, so this is also where a size attributed to the wrong column
2004+
// surfaces. And over the dictionary relation, whose columns reach the writer dictionary
2005+
// encoded: a size measured before they are decoded would be the size of their indices.
20072006
def checkSizes(
20082007
relation: org.apache.spark.sql.execution.columnar.InMemoryRelation,
20092008
batches: Array[CachedBatch]): Unit = {
@@ -2028,14 +2027,15 @@ class CometInMemoryCacheSuite extends CometTestBase {
20282027

20292028
withProjectionCache(checkSizes _)
20302029
withNestedProjectionCache(checkSizes _)
2030+
withDictionaryCache(relation =>
2031+
checkSizes(relation, relation.cacheBuilder.cachedColumnBuffers.collect()))
20312032
}
20322033

20332034
test("Comet in-memory cache reports the same relation size under every codec") {
2034-
// Broadcast thresholds and the shuffled hash join build side are compared against this size.
2035-
// Reported compressed, a relation that zstd shrinks several times over would be broadcast
2036-
// where the same relation in Spark's cache format is shuffled, so it must not depend on the
2037-
// codec.
2038-
val sizes = mutable.ArrayBuffer.empty[(String, Long, Long)]
2035+
// The planner compares this size with broadcast thresholds and with the other side of a
2036+
// shuffled hash join, so it must not depend on the codec (see
2037+
// ArrowCachedBatchSerializer.statsRow).
2038+
val measured = Seq.newBuilder[(String, Long, Long)]
20392039
Seq("zstd", "none").foreach { codec =>
20402040
withSQLConf(
20412041
SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
@@ -2054,11 +2054,12 @@ class CometInMemoryCacheSuite extends CometTestBase {
20542054
.cachedRepresentation
20552055
val batches = relation.cacheBuilder.cachedColumnBuffers.collect()
20562056
val payload = batches.map(CometCachedBatchHelper.payloadSize).sum
2057-
sizes += ((codec, relation.computeStats().sizeInBytes.toLong, payload))
2057+
measured += ((codec, relation.computeStats().sizeInBytes.toLong, payload))
20582058
spark.catalog.clearCache()
20592059
}
20602060
}
20612061

2062+
val sizes = measured.result()
20622063
val (_, zstdSize, zstdPayload) = sizes(0)
20632064
val (_, plainSize, _) = sizes(1)
20642065
assert(zstdPayload * 2 < zstdSize, s"zstd should compress this relation: $sizes")

‎spark/src/test/scala/org/apache/spark/sql/comet/execution/arrow/CometCachedBatchHelper.scala‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -199,8 +199,7 @@ object CometCachedBatchHelper {
199199

200200
/**
201201
* Decoded size of each top-level column: the column read back out of the payload the way a scan
202-
* reads it, measured the way Spark's own Arrow cache format measures a column for its
203-
* statistics.
202+
* reads it, measured with `getBufferSize`.
204203
*/
205204
def decodedColumnSizes(
206205
batch: CachedBatch,

0 commit comments

Comments
 (0)