Skip to content

Commit c25c7b3

Browse files
committed
fix: label cached timestamps UTC on the row write path too
The row write path passed conf.sessionLocalTimeZone into rowToArrowBatchIter while the columnar path already encoded with CometArrowStream.NATIVE_TIMEZONE, so the same logical cache had two physical formats depending on which path filled it, and the row one persisted the writing session's timezone into cached data. Unlike Spark's Arrow cache, whose RecordBatch is deliberately schema-less and reconstructs the timezone on read, CometCachedBatch stores a full IPC stream including the schema, so that label really is written down. Standardise on NATIVE_TIMEZONE ("UTC") for TimestampType, per the analysis on the review thread. This is a label only. Spark's internal timestamp representation is micros since the Unix epoch regardless of session timezone, so no values are converted, and matching Comet's native schema also avoids a cast at the native boundary. TimestampNTZType already had no timezone: Utils.toArrowType maps it to Timestamp(MICROSECOND, null) whatever is passed in. The closure no longer needs the session timezone at all, so it no longer captures anything derived from `conf`. Test: "stores timestamps with a UTC schema label" caches a row-based (local Seq) plan under two non-UTC session timezones, decodes the cached batches back through the serializer, and asserts the Arrow field carries "UTC" -- then checks the values and their string rendering still match Spark. Reverting the one-line change fails it with `got [America/Los_Angeles]`, so it pins the format rather than passing vacuously. All 20 tests in CometInMemoryCacheSuite pass on spark-3.5.
1 parent 1367665 commit c25c7b3

2 files changed

Lines changed: 92 additions & 2 deletions

File tree

‎spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -363,14 +363,21 @@ class ArrowCachedBatchSerializer extends SimpleMetricsCachedBatchSerializer {
363363
fallback.convertInternalRowToCachedBatch(input, schema, storageLevel, conf)
364364
} else {
365365
val batchSize = conf.columnBatchSize
366-
val sessionTz = conf.sessionLocalTimeZone
367366

368367
input.mapPartitions { rows =>
369368
val iter = CometArrowConverters.rowToArrowBatchIter(
370369
rows,
371370
Utils.fromAttributes(schema),
372371
batchSize,
373-
sessionTz,
372+
// NATIVE_TIMEZONE ("UTC"), not conf.sessionLocalTimeZone, so both write paths produce
373+
// the same physical format: the columnar path above already encodes with
374+
// NATIVE_TIMEZONE. Unlike Spark's Arrow cache, whose RecordBatch is deliberately
375+
// schema-less, CometCachedBatch stores a full IPC stream including the schema, so a
376+
// session-local label would persist the writing session's mutable timezone into cached
377+
// data. This is a label only: Spark's internal timestamp representation is micros since
378+
// the Unix epoch regardless of session timezone, so no values are converted. It also
379+
// matches Comet's native schema, avoiding a cast at the native boundary.
380+
CometArrowStream.NATIVE_TIMEZONE,
374381
CometArrowAllocator)
375382

376383
encodeBatches(iter, schema)

‎spark/src/test/scala/org/apache/comet/exec/CometInMemoryCacheSuite.scala‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ package org.apache.comet.exec
2121

2222
import java.{util => ju}
2323

24+
import org.apache.arrow.vector.types.pojo.ArrowType
2425
import org.apache.spark.CometDriverPlugin
2526
import org.apache.spark.SparkConf
2627
import org.apache.spark.sql.CometTestBase
@@ -32,9 +33,12 @@ import org.apache.spark.storage.StorageLevel
3233

3334
import org.apache.comet.CometConf
3435
import org.apache.comet.CometSparkSessionExtensions.isSpark40Plus
36+
import org.apache.comet.vector.CometVector
3537

3638
class CometInMemoryCacheSuite extends CometTestBase {
3739

40+
import testImplicits._
41+
3842
// `InMemoryRelation` resolves `spark.sql.cache.serializer` once per JVM and memoizes the
3943
// instance in a static field. Test suites share a forked JVM, so whichever suite caches a
4044
// table first pins the serializer for everything that follows: without this reset the
@@ -521,6 +525,85 @@ class CometInMemoryCacheSuite extends CometTestBase {
521525
}
522526
}
523527

528+
test("Comet in-memory cache stores timestamps with a UTC schema label") {
529+
// Unlike Spark's Arrow cache, whose RecordBatch is deliberately schema-less, CometCachedBatch
530+
// stores a full IPC stream including the schema. Labelling TimestampType with the writing
531+
// session's timezone would persist a mutable session value into cached data and would make the
532+
// row write path disagree with the columnar one, which already encodes with NATIVE_TIMEZONE.
533+
// So both paths must write "UTC". This is a label only -- Spark stores timestamps as micros
534+
// since the Unix epoch regardless of session timezone -- so values must be unaffected.
535+
Seq("America/Los_Angeles", "Asia/Kolkata").foreach { sessionTz =>
536+
withSQLConf(
537+
SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
538+
CometConf.COMET_SHUFFLE_MODE.key -> "jvm",
539+
SQLConf.CACHE_VECTORIZED_READER_ENABLED.key -> "true",
540+
CometConf.COMET_EXEC_IN_MEMORY_CACHE_ENABLED.key -> "true",
541+
"spark.comet.sparkToColumnar.enabled" -> "true",
542+
SQLConf.SESSION_LOCAL_TIMEZONE.key -> sessionTz) {
543+
544+
spark.catalog.clearCache()
545+
546+
// A local Seq gives a row-based plan, so this exercises
547+
// convertInternalRowToCachedBatch rather than the columnar path.
548+
val rows = Seq(
549+
(1, java.sql.Timestamp.valueOf("2024-01-31 12:34:56.789")),
550+
(2, java.sql.Timestamp.valueOf("1970-01-01 00:00:00")),
551+
(3, null))
552+
rows.toDF("id", "ts").createOrReplaceTempView("ts_cache")
553+
554+
spark.catalog.cacheTable("ts_cache")
555+
assert(spark.table("ts_cache").count() == 3)
556+
557+
assert(
558+
cachedBatchTypes("ts_cache").sameElements(
559+
Array("org.apache.spark.sql.comet.execution.arrow.CometCachedBatch")),
560+
s"expected Comet cache format for sessionTz=$sessionTz")
561+
562+
// Decode the cached bytes through the serializer and read the Arrow field metadata back
563+
// out. The timezone has to be extracted inside the closure: ColumnarBatch is not
564+
// serializable.
565+
val relation =
566+
spark.sharedState.cacheManager
567+
.lookupCachedData(spark.table("ts_cache"))
568+
.get
569+
.cachedRepresentation
570+
val tsIndex = relation.output.indexWhere(_.name == "ts")
571+
val labels = relation.cacheBuilder.serializer
572+
.convertCachedBatchToColumnarBatch(
573+
relation.cacheBuilder.cachedColumnBuffers,
574+
relation.output,
575+
relation.output,
576+
spark.sessionState.conf)
577+
.mapPartitions { batches =>
578+
batches.take(1).map { batch =>
579+
batch.column(tsIndex) match {
580+
case v: CometVector =>
581+
v.getValueVector.getField.getType match {
582+
case t: ArrowType.Timestamp => String.valueOf(t.getTimezone)
583+
case other => s"unexpected arrow type $other"
584+
}
585+
case other => s"unexpected vector ${other.getClass.getName}"
586+
}
587+
}
588+
}
589+
.collect()
590+
.distinct
591+
592+
assert(
593+
labels.sameElements(Array("UTC")),
594+
s"expected the cached timestamp schema to be labelled UTC for sessionTz=$sessionTz, " +
595+
s"got ${labels.mkString("[", ",", "]")}")
596+
597+
// The label change must not move any values.
598+
checkSparkAnswer(spark.sql("SELECT id, ts FROM ts_cache ORDER BY id"))
599+
checkSparkAnswer(
600+
spark.sql("SELECT id, CAST(ts AS STRING) AS s FROM ts_cache ORDER BY id"))
601+
602+
spark.catalog.clearCache()
603+
}
604+
}
605+
}
606+
524607
test("Comet plugin respects user-provided cache serializer") {
525608
val serializerKey = StaticSQLConf.SPARK_CACHE_SERIALIZER.key
526609
val cometSerializer =

0 commit comments

Comments
 (0)