Skip to content

Commit 18c147f

Browse files
schenksjclaude
andcommitted
fix: address review feedback on ConstantColumnVector materialization
- Drop the `Utils.materializeConstantColumnVector` forwarder; both callers (`getBatchFieldVectors`, `NativeUtil.exportBatch`) now call `ConstantColumnVectors.materialize` directly, with the "UTC" rationale living once on that method (mbutrovich). - Use the imported `ConstantColumnVector` short name in `NativeUtil` instead of the fully-qualified reference (mbutrovich). - Read `numRows` from `materialised.getValueCount` in the export arm, symmetric with the `CometVector` arm (mbutrovich). - Register `org.apache.comet.vector.NativeUtilSuite` in the pr_build_{linux,macos}.yml workflows to fix the Preflight check (andygrove). Targeted suites pass (NativeUtilSuite + UtilsSuite, 6/6) on Spark 4.1/2.13; also compiles clean under Scala 2.12 (Spark 3.4). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01BtErWgRQKCDRAg8Mk6qR4G
1 parent 2750070 commit 18c147f

5 files changed

Lines changed: 26 additions & 46 deletions

File tree

‎.github/workflows/pr_build_linux.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -358,6 +358,7 @@ jobs:
358358
org.apache.spark.sql.comet.CometShuffleFallbackStickinessSuite
359359
org.apache.spark.sql.comet.CometDecimalArithmeticViewSuite
360360
org.apache.spark.sql.comet.util.UtilsSuite
361+
org.apache.comet.vector.NativeUtilSuite
361362
org.apache.comet.objectstore.NativeConfigSuite
362363
org.apache.spark.sql.CometToPrettyStringSuite
363364
org.apache.spark.sql.CometCollationSuite

‎.github/workflows/pr_build_macos.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -174,6 +174,7 @@ jobs:
174174
org.apache.spark.sql.comet.CometShuffleFallbackStickinessSuite
175175
org.apache.spark.sql.comet.CometDecimalArithmeticViewSuite
176176
org.apache.spark.sql.comet.util.UtilsSuite
177+
org.apache.comet.vector.NativeUtilSuite
177178
org.apache.comet.objectstore.NativeConfigSuite
178179
org.apache.spark.sql.CometToPrettyStringSuite
179180
org.apache.spark.sql.CometCollationSuite

‎spark/src/main/scala/org/apache/comet/vector/NativeUtil.scala‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,9 @@ import org.apache.arrow.c.{ArrowArray, ArrowImporter, ArrowSchema, CDataDictiona
2525
import org.apache.arrow.vector.VectorSchemaRoot
2626
import org.apache.arrow.vector.dictionary.DictionaryProvider
2727
import org.apache.spark.SparkException
28+
import org.apache.spark.sql.comet.execution.arrow.ConstantColumnVectors
2829
import org.apache.spark.sql.comet.util.Utils
30+
import org.apache.spark.sql.execution.vectorized.ConstantColumnVector
2931
import org.apache.spark.sql.vectorized.ColumnarBatch
3032

3133
import org.apache.comet.CometArrowAllocator
@@ -137,16 +139,17 @@ class NativeUtil {
137139
provider,
138140
arrowArray,
139141
arrowSchema)
140-
case cv: org.apache.spark.sql.execution.vectorized.ConstantColumnVector =>
142+
case cv: ConstantColumnVector =>
141143
// Spark uses ConstantColumnVector for partition columns / per-batch constants (e.g.
142144
// partition values, synthetic columns). Materialise to a fresh Arrow vector so Comet's
143145
// native side -- which expects Arrow Arrays only -- can ingest the batch. Without this,
144146
// queries that pull constants through a Comet operator fail with "Comet execution only
145-
// takes Arrow Arrays".
146-
val rows = batch.numRows()
147-
numRows += rows
148-
val materialised = org.apache.spark.sql.comet.util.Utils
149-
.materializeConstantColumnVector(cv, cv.dataType(), rows, s"_const_$index", allocator)
147+
// takes Arrow Arrays". "UTC" is intentional -- see `ConstantColumnVectors`.
148+
val materialised = ConstantColumnVectors
149+
.materialize(cv, cv.dataType(), batch.numRows(), s"_const_$index", allocator, "UTC")
150+
151+
numRows += materialised.getValueCount
152+
150153
val arrowSchema = ArrowSchema.wrap(schemaAddrs(index))
151154
val arrowArray = ArrowArray.wrap(arrayAddrs(index))
152155
Data.exportVector(allocator, materialised, null, arrowArray, arrowSchema)

‎spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowWriters.scala‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,16 @@ private[arrow] object ArrowWriter {
104104
*
105105
* Lives in this package because `ArrowWriter` is `private[arrow]`. The caller owns the returned
106106
* vector and must close it (or hand it to Arrow's exporter, which takes ownership).
107+
*
108+
* Comet's serialize/export callers pass `timeZoneId = "UTC"` -- deliberately, NOT the
109+
* session-local timezone that `toArrowSchema` threads through. These constants are materialised
110+
* alongside non-constant columns in the same batch/`VectorSchemaRoot`, and Comet's non-constant
111+
* `TimestampType` columns are Arrow vectors exported from native execution, where Comet always
112+
* tags them `Timestamp(us, "UTC")` (see native `serde.rs`). Spark itself stores `TimestampType`
113+
* as micros in UTC, so the constant's value is already a UTC instant. Tagging the materialised
114+
* constant "UTC" keeps its Arrow timezone metadata consistent with its sibling timestamp columns;
115+
* threading the session-local timezone here would instead introduce the mismatch.
116+
* `TimestampNTZType` carries no zone regardless of this argument.
107117
*/
108118
object ConstantColumnVectors {
109119
def materialize(

‎spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala‎

Lines changed: 5 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,6 @@ import java.nio.channels.Channels
2626
import scala.jdk.CollectionConverters._
2727

2828
import org.apache.arrow.c.CDataDictionaryProvider
29-
import org.apache.arrow.memory.BufferAllocator
3029
import org.apache.arrow.vector._
3130
import org.apache.arrow.vector.complex.{ListVector, MapVector, StructVector}
3231
import org.apache.arrow.vector.dictionary.DictionaryProvider
@@ -38,7 +37,7 @@ import org.apache.spark.{SparkEnv, SparkException}
3837
import org.apache.spark.internal.Logging
3938
import org.apache.spark.io.CompressionCodec
4039
import org.apache.spark.sql.catalyst.expressions.Attribute
41-
import org.apache.spark.sql.comet.execution.arrow.ArrowReaderIterator
40+
import org.apache.spark.sql.comet.execution.arrow.{ArrowReaderIterator, ConstantColumnVectors}
4241
import org.apache.spark.sql.execution.vectorized.ConstantColumnVector
4342
import org.apache.spark.sql.types._
4443
import org.apache.spark.sql.vectorized.ColumnarBatch
@@ -405,13 +404,14 @@ object Utils extends CometTypeShim with Logging {
405404
case cv: ConstantColumnVector =>
406405
// Spark wraps file-source partition columns and other per-batch constants in
407406
// `ConstantColumnVector`. Materialise to an Arrow vector so the serialisation path
408-
// doesn't reject the batch.
409-
materializeConstantColumnVector(
407+
// doesn't reject the batch. "UTC" is intentional -- see `ConstantColumnVectors`.
408+
ConstantColumnVectors.materialize(
410409
cv,
411410
cv.dataType(),
412411
rows,
413412
s"_const_$index",
414-
org.apache.comet.CometArrowAllocator)
413+
org.apache.comet.CometArrowAllocator,
414+
"UTC")
415415

416416
case c =>
417417
throw new SparkException(
@@ -440,39 +440,4 @@ object Utils extends CometTypeShim with Logging {
440440
throw new SparkException(s"Unsupported Arrow Vector for $reason: ${valueVector.getClass}")
441441
}
442442
}
443-
444-
/**
445-
* Materialize a Spark `ConstantColumnVector` into a fresh Arrow `FieldVector` whose value is
446-
* the same constant repeated `numRows` times.
447-
*
448-
* Spark wraps file-source partition columns and other per-batch constants in
449-
* `ConstantColumnVector`; downstream Comet operators feeding `NativeUtil.exportBatch` or
450-
* `getBatchFieldVectors` trip on it because those paths only handle `CometVector`. This helper
451-
* materializes the constant into an Arrow vector inline.
452-
*
453-
* The caller owns the returned vector and must close it (or hand it to Arrow's exporter, which
454-
* transfers ownership). The vector is allocated against `allocator`, sized to exactly
455-
* `numRows`, and pre-filled with the constant value (or null when `cv.isNullAt(0)`).
456-
*
457-
* All Spark types are supported (delegates to the per-type ArrowFieldWriters, which include
458-
* struct/array/map); throws only for a type Arrow itself can't represent.
459-
*/
460-
def materializeConstantColumnVector(
461-
cv: ConstantColumnVector,
462-
dt: DataType,
463-
numRows: Int,
464-
name: String,
465-
allocator: BufferAllocator): FieldVector = {
466-
// "UTC" is deliberate here, NOT the session-local timezone that `toArrowSchema` threads
467-
// through. These constants are materialised alongside non-constant columns in the same
468-
// batch/`VectorSchemaRoot`, and Comet's non-constant `TimestampType` columns are Arrow
469-
// vectors exported from native execution, where Comet always tags them `Timestamp(us, "UTC")`
470-
// (see native `serde.rs`). Spark itself stores `TimestampType` as micros in UTC, so the
471-
// constant's value is already a UTC instant. Tagging the materialised constant "UTC" keeps its
472-
// Arrow timezone metadata consistent with its sibling timestamp columns; threading the
473-
// session-local timezone here would instead introduce the mismatch. `TimestampNTZType` carries
474-
// no zone regardless of this argument.
475-
org.apache.spark.sql.comet.execution.arrow.ConstantColumnVectors
476-
.materialize(cv, dt, numRows, name, allocator, "UTC")
477-
}
478443
}

0 commit comments

Comments
 (0)