Skip to content

Commit 7de2708

Browse files
committed
test: align the system-function corpus write with Spark 4 datetime defaults
CometIcebergSystemFunctionSuite aborted in beforeAll on Spark 3.4 and 3.5. Three Spark 3.x defaults differ from Spark 4's and each blocks writing the corpus: - datetimeJava8ApiEnabled is off, so the row encoder rejects the java.time values sourceData supplies (ClassCastException: LocalDate -> java.sql.Date) - datetimeRebaseModeInWrite is EXCEPTION, which rejects the deliberately pre-epoch timestamps (the corpus reaches back to 1843) - outputTimestampType is INT96, which has its own ancient-timestamp check Set all three to Spark 4's values around the write only. Every test reads the data back from parquet, so no result comparison is affected.
1 parent f8db9b5 commit 7de2708

1 file changed

Lines changed: 21 additions & 1 deletion

File tree

‎spark/src/test/scala/org/apache/comet/CometIcebergSystemFunctionSuite.scala‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,27 @@ class CometIcebergSystemFunctionSuite
8585
override def beforeAll(): Unit = {
8686
super.beforeAll()
8787
sourceDir = Files.createTempDirectory("comet-iceberg-system-functions").toFile
88-
sourceData().write.parquet(sourcePath)
88+
// Three Spark 3.x defaults differ from Spark 4's in ways that block writing this corpus. All
89+
// are set to Spark 4's value so one corpus works on every profile, and none affects how a
90+
// result is compared, since every test reads the data back from parquet.
91+
//
92+
// - `datetimeJava8ApiEnabled`: `sourceData` supplies java.time values for the date and
93+
// timestamp columns. Spark 4 resolves those external types; on Spark 3.x the row encoder
94+
// expects java.sql.Date / java.sql.Timestamp instead. The encoder is built here on the
95+
// driver, so setting the flag around the write is enough.
96+
// - `datetimeRebaseModeInWrite`: the timestamp corpus reaches back to 1843, and the corpus
97+
// is deliberately pre-epoch in places, since the temporal transforms go negative before
98+
// 1970. Spark 3.x throws on writing a timestamp before 1900; Spark 4 defaults to
99+
// CORRECTED, which writes the value as-is.
100+
// - `outputTimestampType`: Spark 3.x defaults to INT96, which has its own separate ancient
101+
// timestamp check. Spark 4 defaults to TIMESTAMP_MICROS, which is also what Iceberg
102+
// itself writes.
103+
withSQLConf(
104+
SQLConf.DATETIME_JAVA8API_ENABLED.key -> "true",
105+
SQLConf.PARQUET_REBASE_MODE_IN_WRITE.key -> "CORRECTED",
106+
SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> "TIMESTAMP_MICROS") {
107+
sourceData().write.parquet(sourcePath)
108+
}
89109
}
90110

91111
override def afterAll(): Unit = {

0 commit comments

Comments
 (0)