Skip to content

Commit 8c6a064

Browse files
committed
test: partition the Iceberg write test on the string column again
The multi-byte string column came out of the partitioned write test's spec because the test failed on the Linux CI runners with a missing data file, which never reproduced locally. The cause was iceberg-rust writing partition directory names raw where iceberg-java percent-encodes them, so a name containing multi-byte characters could not be reopened under the runners' ASCII platform charset. apache/iceberg-rust#2875 fixed that upstream and #5651 bumped the pin, which is now merged into this branch, so put the column back. The test then also covers the escaping. Dropping `truncate(1000, i64)` from the spec removes the reason for the Long.MinValue row filter as well, so the whole corpus is written again.
1 parent f2f0c70 commit 8c6a064

1 file changed

Lines changed: 9 additions & 14 deletions

File tree

‎spark/src/test/scala/org/apache/comet/CometIcebergSystemFunctionSuite.scala‎

Lines changed: 9 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -207,20 +207,15 @@ class CometIcebergSystemFunctionSuite
207207
val table = s"$catalog.db.hidden_partitioning"
208208
// No `write.distribution-mode`: Iceberg picks hash distribution for a partitioned table,
209209
// which plans a shuffle and a local sort on the partition transforms. The string column is
210-
// deliberately not a partition source: with it in the spec this test failed on the Linux
211-
// CI runners with a missing data file, which does not reproduce locally, so the multi-byte
212-
// values it would put in partition directory names stay out of the spec until that is
213-
// understood. `bucket` and `truncate` over strings are covered by the comparison, filter,
214-
// sort, and shuffle tests above.
210+
// a partition source on purpose: its values include multi-byte characters and a surrogate
211+
// pair, which end up in partition directory names. Those names were written raw until
212+
// apache/iceberg-rust#2875, which #5651 picked up, so this also covers the URL escaping
213+
// iceberg-java's `PartitionSpec.partitionToPath` applies.
215214
sql(s"""
216-
CREATE TABLE $table (i32 INT, i64 BIGINT, ts TIMESTAMP, dt DATE)
215+
CREATE TABLE $table (i32 INT, str STRING, ts TIMESTAMP, dt DATE)
217216
USING iceberg
218-
PARTITIONED BY (bucket(4, i32), truncate(1000, i64), days(ts), months(dt))""")
219-
// iceberg-rust's own truncate transform, which the writer uses for partition values,
220-
// overflows on Long.MinValue in a debug build where Java wraps
221-
// (apache/iceberg-rust#3141), so the boundary row stays out of the written set.
222-
val rows =
223-
s"SELECT i32, i64, ts, dt FROM $source WHERE i64 IS NULL OR i64 <> ${Long.MinValue}"
217+
PARTITIONED BY (bucket(4, i32), truncate(2, str), days(ts), months(dt))""")
218+
val rows = s"SELECT i32, str, ts, dt FROM $source"
224219
try {
225220
val plans = capturePlans(spark) {
226221
sql(s"INSERT INTO $table $rows")
@@ -241,14 +236,14 @@ class CometIcebergSystemFunctionSuite
241236
s"the distribution shuffle stayed on Spark:\n$plan")
242237
}
243238

244-
checkAnswer(sql(s"SELECT i32, i64, ts, dt FROM $table"), sql(rows).collect())
239+
checkAnswer(sql(s"SELECT i32, str, ts, dt FROM $table"), sql(rows).collect())
245240

246241
// Iceberg's own view of the partitions must match what the JVM transforms compute over
247242
// the written rows: one partition per distinct transform tuple.
248243
withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
249244
val expected = sql(s"""
250245
SELECT COUNT(*) FROM (
251-
SELECT DISTINCT $catalog.system.bucket(4, i32), $catalog.system.truncate(1000, i64),
246+
SELECT DISTINCT $catalog.system.bucket(4, i32), $catalog.system.truncate(2, str),
252247
$catalog.system.days(ts), $catalog.system.months(dt)
253248
FROM $table)""").collect().head.getLong(0)
254249
val actual = sql(s"SELECT COUNT(*) FROM $table.partitions").collect().head.getLong(0)

0 commit comments

Comments
 (0)