Skip to content

Commit 0a57b6a

Browse files
committed
fix: decline Iceberg's decimal truncate instead of nulling early
Iceberg's `TruncateDecimal.invoke` returns a `Decimal` that can exceed the column's declared precision, and Spark turns it into null only when the row is materialized. An Arrow `Decimal128(precision, scale)` array has no encoding for that intermediate, so the kernel nulled it during evaluation, which changes what an enclosing predicate or hash sees. Report decimal inputs as `Unsupported` so they stay with Spark. `CometStaticInvoke.getUnsupportedReasons()` now aggregates the per-function handlers' notes, since `GenerateDocs` only asks the serde registered for the expression class and the note would otherwise reach the Iceberg guide but not the compatibility page. Also drops the claim that Comet and Spark agree on the sort key. `SortExec` orders rows with `RowOrdering.create`, which evaluates the sort expression per comparison rather than through an `UnsafeRowWriter`, so Spark's sort sees the oversized decimal. That was wrong in `iceberg.md` and in the kernel comment.
1 parent ada706b commit 0a57b6a

5 files changed

Lines changed: 109 additions & 24 deletions

File tree

‎docs/source/user-guide/latest/iceberg.md‎

Lines changed: 10 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -210,21 +210,20 @@ The native kernels reproduce Iceberg's Java semantics exactly rather than approx
210210
and timestamps; UTF-8 for strings; raw bytes for binary; the minimal big-endian two's complement
211211
of the unscaled value for decimals) with 32-bit Murmur3 and masks the sign bit before taking the
212212
modulus.
213-
- `truncate` uses Java's wrapping integer arithmetic, keeps the decimal's precision and scale, and
214-
counts code points (not bytes) for strings.
213+
- `truncate` uses Java's wrapping integer arithmetic and counts code points (not bytes) for
214+
strings. Decimal inputs are the one case that stays with Spark, see below.
215215
- `years`, `months`, `days`, and `hours` are evaluated in UTC regardless of the session timezone
216216
and go negative before the epoch; `days` returns a date, the other three an int. They cover the
217217
whole `DATE` and `TIMESTAMP` domain, as Iceberg's `DateTimeUtil` does.
218218

219-
One difference is worth calling out. Truncating a negative decimal grows its magnitude, so the
220-
result can need one more digit than the column's precision allows: `truncate(10, v)` on a
221-
`decimal(18,4)` value of `-99999999999999.9999` is `-100000000000000.0000`, which has 19 digits.
222-
Iceberg's Java `TruncateDecimal` hands that oversized value back unchanged, and Spark turns it into
223-
null only when the row is materialized. Comet nulls it in the kernel instead. The two therefore
224-
agree wherever Spark writes the value into a row -- the output of a projection, and the key a sort
225-
builds -- and differ where the truncated decimal feeds another expression directly, as in
226-
`WHERE truncate(10, v) IS NULL` or the hash behind `DISTRIBUTE BY truncate(10, v)`. Note that the
227-
Iceberg partition value itself is not affected: the writer computes it from the untruncated column.
219+
`truncate` on a `decimal` column falls back to Spark. Truncating a negative decimal grows its
220+
magnitude, so the result can need one more digit than the column's precision allows:
221+
`truncate(10, v)` on a `decimal(18,4)` value of `-99999999999999.9999` is
222+
`-100000000000000.0000`, which has 19 digits. Iceberg's `TruncateDecimal` hands that oversized
223+
value back unchanged and Spark turns it into null only when the row is materialized. An Arrow
224+
`Decimal128(precision, scale)` array has no encoding for that intermediate, so a native kernel
225+
would have to null it during evaluation, which changes what an enclosing predicate or hash sees.
226+
Every other `truncate` input type, and `bucket` on decimals, runs natively.
228227

229228
This matters most for writes. A partitioned table with the default `write.distribution-mode`
230229
(`hash`) is planned with a shuffle and a local sort keyed on the partition transforms, and with

‎native/spark-expr/src/iceberg_funcs/truncate.rs‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -95,15 +95,16 @@ fn truncate_array(fn_name: &str, array: &ArrayRef, width: i32) -> Result<ArrayRe
9595
.unary::<_, Int64Type>(|v| truncate_i64(v, width as i64)),
9696
),
9797
DataType::Decimal128(precision, scale) => {
98+
// Not reachable from a Spark plan: `CometIcebergTruncate` reports decimal inputs as
99+
// `Unsupported` so they stay with Spark. The reason is this arm's only honest answer.
98100
// Truncating a negative value grows its magnitude by up to `width - 1` units of the
99101
// last digit, so the result can need one more digit than the column allows. Iceberg's
100-
// `TruncateDecimal.invoke` hands that oversized `Decimal` back to Spark unchanged and
101-
// Spark nulls it only when a row is materialized (`UnsafeRowWriter` calls
102-
// `changePrecision`, which fails). Nulling it here is the same answer everywhere Spark
103-
// materializes the value -- the output of a projection, and the key `SortExec` builds
104-
// -- and it is the only answer available to a kernel that has to return a
105-
// `Decimal128(precision, scale)` array. The two differ where the result feeds another
106-
// expression directly, e.g. `truncate(w, v) IS NULL`; see the Iceberg user guide.
102+
// `TruncateDecimal.invoke` hands that oversized `Decimal` back unchanged and Spark
103+
// nulls it only when a row is materialized, but a `Decimal128(precision, scale)` array
104+
// has nowhere to put it, and nulling during evaluation changes what an enclosing
105+
// predicate or hash sees. The arm is kept because it is what iceberg-rust's `Truncate`
106+
// is compared against in the writer's parity tests, and because the choice of gate is
107+
// serde policy that may be revisited.
107108
let truncated: Decimal128Array =
108109
array.as_primitive::<Decimal128Type>().unary_opt(|v| {
109110
let truncated = truncate_i128(v, width as i128);

‎spark/src/main/scala/org/apache/comet/serde/icebergFunctions.scala‎

Lines changed: 39 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -172,17 +172,51 @@ object CometIcebergBucket
172172
case _ => false
173173
})
174174

175-
/** `truncate(width, value)` over the types `TruncateFunction.bind` accepts. */
175+
/**
176+
* `truncate(width, value)` over the types `TruncateFunction.bind` accepts, minus decimals.
177+
*
178+
* Decimals are declined for a semantic reason rather than a missing kernel; see
179+
* [[CometIcebergTruncate.DecimalNote]].
180+
*/
176181
object CometIcebergTruncate
177182
extends CometIcebergParameterizedTransform(
178183
"iceberg_truncate",
179184
"width",
180185
{
181-
case ByteType | ShortType | IntegerType | LongType | StringType |
182-
BinaryType | _: DecimalType =>
183-
true
186+
case ByteType | ShortType | IntegerType | LongType | StringType | BinaryType => true
184187
case _ => false
185-
})
188+
}) {
189+
190+
/**
191+
* Why decimal `truncate` stays with Spark. Truncating a negative decimal grows its magnitude,
192+
* so the result can need one more digit than the column's precision allows. Iceberg's
193+
* `TruncateDecimal.invoke` hands that oversized `Decimal` back unchanged and Spark turns it
194+
* into null only when the row is materialized, whereas an Arrow `Decimal128(precision, scale)`
195+
* array has no encoding for it -- a native kernel would have to null it during evaluation,
196+
* changing what an enclosing predicate or hash sees.
197+
*/
198+
val DecimalNote: String =
199+
"Iceberg's TruncateDecimal returns a Decimal that can exceed the column's declared " +
200+
"precision, and Spark only turns that into null when the row is materialized. An Arrow " +
201+
"Decimal128(precision, scale) array cannot carry that intermediate, so a native kernel " +
202+
"would null it during evaluation and change what an enclosing predicate or hash sees."
203+
204+
// Ordered after the parameter check so that `truncate(0, decimal_col)` still reports the width
205+
// problem, which is the one that changes whether Iceberg's own ArithmeticException is raised.
206+
override def getSupportLevel(expr: StaticInvoke): SupportLevel = expr.arguments match {
207+
case Seq(parameter, value)
208+
if value.dataType.isInstanceOf[DecimalType] &&
209+
CometIcebergSystemFunctions.positiveIntLiteral(parameter).isDefined =>
210+
Unsupported(Some(DecimalNote))
211+
case _ => super.getSupportLevel(expr)
212+
}
213+
214+
override def getUnsupportedReasons(): Seq[String] = Seq(
215+
"Iceberg's `truncate(width, value)` system function on a `decimal` column. " + DecimalNote +
216+
" Truncating `-99999999999999.9999` in a `decimal(18,4)` column by a width of 10 is one " +
217+
"such value: the result has 19 digits. The other `truncate` input types, and `bucket` on " +
218+
"decimals, are unaffected.")
219+
}
186220

187221
/** Shared shape of the single-argument `years`, `months`, `days`, and `hours` transforms. */
188222
abstract class CometIcebergTemporalTransform(

‎spark/src/main/scala/org/apache/comet/serde/statics.scala‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,14 @@ object CometStaticInvoke extends CometExpressionSerde[StaticInvoke] {
6262
override def getSupportLevel(expr: StaticInvoke): SupportLevel =
6363
handlerFor(expr).map(_.getSupportLevel(expr)).getOrElse(Compatible())
6464

65+
/**
66+
* `GenerateDocs` only asks the serde registered for the expression class, which is this object,
67+
* so the per-function handlers' notes have to be collected here or they never reach the
68+
* compatibility guide.
69+
*/
70+
override def getUnsupportedReasons(): Seq[String] =
71+
staticInvokeExpressions.values.toSeq.distinct.flatMap(_.getUnsupportedReasons()).distinct
72+
6573
override def convert(
6674
expr: StaticInvoke,
6775
inputs: Seq[Attribute],

‎spark/src/test/scala/org/apache/comet/CometIcebergSystemFunctionSuite.scala‎

Lines changed: 44 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,8 @@ class CometIcebergSystemFunctionSuite
7676
private val source = "system_function_source"
7777
private val bucketColumns =
7878
Seq("i8", "i16", "i32", "i64", "dec18", "dec38", "str", "bin", "dt", "ts", "ts_ntz")
79-
private val truncateColumns = Seq("i8", "i16", "i32", "i64", "dec18", "dec38", "str", "bin")
79+
private val truncateColumns = Seq("i8", "i16", "i32", "i64", "str", "bin")
80+
private val decimalColumns = Seq("dec18", "dec38")
8081

8182
// The source data is written once per suite; every test reads the same parquet directory.
8283
private var sourceDir: File = _
@@ -133,6 +134,20 @@ class CometIcebergSystemFunctionSuite
133134
}
134135
}
135136

137+
test("truncate on a decimal falls back to Spark") {
138+
withSourceTable {
139+
// Iceberg's `TruncateDecimal` can return a value wider than the column's precision, which
140+
// Spark nulls only on materialization and a `Decimal128(p, s)` array cannot represent at
141+
// all; the expression stays with Spark rather than null early. `bucket` on the same columns
142+
// is unaffected and is covered by the bucket test above.
143+
decimalColumns.foreach { column =>
144+
checkSparkAnswerAndFallbackReason(
145+
s"SELECT $catalog.system.truncate(10, $column) FROM $source",
146+
"Decimal128(precision, scale) array cannot carry that intermediate")
147+
}
148+
}
149+
}
150+
136151
test("years, months, days, and hours match Iceberg regardless of session timezone") {
137152
withSourceTable {
138153
// Iceberg evaluates the temporal transforms in UTC; a shifted session timezone must not
@@ -285,10 +300,38 @@ class CometIcebergSystemFunctionSuite
285300
assert(level(CometIcebergTruncate, truncateInt, Literal(10), value) == Compatible())
286301
assert(level(CometIcebergTruncate, truncateInt, Literal(10), date).isInstanceOf[Unsupported])
287302

303+
// A decimal value reports its own reason, but only once the width is valid: a zero width has
304+
// to keep reporting the width, since that is what decides whether Iceberg's own
305+
// ArithmeticException is raised.
306+
val decimal = AttributeReference("dec", DecimalType(18, 4))()
307+
val decimalLevel = level(CometIcebergTruncate, truncateInt, Literal(10), decimal)
308+
assert(
309+
decimalLevel == Unsupported(Some(CometIcebergTruncate.DecimalNote)),
310+
s"unexpected support level: $decimalLevel")
311+
val zeroWidth = level(CometIcebergTruncate, truncateInt, Literal(0), decimal)
312+
assert(
313+
zeroWidth.asInstanceOf[Unsupported].notes.exists(_.contains("width must be a positive")),
314+
s"unexpected support level: $zeroWidth")
315+
// `bucket` on a decimal stays native; only `truncate` has the precision problem.
316+
val bucketDecimal =
317+
Class.forName("org.apache.iceberg.spark.functions.BucketFunction$BucketDecimal")
318+
assert(level(CometIcebergBucket, bucketDecimal, Literal(4), decimal) == Compatible())
319+
320+
// The reason also has to reach the generated compatibility guide, which only asks the serde
321+
// registered for `StaticInvoke`.
322+
assert(
323+
CometStaticInvoke
324+
.getUnsupportedReasons()
325+
.exists(_.contains(CometIcebergTruncate.DecimalNote)),
326+
"the decimal truncate note is missing from CometStaticInvoke.getUnsupportedReasons")
327+
288328
// CometStaticInvoke dispatches on (functionName, class name), so the same expressions reach
289329
// the Iceberg handlers from there too.
290330
assert(level(CometStaticInvoke, bucketInt, Literal(4), value) == Compatible())
291331
assert(level(CometStaticInvoke, bucketInt, Literal(0), value).isInstanceOf[Unsupported])
332+
assert(
333+
level(CometStaticInvoke, truncateInt, Literal(10), decimal) ==
334+
Unsupported(Some(CometIcebergTruncate.DecimalNote)))
292335
}
293336

294337
test("fallback reason for an unlisted static invoke names the declaring class") {

0 commit comments

Comments
 (0)