Skip to content

Commit 3dc1a2d

Browse files
committed
docs: correct which paths agree on the truncated decimal
`HashPartitioning` hashes the `StaticInvoke` result directly rather than through an `UnsafeRowWriter`, so the shuffle hash is on the differing side, not the agreeing one. The Iceberg partition value does not come from this kernel at all.
1 parent fa0158d commit 3dc1a2d

2 files changed

Lines changed: 9 additions & 9 deletions

File tree

‎docs/source/user-guide/latest/iceberg.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -221,9 +221,10 @@ result can need one more digit than the column's precision allows: `truncate(10,
221221
`decimal(18,4)` value of `-99999999999999.9999` is `-100000000000000.0000`, which has 19 digits.
222222
Iceberg's Java `TruncateDecimal` hands that oversized value back unchanged, and Spark turns it into
223223
null only when the row is materialized. Comet nulls it in the kernel instead. The two therefore
224-
agree wherever the value is written into a row -- a projection, a sort key, a shuffle key, a
225-
partition value -- and differ only where the truncated decimal feeds another expression without
226-
being materialized, as in `WHERE truncate(10, v) IS NULL`.
224+
agree wherever Spark writes the value into a row -- the output of a projection, and the key a sort
225+
builds -- and differ where the truncated decimal feeds another expression directly, as in
226+
`WHERE truncate(10, v) IS NULL` or the hash behind `DISTRIBUTE BY truncate(10, v)`. Note that the
227+
Iceberg partition value itself is not affected: the writer computes it from the untruncated column.
227228

228229
This matters most for writes. A partitioned table with the default `write.distribution-mode`
229230
(`hash`) is planned with a shuffle and a local sort keyed on the partition transforms, and with

‎native/spark-expr/src/iceberg_funcs/truncate.rs‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -99,12 +99,11 @@ fn truncate_array(fn_name: &str, array: &ArrayRef, width: i32) -> Result<ArrayRe
9999
// last digit, so the result can need one more digit than the column allows. Iceberg's
100100
// `TruncateDecimal.invoke` hands that oversized `Decimal` back to Spark unchanged and
101101
// Spark nulls it only when a row is materialized (`UnsafeRowWriter` calls
102-
// `changePrecision`, which fails). Nulling it here is the same answer for every path
103-
// that writes the value into a row -- a projection, a sort key, a shuffle key, an
104-
// Iceberg partition value -- and it is the only answer available to a kernel that has
105-
// to return a `Decimal128(precision, scale)` array. The two differ where the result
106-
// feeds another expression without being materialized, e.g. `truncate(w, v) IS NULL`;
107-
// see the Iceberg user guide.
102+
// `changePrecision`, which fails). Nulling it here is the same answer everywhere Spark
103+
// materializes the value -- the output of a projection, and the key `SortExec` builds
104+
// -- and it is the only answer available to a kernel that has to return a
105+
// `Decimal128(precision, scale)` array. The two differ where the result feeds another
106+
// expression directly, e.g. `truncate(w, v) IS NULL`; see the Iceberg user guide.
108107
let truncated: Decimal128Array =
109108
array.as_primitive::<Decimal128Type>().unary_opt(|v| {
110109
let truncated = truncate_i128(v, width as i128);

0 commit comments

Comments
 (0)