Skip to content

Commit 5047c16

Browse files
authored
Fix decimal scalar multiplication and division (#8888)
Decimal scalar `Mul` and `Div` produced values at the wrong scale. They applied the operator to the raw stored integers and kept the operands' dtype, so at `decimal(10, 2)` `0.50 * 0.10` came back as `5.00` and `10.00 / 0.10` as `1.00`. `3000.00 * 3000.00` reported a false overflow even though `9000000.00` fits precision 10. `DecimalScalar::checked_binary_numeric` now follows Arrow's decimal arithmetic rules, which are already used by Decimal Add/Sub array kernels from #8724. Vortex coerces both operands to a single dtype, so Arrow's general `(p1, s1) op (p2, s2)` formulas collapse to: | operator | result precision | result scale | | --- | --- | --- | | Add, Sub | `p + 1` | `s` | | Mul | `2p + 1` | `2s` | | Div | `p + s + 4` | `s + 4` | Precision saturates at `MAX_PRECISION`. Mul is exact — the doubled scale leaves the raw product correctly scaled, so there is no rounding decision to make — while Div scales the dividend by `10^result_scale` and truncates toward zero. Because the result widens, the operation can no longer return a `DecimalScalar<'a>`: that borrows its `&'a DType` from an operand and has nowhere to put a wider type. It now returns `VortexResult<Option<Scalar>>`, where `Ok(None)` is overflow or division by zero. Mismatched operand dtypes became an error rather than a panic. Add's result now carries `p + 1`, so `stats_set::merge_sum` and the sum test helper restate the merged sum in the summands' own decimal type and treat a value that no longer fits as overflow, preserving current behavior. That also removes a latent panic in `merge_sum` when a merged sum is null. Div is the one operator whose intermediate can outgrow the widest native width: scaling the dividend by `10^result_scale` overflows `i256` once `p + result_scale` passes `MAX_PRECISION`, even for a quotient that would have fit the result precision. That is reported as an overflow and pinned by a test. The array kernels size their working width the same way, so both paths agree on which operations are representable. --------- Signed-off-by: Matt Katz <[email protected]>
1 parent a1057db commit 5047c16

8 files changed

Lines changed: 525 additions & 241 deletions

File tree

‎vortex-array/src/aggregate_fn/fns/sum/mod.rs‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -476,10 +476,14 @@ mod tests {
476476
.checked_add(&rhs.as_primitive())
477477
.map(Scalar::from)
478478
.unwrap_or_else(|| Scalar::null(sum_dtype.as_nullable())),
479-
DType::Decimal(..) => lhs
479+
// Add widens the result precision, so restate the sum in the accumulator's own
480+
// decimal type, treating a value that no longer fits as an overflow.
481+
DType::Decimal(decimal_dtype, _) => lhs
480482
.as_decimal()
481-
.checked_binary_numeric(&rhs.as_decimal(), NumericOperator::Add)
482-
.map(Scalar::from)
483+
.checked_binary_numeric(&rhs.as_decimal(), NumericOperator::Add)?
484+
.and_then(|scalar| scalar.as_decimal().decimal_value())
485+
.filter(|value| value.fits_in_precision(*decimal_dtype))
486+
.map(|value| Scalar::decimal(value, *decimal_dtype, Nullable))
483487
.unwrap_or_else(|| Scalar::null(sum_dtype.as_nullable())),
484488
_ => unreachable!("Sum will always be a decimal or a primitive dtype"),
485489
})
Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
// SPDX-License-Identifier: Apache-2.0
2+
// SPDX-FileCopyrightText: Copyright the Vortex contributors
3+
4+
//! Arrow's decimal arithmetic rules, and their evaluation over [`DecimalValue`].
5+
//!
6+
//! Vortex coerces both operands of a decimal arithmetic expression to a single [`DecimalDType`],
7+
//! so Arrow's general `(p1, s1) op (p2, s2)` formulas collapse to a function of one input type:
8+
//!
9+
//! | operator | result precision | result scale |
10+
//! | -------- | ---------------- | ------------ |
11+
//! | Add, Sub | `p + 1` | `s` |
12+
//! | Mul | `2p + 1` | `2s` |
13+
//! | Div | `p + s + 4` | `s + 4` |
14+
//!
15+
//! Precision saturates at [`MAX_PRECISION`]. Because Mul widens the scale, the product of the
16+
//! stored integers already sits at the result scale and no rounding is needed. Div instead scales
17+
//! the dividend up front and truncates toward zero, which is what Arrow does.
18+
//!
19+
//! Div is the one operator whose intermediate can outgrow the widest native width: scaling the
20+
//! dividend by `10^result_scale` overflows `i256` once `p + result_scale` passes
21+
//! [`MAX_PRECISION`], even for a quotient that would have fit the result precision. That is
22+
//! reported as an overflow. The array kernels derive their working width the same way, so both
23+
//! paths agree on which operations are representable.
24+
25+
use num_traits::CheckedAdd;
26+
use num_traits::CheckedDiv;
27+
use num_traits::CheckedMul;
28+
use num_traits::CheckedSub;
29+
use vortex_error::VortexResult;
30+
use vortex_error::vortex_bail;
31+
use vortex_error::vortex_err;
32+
33+
use crate::dtype::BigCast;
34+
use crate::dtype::DecimalDType;
35+
use crate::dtype::MAX_PRECISION;
36+
use crate::dtype::MAX_SCALE;
37+
use crate::dtype::NativeDecimalType;
38+
use crate::match_each_decimal_value_type;
39+
use crate::scalar::DecimalValue;
40+
use crate::scalar::NumericOperator;
41+
42+
/// Derive the result decimal dtype of a numeric operation over two operands of `input`.
43+
///
44+
/// # Errors
45+
///
46+
/// Returns an error if the operation has no valid result type: a Mul whose doubled scale is
47+
/// unrepresentable — above [`MAX_SCALE`], or below the `i8::MIN` floor of the scale field — or a
48+
/// Div whose precision would fall outside the legal range.
49+
pub(crate) fn decimal_numeric_result_dtype(
50+
input: DecimalDType,
51+
op: NumericOperator,
52+
) -> VortexResult<DecimalDType> {
53+
match op {
54+
NumericOperator::Add | NumericOperator::Sub => {
55+
// A p-digit Add/Sub result needs at most one carry digit:
56+
// 2 * (10^p - 1) < 10^(p + 1).
57+
Ok(DecimalDType::new(
58+
input.precision().saturating_add(1).min(MAX_PRECISION),
59+
input.scale(),
60+
))
61+
}
62+
NumericOperator::Mul => {
63+
// Doubling the scale in i8 would saturate a very negative sum into a legal-looking
64+
// scale, so widen first. The SQL standard rejects a product whose scale cannot be
65+
// represented rather than rounding it away.
66+
let result_scale = <i16 as From<i8>>::from(input.scale()) * 2;
67+
let Some(result_scale) = i8::try_from(result_scale)
68+
.ok()
69+
.filter(|scale| *scale <= MAX_SCALE)
70+
else {
71+
vortex_bail!(
72+
"output scale {result_scale} of {input} {op} {input} is outside the \
73+
representable scale range of {} to {MAX_SCALE}",
74+
i8::MIN
75+
);
76+
};
77+
let result_precision = input
78+
.precision()
79+
.saturating_add(input.precision().saturating_add(1))
80+
.min(MAX_PRECISION);
81+
DecimalDType::try_new(result_precision, result_scale)
82+
}
83+
NumericOperator::Div => {
84+
// Arrow follows Postgres and MySQL in adding a fixed four fractional digits. Its
85+
// precision formula `p1 - s1 + s2 + result_scale` simplifies to `p + result_scale`
86+
// once both operands share a dtype.
87+
let result_scale = input.scale().saturating_add(4).min(MAX_SCALE);
88+
let result_precision =
89+
<i16 as From<u8>>::from(input.precision()) + <i16 as From<i8>>::from(result_scale);
90+
let result_precision = u8::try_from(result_precision).map_err(|_| {
91+
vortex_err!(
92+
InvalidArgument:
93+
"decimal division result precision {result_precision} is invalid"
94+
)
95+
})?;
96+
DecimalDType::try_new(result_precision.min(MAX_PRECISION), result_scale)
97+
}
98+
}
99+
}
100+
101+
/// Apply `op` to two stored values of `input`, returning the result stored in `result`'s width.
102+
///
103+
/// Returns `None` if the result overflows `result`'s precision, or if `op` is a division by zero.
104+
pub(crate) fn checked_decimal_numeric(
105+
lhs: DecimalValue,
106+
rhs: DecimalValue,
107+
input: DecimalDType,
108+
result: DecimalDType,
109+
op: NumericOperator,
110+
) -> Option<DecimalValue> {
111+
let work = work_decimal_dtype(input, result, op);
112+
match_each_decimal_value_type!(DecimalType::smallest_decimal_value_type(&work), |W| {
113+
let value = checked_at_width::<W>(lhs.cast::<W>()?, rhs.cast::<W>()?, result.scale(), op)?;
114+
DecimalValue::from(value).normalize(result)
115+
})
116+
}
117+
118+
/// Pick a native width wide enough to hold every intermediate of an in-precision operation.
119+
///
120+
/// The result precision covers Add, Sub and Mul, whose intermediates are the result itself. Div
121+
/// scales the dividend (or the divisor, for a negative result scale) by `10^result_scale` before
122+
/// dividing, so it needs room for `p + |result_scale|` digits.
123+
fn work_decimal_dtype(
124+
input: DecimalDType,
125+
result: DecimalDType,
126+
op: NumericOperator,
127+
) -> DecimalDType {
128+
let precision = match op {
129+
NumericOperator::Add | NumericOperator::Sub | NumericOperator::Mul => result.precision(),
130+
NumericOperator::Div => input
131+
.precision()
132+
.saturating_add(result.scale().unsigned_abs())
133+
.max(result.precision())
134+
.min(MAX_PRECISION),
135+
};
136+
DecimalDType::new(precision, 0)
137+
}
138+
139+
fn checked_at_width<W>(lhs: W, rhs: W, result_scale: i8, op: NumericOperator) -> Option<W>
140+
where
141+
W: NativeDecimalType + CheckedAdd + CheckedSub + CheckedMul + CheckedDiv,
142+
{
143+
match op {
144+
NumericOperator::Add => lhs.checked_add(&rhs),
145+
NumericOperator::Sub => lhs.checked_sub(&rhs),
146+
// The result scale is the sum of the operand scales, so the raw product is already
147+
// correctly scaled.
148+
NumericOperator::Mul => lhs.checked_mul(&rhs),
149+
NumericOperator::Div => {
150+
// Arrow scales the quotient by `10^(result_scale - lhs_scale + rhs_scale)`, which is
151+
// `10^result_scale` for equal operand dtypes. A negative exponent scales the divisor
152+
// instead of the dividend.
153+
let factor =
154+
decimal_scale_factor::<W>(<u32 as From<u8>>::from(result_scale.unsigned_abs()))?;
155+
if result_scale >= 0 {
156+
lhs.checked_mul(&factor)?.checked_div(&rhs)
157+
} else {
158+
lhs.checked_div(&rhs.checked_mul(&factor)?)
159+
}
160+
}
161+
}
162+
}
163+
164+
/// `10^exp` at width `W`, or `None` if it is not representable there.
165+
fn decimal_scale_factor<W>(exp: u32) -> Option<W>
166+
where
167+
W: NativeDecimalType + CheckedAdd,
168+
{
169+
let max = *W::MAX_BY_PRECISION.get(usize::try_from(exp).ok()?)?;
170+
max.checked_add(&<W as BigCast>::from(1_i8)?)
171+
}

‎vortex-array/src/scalar/typed_view/decimal/dvalue.rs‎

Lines changed: 18 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,6 @@ use vortex_error::VortexResult;
1616
use vortex_error::vortex_bail;
1717
use vortex_error::vortex_err;
1818

19-
use crate::dtype::BigCast;
2019
use crate::dtype::DecimalDType;
2120
use crate::dtype::DecimalType;
2221
use crate::dtype::NativeDecimalType;
@@ -173,24 +172,12 @@ impl DecimalValue {
173172
/// `decimal_dtype`, after enforcing the dtype precision.
174173
pub(crate) fn try_from_i256(value: i256, decimal_dtype: DecimalDType) -> VortexResult<Self> {
175174
let decimal_value = Self::I256(value);
176-
if !decimal_value.fits_in_precision(decimal_dtype) {
177-
vortex_bail!(
175+
decimal_value.normalize(decimal_dtype).ok_or_else(|| {
176+
vortex_err!(
178177
"decimal value {} does not fit in precision of {}",
179178
decimal_value,
180179
decimal_dtype
181-
);
182-
}
183-
184-
let target_type = DecimalType::smallest_decimal_value_type(&decimal_dtype);
185-
match_each_decimal_value_type!(target_type, |T| {
186-
let value = <T as BigCast>::from(value).ok_or_else(|| {
187-
vortex_err!(
188-
"decimal value {} cannot be represented as {}",
189-
decimal_value,
190-
target_type
191-
)
192-
})?;
193-
Ok(Self::from(value))
180+
)
194181
})
195182
}
196183

@@ -214,19 +201,19 @@ impl DecimalValue {
214201
/// The stored value (regardless of scale) must fit within the range defined by precision.
215202
/// For precision P, the maximum absolute stored value is 10^P - 1.
216203
pub fn fits_in_precision(&self, decimal_type: DecimalDType) -> bool {
217-
// Convert to i256 for comparison
218-
let value_i256 = self.as_i256();
219-
220-
// Calculate the maximum stored value that can be represented with this precision
221-
// For precision P, the max stored value is 10^P - 1
222-
// This is independent of scale - scale only affects how we interpret the value
223-
let ten = i256::from_i128(10);
224-
let max_value = ten
225-
.checked_pow(decimal_type.precision() as _)
226-
.vortex_expect("precision must exist in i256");
227-
let min_value = -max_value;
228-
229-
value_i256 > min_value && value_i256 < max_value
204+
self.normalize(decimal_type).is_some()
205+
}
206+
207+
/// Checks the dtype precision and returns the value in the dtype's native storage width.
208+
pub(crate) fn normalize(&self, decimal_type: DecimalDType) -> Option<Self> {
209+
let value_type = DecimalType::smallest_decimal_value_type(&decimal_type);
210+
match_each_decimal_value_type!(value_type, |T| {
211+
let value = self.cast::<T>()?;
212+
// `T` is the narrowest type covering `precision`, so both tables index in bounds.
213+
let precision = usize::from(decimal_type.precision());
214+
(T::MIN_BY_PRECISION[precision] <= value && value <= T::MAX_BY_PRECISION[precision])
215+
.then_some(Self::from(value))
216+
})
230217
}
231218

232219
/// Checked addition. Returns `None` on overflow.
@@ -239,12 +226,12 @@ impl DecimalValue {
239226
checked_widening_binary_op!(self, other, CheckedSub::checked_sub)
240227
}
241228

242-
/// Checked multiplication. Returns `None` on overflow.
229+
/// Checked multiplication of the raw stored integers. Returns `None` on overflow.
243230
pub fn checked_mul(&self, other: &Self) -> Option<Self> {
244231
checked_widening_binary_op!(self, other, CheckedMul::checked_mul)
245232
}
246233

247-
/// Checked division. Returns `None` on overflow or division by zero.
234+
/// Checked division of the raw stored integers. Returns `None` on overflow or division by zero.
248235
pub fn checked_div(&self, other: &Self) -> Option<Self> {
249236
checked_widening_binary_op!(self, other, CheckedDiv::checked_div)
250237
}

‎vortex-array/src/scalar/typed_view/decimal/mod.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,11 @@
33

44
//! Definition and implementation of [`DecimalScalar`] and [`DecimalValue`].
55
6+
mod arithmetic;
67
mod dvalue;
78
mod scalar;
89

10+
pub(crate) use arithmetic::decimal_numeric_result_dtype;
911
pub use dvalue::DecimalValue;
1012
pub use scalar::DecimalScalar;
1113

‎vortex-array/src/scalar/typed_view/decimal/scalar.rs‎

Lines changed: 36 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,9 @@ use num_traits::ToPrimitive as NumToPrimitive;
1010
use vortex_error::VortexResult;
1111
use vortex_error::vortex_bail;
1212
use vortex_error::vortex_err;
13-
use vortex_error::vortex_panic;
1413

14+
use super::arithmetic::checked_decimal_numeric;
15+
use super::arithmetic::decimal_numeric_result_dtype;
1516
use crate::dtype::DType;
1617
use crate::dtype::DecimalDType;
1718
use crate::dtype::PType;
@@ -179,62 +180,58 @@ impl<'a> DecimalScalar<'a> {
179180

180181
/// Apply the (checked) operator to self and other using SQL-style null semantics.
181182
///
182-
/// If the operation overflows, None is returned.
183+
/// Both operands must share the same decimal type `(p, s)`. The result type follows Arrow's
184+
/// decimal arithmetic rules, so it is generally *wider* than the operands, and the result is
185+
/// therefore an owned [`Scalar`] rather than a view:
183186
///
184-
/// If the types are incompatible (ignoring nullability and precision/scale), an error is returned.
187+
/// | operator | result precision | result scale |
188+
/// | -------- | ---------------- | ------------ |
189+
/// | Add, Sub | `p + 1` | `s` |
190+
/// | Mul | `2p + 1` | `2s` |
191+
/// | Div | `p + s + 4` | `s + 4` |
192+
///
193+
/// Precision saturates at the maximum decimal precision. Mul is exact — the doubled scale
194+
/// leaves the raw product of the stored integers correctly scaled — while Div truncates
195+
/// toward zero.
185196
///
186197
/// If either value is null, the result is null.
187198
///
188-
/// The result will have the same decimal type (precision/scale) as `self`, and the result
189-
/// is checked to ensure it fits within the precision constraints.
199+
/// `Ok(None)` means the operation overflowed the result precision or divided by zero. Note
200+
/// that the array kernels raise an error in that situation rather than yielding null.
201+
///
202+
/// # Errors
203+
///
204+
/// Returns an error if the operands have different decimal types, or if the operation has no
205+
/// valid result type: a Mul whose doubled scale is unrepresentable — either above the maximum
206+
/// scale or below the minimum an `i8` scale can hold — or a Div whose precision would fall
207+
/// outside the legal range.
190208
pub fn checked_binary_numeric(
191209
&self,
192-
other: &DecimalScalar<'a>,
210+
other: &DecimalScalar<'_>,
193211
op: NumericOperator,
194-
) -> Option<DecimalScalar<'a>> {
212+
) -> VortexResult<Option<Scalar>> {
195213
// We could have ops between different types but need to add rules for type inference.
196214
if self.decimal_type != other.decimal_type {
197-
vortex_panic!(
215+
vortex_bail!(
198216
"decimal types must match: {} vs {}",
199217
self.decimal_type,
200218
other.decimal_type
201219
);
202220
}
203221

204-
// Use the more nullable dtype as the result type
205-
let result_dtype = if self.dtype.is_nullable() {
206-
self.dtype
207-
} else {
208-
other.dtype
209-
};
210-
211-
// Handle null cases using SQL semantics
212-
let result_value = match (self.decimal_value, other.decimal_value) {
213-
(None, _) | (_, None) => None,
214-
(Some(lhs), Some(rhs)) => {
215-
// Perform the operation
216-
let operation_result = match op {
217-
NumericOperator::Add => lhs.checked_add(&rhs),
218-
NumericOperator::Sub => lhs.checked_sub(&rhs),
219-
NumericOperator::Mul => lhs.checked_mul(&rhs),
220-
NumericOperator::Div => lhs.checked_div(&rhs),
221-
}?;
222+
let result_decimal_type = decimal_numeric_result_dtype(self.decimal_type, op)?;
223+
let nullability = self.dtype.nullability() | other.dtype.nullability();
224+
let result_dtype = DType::Decimal(result_decimal_type, nullability);
222225

223-
// Check if the result fits within the precision constraints
224-
if operation_result.fits_in_precision(self.decimal_type) {
225-
Some(operation_result)
226-
} else {
227-
// Result exceeds precision, return None (overflow)
228-
return None;
229-
}
230-
}
226+
// Handle null cases using SQL semantics.
227+
let (Some(lhs), Some(rhs)) = (self.decimal_value, other.decimal_value) else {
228+
return Ok(Some(Scalar::null(result_dtype.as_nullable())));
231229
};
232230

233-
Some(DecimalScalar {
234-
dtype: result_dtype,
235-
decimal_type: self.decimal_type,
236-
decimal_value: result_value,
237-
})
231+
Ok(
232+
checked_decimal_numeric(lhs, rhs, self.decimal_type, result_decimal_type, op)
233+
.map(|value| Scalar::decimal(value, result_decimal_type, nullability)),
234+
)
238235
}
239236
}
240237

0 commit comments

Comments
 (0)