Skip to content

Commit 46b0b2b

Browse files
authored
Decimal Add/Sub kernels (#8724)
Adds shared infrastructure for decimal arithmetic and native Add/Sub kernels. ### Operand alignment The execution kernel requires both operands to have the same precision and scale (nullability may differ). Addition and subtraction can then operate directly and exactly on the unscaled integers because both values use the same power-of-ten factor. Vortex's optional coercion pass can align operands with different decimal dtypes before execution. Query-engine integrations may instead insert their own planner casts. If neither layer aligns the operands, the kernel rejects them rather than performing an implicit per-kernel rescale. ### Result precision Adding or subtracting two `p`-digit integers can require one additional carry digit. This PR follows the Arrow/Hive result-type rule for already aligned operands, capped at Vortex's maximum decimal precision: ```text decimal(p, s) +/- decimal(p, s) -> decimal(min(p + 1, 76), s) ``` The scale remains unchanged because the operation is performed on integers at the same scale. For example: ```text decimal(2, 0): 60 + 60 = 120 -> decimal(3, 0) decimal(5, 2): 123.45 + 0.55 = 124.00 -> decimal(6, 2) ``` Widening the logical precision can also widen the native storage used for the result. The kernel executes at the smallest native width capable of representing the result dtype, including the important `decimal(18, s) -> decimal(19, s)` (`i64 -> i128`) and `decimal(38, s) -> decimal(39, s)` (`i128 -> i256`) transitions. Inputs may already use a wider physical representation; they are converted into the selected working width before arithmetic. At precision 76 the result dtype cannot widen to precision 77, so it stays at precision 76 and the kernel explicitly checks the mathematical result against the precision-76 bound. ### Nulls, constants, and errors - Output nullability is the union of the input nullabilities, while a lane is valid only when both input lanes are valid. - Validity is combined and materialized once. Arithmetic runs only on valid lanes, so arbitrary values underneath null lanes cannot trigger an overflow error. - Any native-width overflow or result-precision overflow on a valid lane returns an error for the operation. - Constant and all-null operands are handled directly instead of first materializing full decimal buffers. Constant folding is preserved, and empty Add/Sub inputs produce a correctly widened empty dtype without evaluating a constant value. - Decimal Mul/Div remain unsupported because they require separate result-type and rescaling rules. ### Implementation - Splits numeric execution into `numeric/{mod,primitive,decimal,tests}.rs`, mirroring the compare implementation. - Canonicalizes encoded decimal operands through `execute::<DecimalArray>`, while extracting constant and null operands directly. - Uses a checked lane loop at the result's working width, followed by an explicit precision-bound check. - Shares `widened_buffer` from `arrays/decimal` between comparison and numeric execution. ### Comparative performance Measured over arrays with ~60k elements, using same result width in Vortex and Arrow. | Case | Vortex Add | Arrow Add | Vortex Sub | Arrow Sub | Vortex speedup | | --- | ---: | ---: | ---: | ---: | ---: | | `i32`, non-null | 59.87 us | 100.4 us | 59.74 us | 100.4 us | 1.68x | | `i64`, non-null | 61.66 us | 101.1 us | 62.29 us | 101.2 us | 1.62-1.64x | | `i128`, non-null | 68.91 us | 145.4 us | 69.54 us | 145.6 us | 2.09-2.11x | | `i256`, nullable | 175.4 us | 564.7 us | 163.2 us | 561.5 us | 3.22-3.44x | --------- Signed-off-by: Matt Katz <[email protected]>
1 parent 3a098b4 commit 46b0b2b

16 files changed

Lines changed: 1642 additions & 411 deletions

File tree

‎encodings/decimal-byte-parts/src/decimal_byte_parts/compute/mod.rs‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ mod tests {
1616
use vortex_array::VortexSessionExecute;
1717
use vortex_array::array_session;
1818
use vortex_array::arrays::PrimitiveArray;
19+
use vortex_array::compute::conformance::binary_numeric::test_binary_numeric_array;
1920
use vortex_array::compute::conformance::consistency::test_array_consistency;
2021
use vortex_array::dtype::DecimalDType;
2122
use vortex_buffer::buffer;
@@ -74,4 +75,22 @@ mod tests {
7475
let ctx = &mut array_session().create_execution_ctx();
7576
test_array_consistency(&array.into_array(), ctx);
7677
}
78+
79+
#[rstest]
80+
#[case::decimal_i32(DecimalByteParts::try_new(
81+
buffer![100i32, 200, 300, 400, 500].into_array(),
82+
DecimalDType::new(10, 2)
83+
).unwrap())]
84+
#[case::decimal_nullable_i64(DecimalByteParts::try_new(
85+
PrimitiveArray::from_option_iter([Some(1000i64), None, Some(3000), Some(4000), None]).into_array(),
86+
DecimalDType::new(19, 4)
87+
).unwrap())]
88+
#[case::decimal_negative(DecimalByteParts::try_new(
89+
buffer![-100i32, -200, 300, -400, 500].into_array(),
90+
DecimalDType::new(10, 2)
91+
).unwrap())]
92+
fn test_decimal_byte_parts_binary_numeric(#[case] array: DecimalBytePartsArray) {
93+
let ctx = &mut array_session().create_execution_ctx();
94+
test_binary_numeric_array(&array.into_array(), ctx);
95+
}
7796
}

‎vortex-array/benches/binary_ops.rs‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,10 @@ use vortex_array::VortexSessionExecute;
1919
use vortex_array::array_session;
2020
use vortex_array::arrays::BoolArray;
2121
use vortex_array::arrays::ConstantArray;
22+
use vortex_array::arrays::DecimalArray;
2223
use vortex_array::arrays::PrimitiveArray;
2324
use vortex_array::builtins::ArrayBuiltins;
25+
use vortex_array::dtype::DecimalDType;
2426
use vortex_array::scalar_fn::fns::operators::Operator;
2527
use vortex_session::VortexSession;
2628

@@ -140,6 +142,22 @@ fn sub_i64_constant(bencher: Bencher) {
140142
bench_primitive(bencher, lhs, rhs, Operator::Sub);
141143
}
142144

145+
#[divan::bench]
146+
fn add_decimal_i64_nonnull(bencher: Bencher) {
147+
let lhs = decimal_i64_nonnull(0).into_array();
148+
let rhs = decimal_i64_nonnull(1_000_000).into_array();
149+
150+
bench_decimal(bencher, lhs, rhs, Operator::Add);
151+
}
152+
153+
#[divan::bench]
154+
fn add_decimal_i128_nullable(bencher: Bencher) {
155+
let lhs = decimal_i128_nullable(0, 7).into_array();
156+
let rhs = decimal_i128_nullable(1_000_000, 5).into_array();
157+
158+
bench_decimal(bencher, lhs, rhs, Operator::Add);
159+
}
160+
143161
#[divan::bench]
144162
fn eq_i64_constant(bencher: Bencher) {
145163
let lhs = primitive_nonnull(0).into_array();
@@ -176,6 +194,10 @@ fn bench_primitive(bencher: Bencher, lhs: ArrayRef, rhs: ArrayRef, operator: Ope
176194
bench_binary::<PrimitiveArray>(bencher, lhs, rhs, operator);
177195
}
178196

197+
fn bench_decimal(bencher: Bencher, lhs: ArrayRef, rhs: ArrayRef, operator: Operator) {
198+
bench_binary::<DecimalArray>(bencher, lhs, rhs, operator);
199+
}
200+
179201
fn bench_bool(bencher: Bencher, lhs: ArrayRef, rhs: ArrayRef, operator: Operator) {
180202
bench_binary::<BoolArray>(bencher, lhs, rhs, operator);
181203
}
@@ -201,6 +223,17 @@ fn primitive_nonnull(base: i64) -> PrimitiveArray {
201223
PrimitiveArray::from_iter((0..LEN as i64).map(|i| base + i))
202224
}
203225

226+
fn decimal_i64_nonnull(base: i64) -> DecimalArray {
227+
DecimalArray::from_iter::<i64, _>((0..LEN as i64).map(|i| base + i), DecimalDType::new(18, 2))
228+
}
229+
230+
fn decimal_i128_nullable(base: i128, null_every: usize) -> DecimalArray {
231+
DecimalArray::from_option_iter::<i128, _>(
232+
(0..LEN as i128).map(|i| (!(i as usize).is_multiple_of(null_every)).then_some(base + i)),
233+
DecimalDType::new(38, 2),
234+
)
235+
}
236+
204237
fn primitive_small_nonnull(offset: i64) -> PrimitiveArray {
205238
PrimitiveArray::from_iter((0..LEN as i64).map(|i| ((i + offset) % 1024) + 1))
206239
}

‎vortex-array/src/arrays/chunked/compute/mod.rs‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,12 @@ mod tests {
2121
use crate::VortexSessionExecute;
2222
use crate::array_session;
2323
use crate::arrays::ChunkedArray;
24+
use crate::arrays::DecimalArray;
2425
use crate::arrays::PrimitiveArray;
2526
use crate::compute::conformance::binary_numeric::test_binary_numeric_array;
2627
use crate::compute::conformance::consistency::test_array_consistency;
2728
use crate::dtype::DType;
29+
use crate::dtype::DecimalDType;
2830
use crate::dtype::Nullability;
2931
use crate::dtype::PType;
3032

@@ -158,6 +160,13 @@ mod tests {
158160
],
159161
DType::Primitive(PType::I32, Nullability::NonNullable),
160162
).unwrap())]
163+
#[case::chunked_decimal(ChunkedArray::try_new(
164+
vec![
165+
DecimalArray::from_iter::<i64, _>([100, 250], DecimalDType::new(10, 2)).into_array(),
166+
DecimalArray::from_iter::<i64, _>([300, 400, 500], DecimalDType::new(10, 2)).into_array(),
167+
],
168+
DType::Decimal(DecimalDType::new(10, 2), Nullability::NonNullable),
169+
).unwrap())]
161170
fn test_chunked_binary_numeric(#[case] array: ChunkedArray) {
162171
test_binary_numeric_array(
163172
&array.into_array(),

‎vortex-array/src/arrays/decimal/compute/mod.rs‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,10 @@ mod tests {
1717
use crate::VortexSessionExecute;
1818
use crate::array_session;
1919
use crate::arrays::DecimalArray;
20+
use crate::compute::conformance::binary_numeric::test_binary_numeric_array;
2021
use crate::compute::conformance::consistency::test_array_consistency;
2122
use crate::dtype::DecimalDType;
23+
use crate::dtype::NativeDecimalType;
2224
use crate::validity::Validity;
2325

2426
#[rstest]
@@ -60,4 +62,60 @@ mod tests {
6062
&mut array_session().create_execution_ctx(),
6163
);
6264
}
65+
66+
#[rstest]
67+
#[case::decimal_array(DecimalArray::new(
68+
buffer![100i128, 200i128, 300i128, 400i128, 500i128],
69+
DecimalDType::new(19, 2),
70+
Validity::NonNullable,
71+
))]
72+
#[case::decimal_nullable(DecimalArray::new(
73+
buffer![1000i128, 2000i128, 3000i128, 4000i128, 5000i128],
74+
DecimalDType::new(19, 3),
75+
Validity::from_iter([true, false, true, true, false]),
76+
))]
77+
#[case::decimal_small_precision(DecimalArray::new(
78+
buffer![10i128, 20i128, 30i128],
79+
DecimalDType::new(5, 1),
80+
Validity::NonNullable,
81+
))]
82+
#[case::decimal_narrow_storage(DecimalArray::new(
83+
buffer![10i8, 20i8, 30i8],
84+
DecimalDType::new(5, 1),
85+
Validity::NonNullable,
86+
))]
87+
#[case::decimal_widened_carry(DecimalArray::new(
88+
buffer![
89+
<i128 as NativeDecimalType>::MAX_BY_PRECISION[38],
90+
<i128 as NativeDecimalType>::MIN_BY_PRECISION[38],
91+
],
92+
DecimalDType::new(38, 0),
93+
Validity::NonNullable,
94+
))]
95+
#[case::decimal_single(DecimalArray::new(
96+
buffer![42i128],
97+
DecimalDType::new(10, 0),
98+
Validity::NonNullable,
99+
))]
100+
#[case::decimal_large_scale(DecimalArray::new(
101+
buffer![123456789012345i128, 987654321098765i128],
102+
DecimalDType::new(20, 10),
103+
Validity::NonNullable,
104+
))]
105+
#[case::decimal_negative(DecimalArray::new(
106+
buffer![-100i128, -200i128, 300i128, -400i128],
107+
DecimalDType::new(10, 2),
108+
Validity::NonNullable,
109+
))]
110+
#[case::decimal_negative_scale(DecimalArray::new(
111+
buffer![5i64, -3i64, 600i64],
112+
DecimalDType::new(10, -2),
113+
Validity::NonNullable,
114+
))]
115+
fn test_decimal_binary_numeric(#[case] array: DecimalArray) {
116+
test_binary_numeric_array(
117+
&array.into_array(),
118+
&mut array_session().create_execution_ctx(),
119+
);
120+
}
63121
}

‎vortex-array/src/arrays/decimal/utils.rs‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,30 @@
33

44
use itertools::Itertools;
55
use itertools::MinMaxResult;
6+
use vortex_buffer::Buffer;
67
use vortex_error::VortexExpect;
78

89
use crate::arrays::DecimalArray;
10+
use crate::arrays::decimal::DecimalArrayExt;
911
use crate::dtype::DecimalType;
12+
use crate::dtype::NativeDecimalType;
1013
use crate::dtype::i256;
14+
use crate::match_each_decimal_value_type;
15+
16+
/// Return the array's unscaled values widened to `W`, which must be at least as wide as the
17+
/// array's storage type.
18+
pub(crate) fn widened_buffer<W: NativeDecimalType>(array: &DecimalArray) -> Buffer<W> {
19+
if array.values_type() == W::DECIMAL_TYPE {
20+
return array.buffer::<W>();
21+
}
22+
match_each_decimal_value_type!(array.values_type(), |T| {
23+
array
24+
.buffer::<T>()
25+
.iter()
26+
.map(|v| W::from(*v).vortex_expect("widening decimal cast must succeed"))
27+
.collect()
28+
})
29+
}
1130

1231
macro_rules! try_downcast {
1332
($array:expr, from: $src:ty, to: $($dst:ty),*) => {{

0 commit comments

Comments
 (0)