Skip to content

Commit 0da7340

Browse files
committed
Speed up the decimal widening copy
LLVM left some widening pairs scalar, such as `i64` to `i128`. `widen_values` and `widened_buffer` widened each value with a checked cast and an `expect`, and collected through `BufferMut::extend_iter`, whose write loop has two exits. Build the buffer with `from_trusted_len_iter`, like the Arrow exporter, and fall back to zero instead of panicking. The fallback is dead when the target is at least as wide as the source, so the loop has one exit and no panic edge. `widened_buffer` now calls `widen_values`. A runtime assert guards the widths and folds away. A `const` assert does not compile, because the disabled `if const` branches of the callers still name the narrower pairs. On Apple M4, `i64` to `i128` is about 1.75x faster and `i32` to `i64` is unchanged, but widening into `i256` is 1.24x to 1.7x slower, for a reason we did not find. That widening only happens at precision 39 to 76, and once #10946 and #10905 land, most such columns decode to a `Widen`, so one loop for every width is the simpler trade. Signed-off-by: Connor Tsui <[email protected]>
1 parent 1850cab commit 0da7340

1 file changed

Lines changed: 72 additions & 14 deletions

File tree

‎vortex-array/src/arrays/decimal/utils.rs‎

Lines changed: 72 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -34,11 +34,7 @@ pub(crate) fn widened_buffer<W: NativeDecimalType>(array: &DecimalArray) -> Buff
3434
// Only storage types narrower than `W` reach here. The const condition lets the compiler
3535
// skip codegen of the other arms.
3636
if const { (T::DECIMAL_TYPE as u8) < (W::DECIMAL_TYPE as u8) } {
37-
array
38-
.buffer::<T>()
39-
.iter()
40-
.map(|v| W::from(*v).vortex_expect("widening decimal cast must succeed"))
41-
.collect()
37+
widen_values::<T, W>(&array.buffer::<T>())
4238
} else {
4339
unreachable!(
4440
"{} is not narrower than {}",
@@ -156,16 +152,30 @@ pub fn copy_to_canonical_width<F: NativeDecimalType>(
156152
Ok(BufferHandle::new_host(copy))
157153
}
158154

159-
/// Widens every value to `W`, which **must** be at least as wide as `F`. Null slots keep their
160-
/// values.
155+
/// Widens every value to `W`, which **must** be at least as wide as `F`, or this panics. Null slots
156+
/// keep their values.
161157
fn widen_values<F: NativeDecimalType, W: NativeDecimalType>(values: &[F]) -> Buffer<W> {
162-
values
163-
.iter()
164-
.map(|&value| {
165-
<W as BigCast>::from(value)
166-
.vortex_expect("W is at least as wide as F, so every value of F fits W")
167-
})
168-
.collect()
158+
// This cannot be a `const` assert: the disabled `if const` branches of `widened_buffer` and
159+
// `copy_to_canonical_width` still name narrower pairs, and rustc evaluates the inline constants
160+
// of every function they name. Both sides are constants, so the check folds away.
161+
assert!(
162+
(F::DECIMAL_TYPE as u8) <= (W::DECIMAL_TYPE as u8),
163+
"widen_values requires a target at least as wide as the source, got {} into {}",
164+
F::DECIMAL_TYPE,
165+
W::DECIMAL_TYPE
166+
);
167+
168+
// NB: Keep the `unwrap_or_default` fallback and `from_trusted_len_iter`. The fallback is dead
169+
// because `W` is at least as wide as `F`, so the write loop has one exit and no panic edge. A
170+
// `collect` writes through `BufferMut::extend_iter`, whose loop also exits when the reserved
171+
// capacity runs out, and the LLVM loop vectorizer leaves some pairs scalar there, such as `i64`
172+
// to `i128`. The accepted trade-off is that this loop is slower than `collect` when `W` is
173+
// `i256`, for an unknown reason. The `copy_to_canonical_width` benchmark covers these pairs.
174+
Buffer::from_trusted_len_iter(
175+
values
176+
.iter()
177+
.map(|&value| <W as BigCast>::from(value).unwrap_or_default()),
178+
)
169179
}
170180

171181
/// Narrows values that [`check_precision`] accepted to `W`, and writes zeros into the null slots.
@@ -320,6 +330,7 @@ mod tests {
320330
use super::converted_buffer;
321331
use super::copy_to_canonical_width;
322332
use super::valid_range;
333+
use super::widen_values;
323334
use crate::arrays::DecimalArray;
324335
use crate::dtype::BigCast;
325336
use crate::dtype::DecimalDType;
@@ -504,6 +515,53 @@ mod tests {
504515
Ok(())
505516
}
506517

518+
/// Each case widens 130 values, so a vectorized loop runs its vector body and a scalar tail.
519+
#[rstest]
520+
#[case::i8_to_i16(DecimalType::I8, DecimalType::I16)]
521+
#[case::i8_to_i32(DecimalType::I8, DecimalType::I32)]
522+
#[case::i8_to_i64(DecimalType::I8, DecimalType::I64)]
523+
#[case::i8_to_i128(DecimalType::I8, DecimalType::I128)]
524+
#[case::i8_to_i256(DecimalType::I8, DecimalType::I256)]
525+
#[case::i16_to_i32(DecimalType::I16, DecimalType::I32)]
526+
#[case::i16_to_i64(DecimalType::I16, DecimalType::I64)]
527+
#[case::i16_to_i128(DecimalType::I16, DecimalType::I128)]
528+
#[case::i16_to_i256(DecimalType::I16, DecimalType::I256)]
529+
#[case::i32_to_i64(DecimalType::I32, DecimalType::I64)]
530+
#[case::i32_to_i128(DecimalType::I32, DecimalType::I128)]
531+
#[case::i32_to_i256(DecimalType::I32, DecimalType::I256)]
532+
#[case::i64_to_i128(DecimalType::I64, DecimalType::I128)]
533+
#[case::i64_to_i256(DecimalType::I64, DecimalType::I256)]
534+
#[case::i128_to_i256(DecimalType::I128, DecimalType::I256)]
535+
fn test_widen_values_sign_extends(#[case] source: DecimalType, #[case] target: DecimalType) {
536+
match_each_decimal_value_type!(source, |F| {
537+
let small = |value: i8| {
538+
<F as BigCast>::from(value).vortex_expect("small values fit every decimal type")
539+
};
540+
let pattern = [F::MIN, small(-1), small(0), small(1), F::MAX];
541+
let values: Vec<F> = pattern.into_iter().cycle().take(130).collect();
542+
543+
match_each_decimal_value_type!(target, |W| {
544+
let expected: Vec<W> = values
545+
.iter()
546+
.map(|&value| {
547+
<W as BigCast>::from(value)
548+
.vortex_expect("W is at least as wide as F, so every value fits")
549+
})
550+
.collect();
551+
assert_eq!(
552+
widen_values::<F, W>(&values).as_slice(),
553+
expected.as_slice()
554+
);
555+
});
556+
});
557+
}
558+
559+
#[test]
560+
#[should_panic(expected = "widen_values requires a target at least as wide as the source")]
561+
fn test_widen_values_rejects_narrower_target() {
562+
widen_values::<i64, i32>(&[1, 2, 3]);
563+
}
564+
507565
/// Each case puts `10^p` inside the target width, so only the precision check can reject it.
508566
#[rstest]
509567
#[case::i16_to_i8(DecimalType::I16, 2)]

0 commit comments

Comments
 (0)