Skip to content

Commit d8a4952

Browse files
authored
Plan and execute common RowFn batches in one dispatch (#10675)
Tracking Issue: #9128 ## Summary Combines planning and execution for eligible RowFn batches in one type dispatch. This is the shared framework layer for #10676, which contains the primitive comparison migration. It builds on #10899, which separates batch inspection from planning and adds the regression cases this PR must keep passing. ## Changes - Decodes a direct Boolean batch straight from the borrowed inputs, after `BatchProbe` has classified all of them, so a later null literal still skips an earlier fallible expression or device transfer. Fallible decoders keep valid-row execution. - Finishes all-null, all-constant, null-free, and dense batches in the planning visit. Every other batch comes back as a `PartiallyValidBatch`, the only input that dense retry and valid-row execution accept, so every caller runs the fast paths first. - Constructs Boolean output with its validity in the shared executor. Dense execution masks any output that does not carry that validity, and a validity of the wrong length returns an error. Primitive-specific CPU and lane-width packing policies stay out of this layer. - In builds that enable AVX-512F and AVX-512BW at compile time, `collect_bool_words_multiversioned` uses the inline packing loop instead of the `#[target_feature]` copy, so its inlining no longer depends on codegen-unit placement. ## Performance This PR lowers the fixed cost of each RowFn call, so CodSpeed shows no gain for it alone. The gain shows up in #10676, which moves primitive comparison onto the RowFn executor. To isolate this PR, #11220 applied the two #10676 commits directly on `develop`. Without this PR, comparisons on small arrays are 10% to 16% slower than `develop` in CPU simulation, including the zone map pruning benchmarks, `i64_lt`, and `timestamp_gte`. The added time barely changes between 16 and 1,024 zones (18.0 µs and 17.3 µs for `int_gt`), so it is paid per call, not per row. On top of this PR, #10676 shows none of these regressions, and its walltime comparison gains are larger: `compare_u64_avx2` improves by 31% instead of 16%, and seven more comparison benchmarks improve by 12% to 18%. In the AVX-512 build, the RowFn changes moved the deferred Boolean kernel's AVX-512 packing loop out of its caller's codegen unit, so LLVM could no longer inline it. CodSpeed reported the two partially valid `deferred_bool_avx512` cases as 42% and 16% slower. With the last commit, CodSpeed no longer flags either case. Builds that detect AVX-512 at runtime compile the same code as before. #11220 also shows `compare_u8_avx512` and `compare_f32_avx512` as 14% and 10% slower without this PR, which matches the same packing problem, but that build's assembly wasn't inspected. CodSpeed on the current head flags three varbin comparison cases and `random_i8[0.01]` as 12% to 18% slower, and one `take_filter` case as 13% faster. None of them runs a RowFn. #10676 reverses the varbin and `take_filter` changes without touching their code, so they follow code placement. `random_i8[0.01]` isn't a stable signal either: its `develop` baseline moved from 99.6 µs to 119.3 µs between `bc58587` and `c9a3fbe`, and #11220 moved it 13% the other way. <details> <summary>CodSpeed results for #10676 with and without this PR</summary> The first column is #11220, which compares the #10676 commits on `develop` with `develop` (`c9a3fbe`). The second column is the #10676 report, which compares the same commits on this PR with this PR (`b0a64dd`). Against `develop` (`bc58587`), this PR flags none of these benchmarks. CodSpeed marks the walltime runners as unreliable, and the two columns come from different runs. | Benchmark | Mode | Without this PR | With this PR | | --- | --- | ---: | ---: | | `int_gt[16]` | CPU simulation | 15.6% slower | not flagged | | `int_gt[1024]` | CPU simulation | 14.5% slower | not flagged | | `float_gt[16]` | CPU simulation | 12.2% slower | not flagged | | `float_gt[1024]` | CPU simulation | 12.1% slower | not flagged | | `or_chain[16]` | CPU simulation | 12.3% slower | not flagged | | `or_chain[1024]` | CPU simulation | 11.9% slower | not flagged | | `is_not_null_pred[16]` | CPU simulation | 11.6% slower | not flagged | | `is_not_null_pred[1024]` | CPU simulation | 10.5% slower | not flagged | | `i64_lt` | CPU simulation | 11.7% slower | not flagged | | `timestamp_gte` | CPU simulation | 10.3% slower | not flagged | | `bench_compare_sliced_dict_primitive[(1000, 10000)]` | CPU simulation | 11.5% slower | not flagged | | `encode_primitives[f32, (2000, 2)]` | CPU simulation | 12.8% slower | not flagged | | `compare_u8_avx512` | Walltime | 14.4% slower | not flagged | | `compare_u8_neon` | Walltime | 13.3% slower | not flagged | | `compare_f32_avx512` | Walltime | 10.4% slower | not flagged | | `compare_float_avx2` | Walltime | 18.4% faster | 28.2% faster | | `compare_u64_avx2` | Walltime | 15.7% faster | 30.8% faster | | `lt_i64_nullable_avx2` | Walltime | not flagged | 18.4% faster | | `compare_int_constant_neon` | Walltime | not flagged | 16.2% faster | | `compare_int_eq_avx2` | Walltime | not flagged | 15.4% faster | | `compare_f32_avx2` | Walltime | not flagged | 15.4% faster | | `compare_int_constant_avx2` | Walltime | not flagged | 15.3% faster | | `compare_int_avx2` | Walltime | not flagged | 13.5% faster | | `compare_int_nullable_avx2` | Walltime | not flagged | 12.5% faster | </details> <details> <summary>AVX-512 investigation on EC2</summary> C7i metal (Xeon Platinum 8488C) with the CodSpeed AVX-512 build flags, the bench profile, one isolated CPU, and 21 paired blocks of 1,000 samples. Ratios are the median paired ratio to `develop` (`8c87c4c457`). Below 1 is faster. An A/A comparison of two identical binaries flags none of the 100 cases. | `deferred_bool_avx512` case | `develop` | Before fix | With fix | | --- | ---: | ---: | ---: | | `ConstantLhs, PartialAccepted` | 32.1 µs | 1.354 | 1.025 | | `Columns, PartialAccepted` | 37.7 µs | 1.159 | 1.018 | Before the fix, the kernel executed 15% more instructions per row. LLVM inline remarks show `collect_bool_words_avx512` inlined on `develop` and "definition is unavailable" on this branch. Across all 100 `#[cpu_features]` cases, the fix also returns `compare_string_lt` to the `develop` speed. It makes two `intersect_by_rank` cases and one `starts_with` case 3.5% to 5.6% slower, but those execute the same instructions as before, so the difference is code placement. The AVX2 build is unchanged. </details> <details> <summary>Historical EC2 framework measurements</summary> These numbers belong to an older framework replay and do not establish the performance of this revision. Ratios are the median of 21 paired candidate/baseline elapsed-time ratios. Below 1 is faster. This compares the framework alone, F `94be28140d5bd7e6b654af5e462253add2e49044`, with D `0291778db36809f77ce0219bf9850506d3cba3b6`. | Workload | C7g NEON | C6a AVX2 | C7i AVX2 build | C7i AVX-512 build | | --- | ---: | ---: | ---: | ---: | | `u64` multiply, 8,192 rows | 1.732 | 0.979 | 0.992 | 0.998 | | Nullable `i64` add, 32,768 rows | 1.006 | 1.004 | 1.154 | 1.005 | The actual framework pair order was imbalanced: D ran before F in 16 blocks and after it in five. Treat these figures as diagnostic observations, not reliable causal effect sizes. EC2 C7g (Neoverse V1), C6a (EPYC 7R13), and C7i (Xeon Platinum 8488C). Rust 1.99.1 / LLVM 23.1.1, bench profile with 16 CGUs and no LTO, MiMalloc, CPU 2 pinned. The same 455-case public-path microbenchmark suite ran serially across hosts, with 21 paired blocks, 10 ms per case, A/A calibration, and a separate control executable. These are microbenchmarks, not query results. C7i can dispatch to AVX-512 at runtime even in its AVX2 build. </details> The separately confirmed NEON comparison gains were published in #10843, which has merged. They do not measure this framework. CI passed on `b0a64dd`, including all-target, all-feature Clippy with warnings denied and the Linux, Windows, sanitizer, and Miri test jobs. The CodSpeed check fails on the flags described above. --------- Signed-off-by: Connor Tsui <[email protected]>
1 parent c15a41d commit d8a4952

11 files changed

Lines changed: 431 additions & 64 deletions

File tree

‎vortex-array/src/scalar_fn/unstable/row/batch/execute/constant.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ impl RowFnExecutionArgs<'_> {
1414
/// Execute all-constant inputs by evaluating one row and broadcasting the validated result.
1515
pub(super) fn execute_all_constant(
1616
&self,
17-
kernel: impl Fn(BorrowedRowFnArgs<'_>, &mut ExecutionCtx) -> VortexResult<ArrayRef>,
17+
kernel: impl FnOnce(BorrowedRowFnArgs<'_>, &mut ExecutionCtx) -> VortexResult<ArrayRef>,
1818
ctx: &mut ExecutionCtx,
1919
) -> VortexResult<ArrayRef> {
2020
let result = self.validate_kernel_output(

‎vortex-array/src/scalar_fn/unstable/row/batch/execute/dense.rs‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ impl RowFnExecutionArgs<'_> {
2222
/// Run every stored payload, then attach the input validity without materializing its mask.
2323
pub(super) fn execute_dense(
2424
&self,
25-
kernel: impl Fn(BorrowedRowFnArgs<'_>, &mut ExecutionCtx) -> VortexResult<ArrayRef>,
25+
kernel: impl FnOnce(BorrowedRowFnArgs<'_>, &mut ExecutionCtx) -> VortexResult<ArrayRef>,
2626
ctx: &mut ExecutionCtx,
2727
) -> VortexResult<ArrayRef> {
2828
let values = kernel(
@@ -90,10 +90,9 @@ impl RowFnExecutionArgs<'_> {
9090
.validity
9191
.execute_mask(self.batch.probe.row_count, ctx)?
9292
{
93-
// An array-backed validity can materialize to all valid even though the cheap checks
94-
// in `RowFnExecutionArgs::execute` could not prove that. The deferred error therefore
95-
// came from an observable row and remains terminal. An empty mask is both all-valid
96-
// and all-null, so preserve the all-valid behavior.
93+
// The cheap checks in `BatchInputs::try_execute_without_retry` cannot see that an
94+
// array-backed validity is all valid. If it is, a valid row caused the deferred error,
95+
// so the error is final. An empty mask counts as all valid.
9796
Mask::AllTrue(_) | Mask::AllFalse(0) => return Err(deferred_error),
9897
Mask::AllFalse(_) => return Ok(self.all_null()),
9998
Mask::Values(valid_rows) => valid_rows,

‎vortex-array/src/scalar_fn/unstable/row/batch/execute/mod.rs‎

Lines changed: 71 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -3,17 +3,21 @@
33

44
//! Selects a batch execution strategy.
55
//!
6-
//! [`RowFnExecutionArgs::execute`] handles universal fast paths, then delegates to dense or
7-
//! valid-only execution.
6+
//! [`BatchInputs::try_execute_without_retry`] finishes every batch that at most one kernel call can
7+
//! finish, inside the planning visit. The rest come back as a [`PartiallyValidBatch`] for a dense
8+
//! retry or valid-row execution. The `visitor::fused` module explains why.
89
910
use vortex_error::VortexResult;
1011
use vortex_mask::MaskValuesRef;
1112

12-
use super::RowFnExecutionArgs;
13+
use super::BatchInputs;
14+
use super::BatchPlan;
15+
use super::PartiallyValidBatch;
1316
use super::RowPolicy;
1417
use super::args::BorrowedRowFnArgs;
1518
use crate::ArrayRef;
1619
use crate::ExecutionCtx;
20+
use crate::dtype::DType;
1721
use crate::scalar_fn::unstable::row::execute::DenseAttempt;
1822
use crate::validity::Validity;
1923

@@ -25,16 +29,64 @@ mod valid_only;
2529
mod output;
2630
pub(crate) use output::finalize_kernel_output;
2731

28-
impl RowFnExecutionArgs<'_> {
29-
/// Apply constant folding and null handling around `kernel`.
32+
impl<'a> BatchInputs<'a> {
33+
/// Execute this batch under `plan` when at most one call to `kernel` can finish it. Otherwise
34+
/// return it as a [`PartiallyValidBatch`] that needs a dense retry or valid-row execution.
3035
///
31-
/// For a partially valid batch, `try_valid_rows` can avoid filtering. `Ok(None)` filters the
32-
/// inputs to the valid rows and `execute_filtered_rows` runs the kernel over them while
33-
/// writing into the original row domain. Every result is checked against the planned shape
34-
/// and dtype.
36+
/// A deferred kernel does not report which row failed, and `kernel` returns its failure as an
37+
/// error. That is correct here because this method runs a deferred kernel only when no row is
38+
/// null, so the failing row is valid.
39+
pub(crate) fn try_execute_without_retry(
40+
self,
41+
arg_dtypes: &[DType],
42+
plan: BatchPlan,
43+
kernel: impl FnOnce(BorrowedRowFnArgs<'_>, &mut ExecutionCtx) -> VortexResult<ArrayRef>,
44+
ctx: &mut ExecutionCtx,
45+
) -> VortexResult<Result<ArrayRef, PartiallyValidBatch<'a>>> {
46+
// Every output row is null, so skip the kernel. Checking `has_null_const` as well keeps
47+
// this from relying on `BatchProbe::new` to mark a null literal as `AllInvalid`.
48+
if matches!(self.probe.validity, Validity::AllInvalid) || self.probe.has_null_const {
49+
return Ok(Ok(self.with_plan(arg_dtypes, plan).all_null()));
50+
}
51+
52+
// Every input is constant and no row is null, so one row decides the whole output.
53+
// `all_batch_const` sees through extension and masked wrappers, as argument decoding does.
54+
if self.probe.row_count > 0
55+
&& self.probe.validity.definitely_no_nulls()
56+
&& self.all_batch_const()
57+
{
58+
return self
59+
.with_plan(arg_dtypes, plan)
60+
.execute_all_constant(kernel, ctx)
61+
.map(Ok);
62+
}
63+
64+
// One kernel call is safe when no row is null or the policy lets the kernel see null rows.
65+
// This check does not execute an array-backed validity, because that scans the whole mask.
66+
// Each later strategy does so only when it must.
67+
if self.probe.validity.definitely_no_nulls() || plan.policy() == RowPolicy::Dense {
68+
return self
69+
.with_plan(arg_dtypes, plan)
70+
.execute_dense(kernel, ctx)
71+
.map(Ok);
72+
}
73+
74+
// `PartiallyValidBatch::execute` attaches the plan, so attaching it here would compute the
75+
// result dtype twice.
76+
Ok(Err(PartiallyValidBatch { batch: self, plan }))
77+
}
78+
}
79+
80+
impl PartiallyValidBatch<'_> {
81+
/// Execute the batch with its planned strategy: a dense retry or valid-row execution.
82+
///
83+
/// `try_valid_rows` can run the valid rows without filtering. If it returns `Ok(None)`, the
84+
/// inputs are filtered to the valid rows, and `execute_filtered_rows` writes each result back
85+
/// at its original row. Every result is checked against the planned length and dtype.
3586
pub(crate) fn execute(
36-
&self,
37-
kernel: impl Fn(BorrowedRowFnArgs<'_>, &mut ExecutionCtx) -> VortexResult<ArrayRef>,
87+
self,
88+
arg_dtypes: &[DType],
89+
kernel: impl FnOnce(BorrowedRowFnArgs<'_>, &mut ExecutionCtx) -> VortexResult<ArrayRef>,
3890
execute_dense_attempt: impl FnOnce(
3991
BorrowedRowFnArgs<'_>,
4092
&mut ExecutionCtx,
@@ -51,40 +103,21 @@ impl RowFnExecutionArgs<'_> {
51103
) -> VortexResult<ArrayRef>,
52104
ctx: &mut ExecutionCtx,
53105
) -> VortexResult<ArrayRef> {
54-
// Strictness: an all-null batch has no observable row work. Keep the literal-constant
55-
// check explicit alongside the conjoined validity invariant.
56-
if matches!(self.batch.probe.validity, Validity::AllInvalid)
57-
|| self.batch.probe.has_null_const
58-
{
59-
return Ok(self.all_null());
60-
}
61-
62-
// All inputs are constant, and their conjoined validity proves that every row is non-null.
63-
// The constant check sees through extension and masked wrappers, just like argument
64-
// decoding.
65-
if self.batch.probe.row_count > 0
66-
&& self.batch.probe.validity.definitely_no_nulls()
67-
&& self.batch.all_batch_const()
68-
{
69-
return self.execute_all_constant(kernel, ctx);
70-
}
106+
let Self { batch, plan } = self;
107+
let args = batch.with_plan(arg_dtypes, plan);
71108

72-
// Do not resolve array-backed validity for the uncommon all-valid or all-null cases here.
73-
// That can execute and scan the full mask; each policy resolves it only when necessary.
74-
if self.batch.probe.validity.definitely_no_nulls() {
75-
return self.execute_dense(kernel, ctx);
76-
}
77-
78-
match self.plan.policy() {
79-
RowPolicy::Dense => self.execute_dense(kernel, ctx),
80-
RowPolicy::DenseWithRetry => self.execute_dense_with_retry(
109+
match args.plan.policy() {
110+
RowPolicy::Dense => {
111+
unreachable!("BatchInputs::try_execute_without_retry executes every dense batch")
112+
}
113+
RowPolicy::DenseWithRetry => args.execute_dense_with_retry(
81114
execute_dense_attempt,
82115
try_valid_rows,
83116
execute_filtered_rows,
84117
ctx,
85118
),
86119
RowPolicy::ValidOnly => {
87-
self.execute_valid_only(kernel, try_valid_rows, execute_filtered_rows, ctx)
120+
args.execute_valid_only(kernel, try_valid_rows, execute_filtered_rows, ctx)
88121
}
89122
}
90123
}

‎vortex-array/src/scalar_fn/unstable/row/batch/execute/valid_only.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ impl RowFnExecutionArgs<'_> {
1616
/// Resolve validity and try direct valid-row execution before filtered execution.
1717
pub(super) fn execute_valid_only(
1818
&self,
19-
kernel: impl Fn(BorrowedRowFnArgs<'_>, &mut ExecutionCtx) -> VortexResult<ArrayRef>,
19+
kernel: impl FnOnce(BorrowedRowFnArgs<'_>, &mut ExecutionCtx) -> VortexResult<ArrayRef>,
2020
try_valid_rows: impl FnOnce(
2121
BorrowedRowFnArgs<'_>,
2222
MaskValuesRef,

‎vortex-array/src/scalar_fn/unstable/row/batch/mod.rs‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,9 @@
1212
//! [`RowInputs`] borrows the input columns, and [`BatchProbe`] classifies them before dispatch.
1313
//! [`BatchInputs`] keeps the columns with their probe. [`BatchPlan`] carries the nullable execution
1414
//! strategy selected by a concrete dispatch. [`RowFnExecutionArgs`] applies that strategy, and
15-
//! [`BorrowedRowFnArgs`] pairs each kernel invocation with its planning metadata.
15+
//! [`BorrowedRowFnArgs`] pairs each kernel invocation with its planning metadata. A
16+
//! [`PartiallyValidBatch`] carries a batch that needs a dense retry or valid-row execution after
17+
//! the planning visit.
1618
1719
use std::borrow::Cow;
1820

@@ -67,6 +69,19 @@ pub(crate) struct BatchInputs<'a> {
6769
probe: BatchProbe,
6870
}
6971

72+
/// A batch, with its plan, that [`BatchInputs::try_execute_without_retry`] could not finish with
73+
/// one kernel call.
74+
///
75+
/// Only that method creates this type, so no caller can reach the slower strategies without trying
76+
/// the cheaper paths first.
77+
pub(crate) struct PartiallyValidBatch<'a> {
78+
/// The input columns and their probe.
79+
batch: BatchInputs<'a>,
80+
81+
/// The plan that a dispatch selected for `batch`.
82+
plan: BatchPlan,
83+
}
84+
7085
/// The same-length input columns and metadata for one row-function execution.
7186
pub(crate) struct RowFnExecutionArgs<'a> {
7287
/// The input columns and their conjoined validity.

‎vortex-array/src/scalar_fn/unstable/row/batch/planning.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,7 @@ impl<'a> BatchInputs<'a> {
119119
}
120120

121121
/// Attach the plan a dispatch selected for these inputs.
122-
pub(crate) fn with_plan<'b>(
122+
pub(super) fn with_plan<'b>(
123123
&'b self,
124124
arg_dtypes: &'b [DType],
125125
plan: BatchPlan,

‎vortex-array/src/scalar_fn/unstable/row/batch/tests.rs‎

Lines changed: 27 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1387,13 +1387,24 @@ impl RowFn for DeclaredSinkOutput {
13871387
#[derive(Clone)]
13881388
struct ChangingOutputDType {
13891389
dispatches: Arc<AtomicUsize>,
1390+
policy: RedispatchPolicy,
1391+
}
1392+
1393+
/// The policy that [`ChangingOutputDType`] selects. Both policies make a partially valid batch
1394+
/// dispatch twice, and the second dispatch drops the declared output dtype.
1395+
#[derive(Clone, Copy)]
1396+
enum RedispatchPolicy {
1397+
/// Use a deferred visit, which selects `RowPolicy::DenseWithRetry`.
1398+
DenseWithRetry,
1399+
/// Use a [`ValidOnlyI64`] input, which is not dense safe and so selects `RowPolicy::ValidOnly`.
1400+
ValidOnly,
13901401
}
13911402

13921403
impl RowFn for ChangingOutputDType {
13931404
type Options = EmptyOptions;
13941405

13951406
const ARG_NAMES: &'static [&'static str] = &["value"];
1396-
const INFALLIBLE: bool = true;
1407+
const INFALLIBLE: bool = false;
13971408

13981409
fn id(&self) -> ScalarFnId {
13991410
static ID: CachedId = CachedId::new("test.changing_output_dtype");
@@ -1412,7 +1423,12 @@ impl RowFn for ChangingOutputDType {
14121423
visitor
14131424
};
14141425

1415-
visitor.visit::<(i64,), i64>(|(value,)| value)
1426+
match self.policy {
1427+
RedispatchPolicy::DenseWithRetry => {
1428+
visitor.visit_deferred::<(i64,), i64, bool>(|(value,)| (value, false), |_| Ok(()))
1429+
}
1430+
RedispatchPolicy::ValidOnly => visitor.visit::<(ValidOnlyI64,), i64>(|(value,)| value),
1431+
}
14161432
}
14171433
}
14181434

@@ -1589,12 +1605,18 @@ fn test_declared_output_dtype_rejects_mismatched_storage() -> VortexResult<()> {
15891605
Ok(())
15901606
}
15911607

1592-
#[test]
1593-
fn test_execution_rejects_a_changed_output_dtype() -> VortexResult<()> {
1608+
#[rstest]
1609+
#[case::dense_with_retry(RedispatchPolicy::DenseWithRetry)]
1610+
#[case::valid_only(RedispatchPolicy::ValidOnly)]
1611+
fn test_execution_rejects_a_changed_output_dtype(
1612+
#[case] policy: RedispatchPolicy,
1613+
) -> VortexResult<()> {
15941614
let function = ChangingOutputDType {
15951615
dispatches: Arc::new(AtomicUsize::new(0)),
1616+
policy,
15961617
};
1597-
let input = PrimitiveArray::from_iter(vec![1_i64, 2]).into_array();
1618+
let validity = Validity::from_iter([true, false]);
1619+
let input = PrimitiveArray::new(vec![1_i64, 2], validity).into_array();
15981620
let args = VecExecutionArgs::new(vec![input], 2);
15991621
let mut ctx = SESSION.create_execution_ctx();
16001622

0 commit comments

Comments
 (0)