Skip to content

Commit 3715f42

Browse files
authored
Let a row dispatch declare a runtime output dtype (#9581)
## Summary - Tracking Issue: #9129 Right now, a row function cannot build an output whose dtype depends on its inputs. For example, if a function can accept multiple extension types and returns the same extension type (fn T -> T), the output type needs to know about the input. Right now, the output sink has to do that as well as record how to physically store this data (which may be different from the logical data type). This PR separates logical and physical types WRT function return. The storage dtype (physical) is what the dispatched `OutputElement` or `OutputSink` builds, and stays a property of the Rust type. The output dtype (logical) is what the function returns, and each dispatch derives it from the options and the argument dtypes. ## Changes Adds `RowVisitor::with_output_dtype`. A dispatch calls it to declare the dtype that the framework labels (basically just reattaches at the end) onto the column the dispatch builds. Batch execution applies the label in `finalize_output`, after it derives nullability and masks the null rows. Sinks become **physical only**. `OutputSink` loses its `Options` type parameter, and `return_dtype` becomes a static `storage_dtype` instead. `RowVisitor` also loses `Options` parameter. ### Limitations There are still limitations. We still do not have a runtime-deciding sink parameterization. For example, if we want a `FixedSizeListSink`, we need a way for the sink to be able to represent different row widths. This is fixed in #9585 The other is that for types that constrain the values of the storage type (think UUIDv7 or even certain decimal types), we do not validate the storage values are correct (which maybe is fine). Additionally, there will need to be extra logic for things that have more complex wrapping logic (decimal would be the main example), but it's not a huge blocker. ## API Changes - `OutputSink<Options>` becomes `OutputSink` - `return_dtype(&Options) -> VortexResult<DType>` becomes `storage_dtype() -> DType` - `RowVisitor<Options>` becomes `RowVisitor` and gains `with_output_dtype`. --------- Signed-off-by: Connor Tsui <[email protected]>
1 parent fe6f44c commit 3715f42

17 files changed

Lines changed: 786 additions & 304 deletions

File tree

‎vortex-array/src/scalar_fn/fns/binary/numeric/row.rs‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ impl RowFn for NumericBinary {
6363
ScalarFnVTable::id(&Binary)
6464
}
6565

66-
fn dispatch<V: RowVisitor<Self::Options>>(
66+
fn dispatch<V: RowVisitor>(
6767
&self,
6868
op: &Self::Options,
6969
args: &[DType],
@@ -89,7 +89,7 @@ fn visit_checked<T, Op, V>(visitor: V) -> VortexResult<V::VisitResult>
8989
where
9090
T: NativePType,
9191
Op: CheckedPrimitiveOp<T>,
92-
V: RowVisitor<NumericOperator>,
92+
V: RowVisitor,
9393
{
9494
visitor.visit_deferred::<(T, T), T, Op::Fail>(
9595
|(lhs, rhs)| Op::apply(lhs, rhs),
@@ -106,7 +106,7 @@ where
106106
fn visit_div<T, V>(visitor: V) -> VortexResult<V::VisitResult>
107107
where
108108
T: CheckedArithmetic,
109-
V: RowVisitor<NumericOperator>,
109+
V: RowVisitor,
110110
{
111111
if T::PTYPE.is_float() {
112112
return visit_checked::<T, CheckedDiv, V>(visitor);

‎vortex-array/src/scalar_fn/unstable/row/batch/args.rs‎

Lines changed: 13 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -3,22 +3,22 @@
33

44
//! Execution arguments paired with the metadata selected during planning.
55
//!
6-
//! [`BorrowedRowFnArgs`] can point at original or sliced arrays while retaining the dtypes,
7-
//! output dtype, and execution policy of the original batch plan.
6+
//! [`BorrowedRowFnArgs`] can point at original or sliced arrays while retaining the dtypes and
7+
//! [`BatchPlan`] of the original batch.
88
99
use vortex_error::VortexResult;
1010
use vortex_error::vortex_err;
1111

12-
use super::RowPolicy;
12+
use super::BatchPlan;
1313
use crate::ArrayRef;
1414
use crate::dtype::DType;
1515
use crate::scalar_fn::ExecutionArgs;
1616

1717
/// A borrowed [`ExecutionArgs`] view with the metadata selected for its row function.
1818
///
19-
/// `arrays` can be sliced, while `dtypes` and `output_dtype` always describe the original planned
20-
/// batch. Keeping them together prevents an execution path from pairing an input view with
21-
/// unrelated planning metadata.
19+
/// `arrays` can be sliced, while `dtypes` and `plan` always describe the original planned batch.
20+
/// Keeping them together prevents an execution path from pairing an input view with unrelated
21+
/// planning metadata.
2222
#[derive(Clone, Copy)]
2323
pub(crate) struct BorrowedRowFnArgs<'a> {
2424
/// The input arrays for this row-function invocation.
@@ -30,11 +30,8 @@ pub(crate) struct BorrowedRowFnArgs<'a> {
3030
/// The original input dtypes used to select the row implementation.
3131
dtypes: &'a [DType],
3232

33-
/// The non-nullable dtype built by the selected output capability.
34-
output_dtype: &'a DType,
35-
36-
/// The nullable execution policy selected during planning.
37-
policy: RowPolicy,
33+
/// The plan an executing dispatch must reproduce.
34+
plan: &'a BatchPlan,
3835
}
3936

4037
impl<'a> BorrowedRowFnArgs<'a> {
@@ -43,15 +40,13 @@ impl<'a> BorrowedRowFnArgs<'a> {
4340
arrays: &'a [ArrayRef],
4441
row_count: usize,
4542
dtypes: &'a [DType],
46-
output_dtype: &'a DType,
47-
policy: RowPolicy,
43+
plan: &'a BatchPlan,
4844
) -> Self {
4945
Self {
5046
arrays,
5147
row_count,
5248
dtypes,
53-
output_dtype,
54-
policy,
49+
plan,
5550
}
5651
}
5752

@@ -60,14 +55,9 @@ impl<'a> BorrowedRowFnArgs<'a> {
6055
self.dtypes
6156
}
6257

63-
/// Return the non-nullable dtype built by the selected output capability.
64-
pub(crate) fn output_dtype(&self) -> &'a DType {
65-
self.output_dtype
66-
}
67-
68-
/// Return the nullable execution policy selected during planning.
69-
pub(crate) fn policy(&self) -> RowPolicy {
70-
self.policy
58+
/// Return the plan an executing dispatch must reproduce.
59+
pub(crate) fn plan(&self) -> &'a BatchPlan {
60+
self.plan
7161
}
7262
}
7363

‎vortex-array/src/scalar_fn/unstable/row/batch/execute/mod.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@ impl RowFnExecutionArgs {
7373
return self.execute_dense(kernel, ctx);
7474
}
7575

76-
match self.policy {
76+
match self.plan.policy() {
7777
RowPolicy::Dense => self.execute_dense(kernel, ctx),
7878
RowPolicy::DenseWithRetry => {
7979
self.execute_dense_with_retry(execute_dense_attempt, try_valid_rows, ctx)

‎vortex-array/src/scalar_fn/unstable/row/batch/execute/output.rs‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,25 +20,37 @@ impl RowFnExecutionArgs {
2020
ConstantArray::new(Scalar::null(self.result_dtype.clone()), self.row_count).into_array()
2121
}
2222

23-
/// Validate the finished output and apply the row function's logical outer nullability.
23+
/// Label the finished column, validate it, and apply the row function's logical outer
24+
/// nullability.
2425
pub(super) fn finalize_output(
2526
&self,
2627
values: ArrayRef,
2728
expected_len: usize,
2829
) -> VortexResult<ArrayRef> {
30+
// Label before validation so the checks below see the dtype this function returns. Every
31+
// batch strategy reaches this method after masking, so an empty, all-null, constant, or
32+
// partially valid batch carries the same metadata as a dense one.
33+
let values = self.plan.relabel_output(values)?;
34+
2935
validate_output(self.id, &self.result_dtype, expected_len, &values)?;
3036

3137
cast_output_nullability(&self.result_dtype, values)
3238
}
3339

34-
/// Validate the output from a row function before batch validity is attached.
40+
/// Validate the unlabelled output from a row function before batch validity is attached.
3541
pub(super) fn validate_kernel_output(
3642
&self,
3743
values: ArrayRef,
3844
expected_len: usize,
3945
ctx: &mut ExecutionCtx,
4046
) -> VortexResult<ArrayRef> {
41-
finalize_kernel_output(self.id, &self.output_dtype, expected_len, values, ctx)
47+
finalize_kernel_output(
48+
self.id,
49+
self.plan.storage_dtype(),
50+
expected_len,
51+
values,
52+
ctx,
53+
)
4254
}
4355
}
4456

‎vortex-array/src/scalar_fn/unstable/row/batch/mod.rs‎

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -49,15 +49,12 @@ pub(crate) struct RowFnExecutionArgs {
4949
/// input. Conjoining is lazy, and null handling materializes the mask only when required.
5050
validity: Validity,
5151

52-
/// The declared output dtype, widened to nullable when any input is nullable. Kernel output is
52+
/// The output dtype, widened to nullable when any input is nullable. The finished column is
5353
/// reconciled against this dtype.
5454
result_dtype: DType,
5555

56-
/// The non-nullable dtype the dispatched output capability builds, computed while planning.
57-
output_dtype: DType,
58-
59-
/// How the concrete dispatch executes nullable rows.
60-
policy: RowPolicy,
56+
/// The storage dtype, output label, and null-handling policy selected while planning.
57+
plan: BatchPlan,
6158
}
6259

6360
#[cfg(test)]

‎vortex-array/src/scalar_fn/unstable/row/batch/planning.rs‎

Lines changed: 2 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -55,8 +55,7 @@ impl RowFnExecutionArgs {
5555
arg_dtypes,
5656
validity,
5757
result_dtype,
58-
output_dtype: plan.output_dtype,
59-
policy: plan.policy,
58+
plan,
6059
})
6160
}
6261

@@ -66,12 +65,6 @@ impl RowFnExecutionArgs {
6665
arrays: &'b [ArrayRef],
6766
row_count: usize,
6867
) -> BorrowedRowFnArgs<'b> {
69-
BorrowedRowFnArgs::new(
70-
arrays,
71-
row_count,
72-
&self.arg_dtypes,
73-
&self.output_dtype,
74-
self.policy,
75-
)
68+
BorrowedRowFnArgs::new(arrays, row_count, &self.arg_dtypes, &self.plan)
7669
}
7770
}

0 commit comments

Comments
 (0)