You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3715f42
Browse filesBrowse the repository at this point in the historyBrowse files
Let a row dispatch declare a runtime output dtype (#9581)
## Summary
- Tracking Issue: #9129
Right now, a row function cannot build an output whose dtype depends on
its inputs. For example, if a function can accept multiple extension
types and returns the same extension type (fn T -> T), the output type
needs to know about the input. Right now, the output sink has to do that
as well as record how to physically store this data (which may be
different from the logical data type).
This PR separates logical and physical types WRT function return. The
storage dtype (physical) is what the dispatched `OutputElement` or
`OutputSink` builds, and stays a property of the Rust type. The output
dtype (logical) is what the function returns, and each dispatch derives
it from the options and the argument dtypes.
## Changes
Adds `RowVisitor::with_output_dtype`. A dispatch calls it to declare the
dtype that the framework labels (basically just reattaches at the end)
onto the column the dispatch builds.
Batch execution applies the label in `finalize_output`, after it derives
nullability and masks the null rows.
Sinks become **physical only**. `OutputSink` loses its `Options` type
parameter, and `return_dtype` becomes a static `storage_dtype` instead.
`RowVisitor` also loses `Options` parameter.
### Limitations
There are still limitations. We still do not have a runtime-deciding
sink parameterization. For example, if we want a `FixedSizeListSink`, we
need a way for the sink to be able to represent different row widths.
This is fixed in #9585
The other is that for types that constrain the values of the storage
type (think UUIDv7 or even certain decimal types), we do not validate
the storage values are correct (which maybe is fine). Additionally,
there will need to be extra logic for things that have more complex
wrapping logic (decimal would be the main example), but it's not a huge
blocker.
## API Changes
- `OutputSink<Options>` becomes `OutputSink`
- `return_dtype(&Options) -> VortexResult<DType>` becomes
`storage_dtype() -> DType`
- `RowVisitor<Options>` becomes `RowVisitor` and gains
`with_output_dtype`.
---------
Signed-off-by: Connor Tsui <[email protected]>
0 commit comments