Skip to content

Commit 5973b3a

Browse files
committed
Move the claims and storage paths of the extension plan to #11115
Opacity in #11114 now lands with hardcoded opt-ins. `ExtensionUse` has only `None`, `Structural` and `Specific`, and the built-in functions answer `Specific` for fixed lists of types that match the claims of X16 (new decision X35). The claims, `ExtensionUse::Storage`, `ResultMapping`, the generic storage rewrite and the storage paths move to #11115 as CL1 to CL3 and SP1 to SP7. So #11114 no longer depends on #11115. The declarations of phase 2 now land before the typing check of phase 3. Before, the check could have rejected timestamp compares until TY1 landed. U9 wraps the results of the storage pushdown with a fixed mapping, so it no longer waits for `ResultMapping`. Signed-off-by: Connor Tsui <[email protected]>
1 parent ea1ac85 commit 5973b3a

8 files changed

Lines changed: 277 additions & 189 deletions

File tree

‎research/decisions.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,9 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors
66
# Decisions
77

88
This file records the decisions that the maintainer made while reviewing these notes on 2026-10-19,
9-
2026-10-22, 2026-10-23, 2026-10-26, 2026-10-28, 2026-10-29 and 2026-10-30. It also lists the
10-
questions that stay open. The other documents follow this file. If a document disagrees with it,
11-
this file wins.
9+
2026-10-22, 2026-10-23, 2026-10-26, 2026-10-28, 2026-10-29, 2026-10-30 and 2026-11-03. It also
10+
lists the questions that stay open. The other documents follow this file. If a document disagrees
11+
with it, this file wins.
1212

1313
"Settled" marks a choice of the maintainer. "Recommendation" marks a proposal of these notes that
1414
the maintainer has not confirmed yet.
@@ -50,9 +50,10 @@ the maintainer has not confirmed yet.
5050
| X29 | Reads and imports are strict. On read, the declared storage must equal the stored storage dtype exactly, including nested nullability. A file in another form is read only through a registered read plugin. An Arrow import of a known type converts children that are marked nullable but hold no nulls into the declared form, and fails when the data falls outside the domain of the type. The `Vector` importer fails instead of declining. An Arrow extension name that Vortex does not know still imports as plain storage | One declared form keeps every consumer simple, and backward compatibility belongs in a read plugin. The importers declined only because no robust design existed. Failing can break downstream code that relied on declining, which the maintainer accepts |
5151
| X30 | The only generic extension cast changes the top-level nullability, and a type cannot override it. Every other cast is a rule that the type owns (C1, X20). The `Rewrap` guard (X19) ignores only the top-level nullability, so a storage result with another nested nullability fails when the call is bound, with an error that names both dtypes. Binding never inserts a cast to make them match (C3) | Nested nullability is part of the type (X26). A cast that binding inserts would be implicit |
5252
| X31 | `Timestamp` accepts every `i64` count of its unit, as Arrow does, and has no value check. `Date` already accepts every storage value. `Time` keeps its limit of less than one day. Code that converts through jiff, such as `Display`, `replace_time_zone` and `TemporalMatcher::to_jiff`, returns an error or prints a fallback for a value that jiff cannot hold. PR [#10953](https://github.com/vortex-data/vortex/pull/10953) makes the change and fixes issue [#8860](https://github.com/vortex-data/vortex/issues/8860) | jiff ends 26 hours before the end of year 9999, so that it can show any timestamp in any time zone. That is a limit of formatting, not of the type. Arrow, Parquet and many SQL engines accept `9999-12-31` (EXT-V9). The limit of `Time` is part of what a time of day is |
53-
| X32 | The compare PR of phase 3 adds a rule that sends `ext_storage(DateTimeParts) < lit` to the existing `DateTimeParts` compare kernel | `TemporalScheme` is in the default compressor list (`vortex-btrblocks/src/session.rs:97`). Without the rule, a compare over `DateTimeParts` decodes the column after the storage rewrite. The move of `DateTimeParts` under the wrapper stays low priority |
53+
| X32 | The compare PR of the storage paths (SP3 in #11115) adds a rule that sends `ext_storage(DateTimeParts) < lit` to the existing `DateTimeParts` compare kernel | `TemporalScheme` is in the default compressor list (`vortex-btrblocks/src/session.rs:97`). Without the rule, a compare over `DateTimeParts` decodes the column after the storage rewrite. The move of `DateTimeParts` under the wrapper stays low priority |
5454
| X33 | Any function can move through a generic structural encoding, because those encodings know nothing about the dtype. Only `ext_storage`, `ext_storage_deep`, a default storage path backed by a claim, or a rule that the type owns can cross the `Extension` wrapper. A rewrite respects opaque types when it gives the same result for every extension type. Keeping the result dtype is not enough: `ts_a < ts_b` to `ext_storage(ts_a) < ext_storage(ts_b)` keeps `Bool`, but it needs the `Ord` claim | This is I5 of the design notes, applied to rewrites. Folding `ext_storage` over a constant, as PR [#11056](https://github.com/vortex-data/vortex/pull/11056) does, computes what `ext_storage` means, and `execute` already does the same (`vortex-array/src/scalar_fn/fns/ext_storage.rs:77-80`) |
5555
| X34 | The pushdown of functions through generic encodings is one shared mechanism, next to issue [#10748](https://github.com/vortex-data/vortex/issues/10748). `Masked` and `Sparse` get a pushdown rule for elementwise, infallible and strict functions, like the rules that `Dict`, `Chunked` and `RunEnd` already have. `ScalarFnVTable` gets a constant-fold hook, for example `reduce_constant`, that the `Constant` encoding and the `Sparse` fill value use. `ext_storage` owns only its constant hook and the unwrap rule `ext_storage(Extension(s))` to `s`. #11056 is the constant case | Probe R6: `ext_storage` passes through `Dict` and `Chunked`, decodes `Masked`, and fails on `Sparse` with extension values. Probe R8: a constant column with some nulls compresses to `Masked(Constant)`. One rule for each generic encoding serves every function, instead of one `ext_storage` case for each encoding. A strict function keeps masked rows null |
56+
| X35 | Opacity lands before claims. Tracking issue [#11114](https://github.com/vortex-data/vortex/issues/11114) turns on typing with hardcoded opt-ins: `ExtensionUse` has only `None`, `Structural` and `Specific`, and the built-in functions answer `Specific` for fixed lists of types, kept in one place. The lists match the claims of X16: the equality list holds `Date`, `Time`, `Timestamp`, `Uuid`, `Vector` and `FixedShapeTensor`, and the order list holds the same types without `Vector` and `FixedShapeTensor`. The hand-written storage paths of today do the work for those types. Tracking issue [#11115](https://github.com/vortex-data/vortex/issues/11115) then adds the claims, the `Storage` variant, `ResultMapping` and the generic storage rewrite, and replaces each list and its hand-written path with claims (settled on 2026-11-03) | Claims, `Storage` and `ResultMapping` exist only so that a type can opt in without hardcoded code. Without them, hardcoded kernels and rules give the same behavior for the types in the repository. So opacity needs nothing from the claims work, and the two can land and be reviewed separately. The lists match the claims, so each switch to claims allows and rejects the same calls. The lists name types by extension ID, because some types live in other crates, and they cannot cover a type outside the repository. Claims remove both limits |
5657

5758
X1 to X9 date from 2026-10-19, and X10 to X12 from 2026-10-29. A design pass later on 2026-10-29
5859
settled XQ2 to XQ7, added X13 to X24, and changed X2 (wording), X10 and X11. A second design pass

‎research/extension-types/02-current-problems.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,7 @@ both paths. The cast rows stay until U1.
104104
[#10566](https://github.com/vortex-data/vortex/pull/10566) and
105105
[#10616](https://github.com/vortex-data/vortex/pull/10616) add separate aggregates, such as
106106
`vortex.is_sorted[vortex.timestamp]`, that run on storage for `Date`, `Time`, `Timestamp` and
107-
`Uuid`. Plain `is_sorted` and `min_max` do not use them. In phase 3, their computation becomes the
107+
`Uuid`. Plain `is_sorted` and `min_max` do not use them. In SP4, their computation becomes the
108108
default storage path of the plain aggregates, and the separate IDs go away (X10, X21). The open
109109
draft
110110
[#10408](https://github.com/vortex-data/vortex/pull/10408) adds yet another opt-in for `between`.

‎research/extension-types/03-design.md‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -190,6 +190,13 @@ on `values`. Every extension type inside one argument must hold the claim of tha
190190
of an argument that needs another claim becomes a separate argument, for example
191191
`sort_by(payload, get_item(row, "ts"))`.
192192

193+
Opacity lands before the claims (X35). Until the claims land, the functions of group 2 answer
194+
`Specific` for hardcoded lists of types that match the claims of X16, and today's hand-written
195+
storage paths do the work for those types. Tracking issue
196+
[#11114](https://github.com/vortex-data/vortex/issues/11114) makes extension types opaque with
197+
these lists, and [#11115](https://github.com/vortex-data/vortex/issues/11115) replaces them with
198+
claims, `Storage` and the result mappings.
199+
193200
### Result mappings
194201

195202
`Storage` carries the result mapping, so the compiler forces every function that runs on storage
@@ -247,8 +254,8 @@ When `AggregateDTypes::try_new` meets an extension input, it asks the type first
247254
A partial state stays in storage form in memory. It gets the logical form when it leaves the
248255
accumulator: `finalize`, `finalize_scalar` and `to_scalar` wrap, and `partial_from_scalar`
249256
unwraps. So consumers see the same partial dtype as today, and the cost is constant for each
250-
partial. The composite IDs and `register_storage_ordered` go away. U9 adds the result mappings
251-
first and uses them in today's adapter, so its results are right before the move.
257+
partial. The composite IDs and `register_storage_ordered` go away. U9 first wraps the results of
258+
today's adapter, so its results are right before the move.
252259

253260
### Operations across dtypes
254261

@@ -345,9 +352,9 @@ The pushdown of functions through generic encodings is one shared mechanism, nex
345352

346353
Type-specific encodings are a separate question. `DateTimeParts` can move under the wrapper as an
347354
`i64` encoding, so that its compare kernel works on storage. The compressor then chooses it from
348-
the extension metadata. This has low priority. Until then, the compare PR of phase 3 adds a rule
349-
that sends `ext_storage(DateTimeParts) < lit` to the existing `DateTimeParts` compare kernel (X32),
350-
so the compare does not decode.
355+
the extension metadata. This has low priority. Until then, the compare PR of #11115 (SP3) adds a
356+
rule that sends `ext_storage(DateTimeParts) < lit` to the existing `DateTimeParts` compare kernel
357+
(X32), so the compare does not decode.
351358

352359
[`agents/design-notes.md`](agents/design-notes.md), section 6, has the details.
353360

‎research/extension-types/04-urgent-changes.md‎

Lines changed: 19 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,8 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors
66
# 4. Urgent changes
77

88
These changes fix wrong results and panics. U1 to U8 do not depend on the design in part 3, and
9-
each one fits in a small PR with a regression test. U9 needs the declaration types of the design, so
10-
it lands in phase 2 of part 5. Finding IDs refer to [`agents/findings.md`](agents/findings.md). On
9+
each one fits in a small PR with a regression test. U9 does not depend on the design either, and it
10+
can land at any time (X35). Finding IDs refer to [`agents/findings.md`](agents/findings.md). On
1111
2026-10-29, every change was still needed on `develop` at `1d09649bbd`, and the line numbers in this
1212
part are for that commit. U5 merged in #11015 on 2026-10-30. U2, U3, U4, U6, U7 and U8 merged on
1313
2026-11-02. U1 and U9 have no PR yet.
@@ -142,26 +142,26 @@ accepts every `i64` (X31, EXT-V9).
142142

143143
## U9. Storage pushdown results
144144

145-
U9 lands at the end of phase 2, after `ExtensionUse` and `ResultMapping` exist (X17, X19).
145+
U9 does not wait for `ResultMapping` (X35). `register_storage_ordered` wraps only `MinMax` and
146+
`IsSorted` (`vortex-array/src/aggregate_fn/session.rs:195-198` at `c7449d567a`), and each of them
147+
maps its result in a fixed way:
146148

147-
- `MinMax`, `Min`, `Max`, `IsSorted`, `BoundedMin` and `BoundedMax` implement `extension_use`. Each
148-
answers `Storage` with the `Ord` claim and a result mapping:
149-
- `Min` and `Max` answer `Rewrap` with the input dtype.
150-
- `MinMax` answers `Fields` with two `Rewrap`s.
151-
- `IsSorted` answers `Keep`.
152-
- `BoundedMin` and `BoundedMax` answer `Keep` over `Utf8` and `Binary` storage, where they can
153-
truncate, and `Rewrap` over other storage.
154-
- `ExtensionStoragePushdown` applies the mapping of `A` to its results (X12). This changes
149+
- `MinMax` gives both fields the input dtype.
150+
- `IsSorted` keeps its `Bool` result.
151+
- `ExtensionStoragePushdown` applies this mapping to its results (X12). This changes
155152
`return_dtype`, `finalize` and `finalize_scalar`
156-
(`vortex-array/src/aggregate_fn/storage_pushdown.rs:200`, `:299`, `:307`).
153+
(`vortex-array/src/aggregate_fn/storage_pushdown.rs:200`, `:299`, `:307`). An inner aggregate
154+
without a fixed mapping is an error when the adapter is built.
155+
- SP4 later replaces the fixed mapping with the result mappings of #11115, when it removes the
156+
adapter.
157157
- Partial states stay in storage form in memory. `to_scalar` gives them the logical form, and
158158
`partial_from_scalar` removes it (X21). So a partial that leaves the adapter has the same dtype as
159159
a partial of plain `min_max`.
160160
- Remove the docs sentence that results are not converted back to `E` (`storage_pushdown.rs:57-59`).
161161

162162
Visible change: `vortex.min_max[vortex.timestamp]` returns `Timestamp` values, like plain `min_max`.
163163
Only tests ask for the composite IDs today, so no other caller changes. The regression test compares
164-
the result of the adapter with the result of plain `min_max`. Phase 3 later moves the body of the
164+
the result of the adapter with the result of plain `min_max`. SP4 later moves the body of the
165165
adapter into aggregate binding, and the composite IDs go away (X10, X21).
166166

167167
## Coordination with open work
@@ -210,13 +210,14 @@ Three PRs added or add mechanisms for extension semantics:
210210
- [#10566](https://github.com/vortex-data/vortex/pull/10566) added `ExtensionStoragePushdown`, an
211211
aggregate adapter with a composite ID, and
212212
[#10616](https://github.com/vortex-data/vortex/pull/10616) registered it for the temporal types
213-
and `Uuid`. Both merged on 2026-10-20. U9 fixes its results. In phase 3, its body becomes the
213+
and `Uuid`. Both merged on 2026-10-20. U9 fixes its results. In SP4, its body becomes the
214214
default storage path of aggregate binding, and its composite IDs go away (X10, X21).
215215
- [#10408](https://github.com/vortex-data/vortex/pull/10408) adds `between` pushdown for
216216
`AnyTemporal`. It is an open draft.
217217

218-
Phase 3 turns the `between` pushdown into a `Storage` declaration, and it replaces the
219-
hand-written registrations of the adapter with the claims of the types.
218+
TY1 gives the `between` pushdown the hardcoded declaration of X35, and SP3 turns it into a
219+
`Storage` declaration. SP4 replaces the hand-written registrations of the adapter with the claims
220+
of the types.
220221

221222
[#10769](https://github.com/vortex-data/vortex/pull/10769) (draft) moves the aggregate
222223
implementations into a new `vortex-aggregates` crate, including the tests of the storage pushdown.
@@ -226,4 +227,5 @@ The new extension types `Duration` ([#10467](https://github.com/vortex-data/vort
226227
and `Interval` ([#10468](https://github.com/vortex-data/vortex/pull/10468)) are open. Both edit
227228
the `AnyTemporal` matcher and the Arrow, DataFusion and DuckDB converters. U1, U2 and phase 5
228229
change the same code. `Duration` claims `Eq` and `Ord`, and `Interval` claims nothing until its PR
229-
decides what interval equality means (X16).
230+
decides what interval equality means (X16). Until the claims land, `Duration` joins both hardcoded
231+
lists of X35.

0 commit comments

Comments
 (0)