You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 5973b3a
Browse filesBrowse the repository at this point in the historyBrowse files
Move the claims and storage paths of the extension plan to #11115
Opacity in #11114 now lands with hardcoded opt-ins. `ExtensionUse` has
only `None`, `Structural` and `Specific`, and the built-in functions
answer `Specific` for fixed lists of types that match the claims of X16
(new decision X35). The claims, `ExtensionUse::Storage`,
`ResultMapping`, the generic storage rewrite and the storage paths move
to #11115 as CL1 to CL3 and SP1 to SP7. So #11114 no longer depends on
#11115.
The declarations of phase 2 now land before the typing check of phase
3. Before, the check could have rejected timestamp compares until TY1
landed. U9 wraps the results of the storage pushdown with a fixed
mapping, so it no longer waits for `ResultMapping`.
Signed-off-by: Connor Tsui <[email protected]>
Copy file name to clipboardExpand all lines: research/decisions.md
+5-4Lines changed: 5 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,9 +6,9 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors
6
6
# Decisions
7
7
8
8
This file records the decisions that the maintainer made while reviewing these notes on 2026-10-19,
9
-
2026-10-22, 2026-10-23, 2026-10-26, 2026-10-28, 2026-10-29 and 2026-10-30. It also lists the
10
-
questions that stay open. The other documents follow this file. If a document disagrees with it,
11
-
this file wins.
9
+
2026-10-22, 2026-10-23, 2026-10-26, 2026-10-28, 2026-10-29, 2026-10-30 and 2026-11-03. It also
10
+
lists the questions that stay open. The other documents follow this file. If a document disagrees
11
+
with it, this file wins.
12
12
13
13
"Settled" marks a choice of the maintainer. "Recommendation" marks a proposal of these notes that
14
14
the maintainer has not confirmed yet.
@@ -50,9 +50,10 @@ the maintainer has not confirmed yet.
50
50
| X29 | Reads and imports are strict. On read, the declared storage must equal the stored storage dtype exactly, including nested nullability. A file in another form is read only through a registered read plugin. An Arrow import of a known type converts children that are marked nullable but hold no nulls into the declared form, and fails when the data falls outside the domain of the type. The `Vector` importer fails instead of declining. An Arrow extension name that Vortex does not know still imports as plain storage | One declared form keeps every consumer simple, and backward compatibility belongs in a read plugin. The importers declined only because no robust design existed. Failing can break downstream code that relied on declining, which the maintainer accepts |
51
51
| X30 | The only generic extension cast changes the top-level nullability, and a type cannot override it. Every other cast is a rule that the type owns (C1, X20). The `Rewrap` guard (X19) ignores only the top-level nullability, so a storage result with another nested nullability fails when the call is bound, with an error that names both dtypes. Binding never inserts a cast to make them match (C3) | Nested nullability is part of the type (X26). A cast that binding inserts would be implicit |
52
52
| X31 |`Timestamp` accepts every `i64` count of its unit, as Arrow does, and has no value check. `Date` already accepts every storage value. `Time` keeps its limit of less than one day. Code that converts through jiff, such as `Display`, `replace_time_zone` and `TemporalMatcher::to_jiff`, returns an error or prints a fallback for a value that jiff cannot hold. PR [#10953](https://github.com/vortex-data/vortex/pull/10953) makes the change and fixes issue [#8860](https://github.com/vortex-data/vortex/issues/8860)| jiff ends 26 hours before the end of year 9999, so that it can show any timestamp in any time zone. That is a limit of formatting, not of the type. Arrow, Parquet and many SQL engines accept `9999-12-31` (EXT-V9). The limit of `Time` is part of what a time of day is |
53
-
| X32 | The compare PR of phase 3 adds a rule that sends `ext_storage(DateTimeParts) < lit` to the existing `DateTimeParts` compare kernel |`TemporalScheme` is in the default compressor list (`vortex-btrblocks/src/session.rs:97`). Without the rule, a compare over `DateTimeParts` decodes the column after the storage rewrite. The move of `DateTimeParts` under the wrapper stays low priority |
53
+
| X32 | The compare PR of the storage paths (SP3 in #11115) adds a rule that sends `ext_storage(DateTimeParts) < lit` to the existing `DateTimeParts` compare kernel |`TemporalScheme` is in the default compressor list (`vortex-btrblocks/src/session.rs:97`). Without the rule, a compare over `DateTimeParts` decodes the column after the storage rewrite. The move of `DateTimeParts` under the wrapper stays low priority |
54
54
| X33 | Any function can move through a generic structural encoding, because those encodings know nothing about the dtype. Only `ext_storage`, `ext_storage_deep`, a default storage path backed by a claim, or a rule that the type owns can cross the `Extension` wrapper. A rewrite respects opaque types when it gives the same result for every extension type. Keeping the result dtype is not enough: `ts_a < ts_b` to `ext_storage(ts_a) < ext_storage(ts_b)` keeps `Bool`, but it needs the `Ord` claim | This is I5 of the design notes, applied to rewrites. Folding `ext_storage` over a constant, as PR [#11056](https://github.com/vortex-data/vortex/pull/11056) does, computes what `ext_storage` means, and `execute` already does the same (`vortex-array/src/scalar_fn/fns/ext_storage.rs:77-80`) |
55
55
| X34 | The pushdown of functions through generic encodings is one shared mechanism, next to issue [#10748](https://github.com/vortex-data/vortex/issues/10748). `Masked` and `Sparse` get a pushdown rule for elementwise, infallible and strict functions, like the rules that `Dict`, `Chunked` and `RunEnd` already have. `ScalarFnVTable` gets a constant-fold hook, for example `reduce_constant`, that the `Constant` encoding and the `Sparse` fill value use. `ext_storage` owns only its constant hook and the unwrap rule `ext_storage(Extension(s))` to `s`. #11056 is the constant case | Probe R6: `ext_storage` passes through `Dict` and `Chunked`, decodes `Masked`, and fails on `Sparse` with extension values. Probe R8: a constant column with some nulls compresses to `Masked(Constant)`. One rule for each generic encoding serves every function, instead of one `ext_storage` case for each encoding. A strict function keeps masked rows null |
56
+
| X35 | Opacity lands before claims. Tracking issue [#11114](https://github.com/vortex-data/vortex/issues/11114) turns on typing with hardcoded opt-ins: `ExtensionUse` has only `None`, `Structural` and `Specific`, and the built-in functions answer `Specific` for fixed lists of types, kept in one place. The lists match the claims of X16: the equality list holds `Date`, `Time`, `Timestamp`, `Uuid`, `Vector` and `FixedShapeTensor`, and the order list holds the same types without `Vector` and `FixedShapeTensor`. The hand-written storage paths of today do the work for those types. Tracking issue [#11115](https://github.com/vortex-data/vortex/issues/11115) then adds the claims, the `Storage` variant, `ResultMapping` and the generic storage rewrite, and replaces each list and its hand-written path with claims (settled on 2026-11-03) | Claims, `Storage` and `ResultMapping` exist only so that a type can opt in without hardcoded code. Without them, hardcoded kernels and rules give the same behavior for the types in the repository. So opacity needs nothing from the claims work, and the two can land and be reviewed separately. The lists match the claims, so each switch to claims allows and rejects the same calls. The lists name types by extension ID, because some types live in other crates, and they cannot cover a type outside the repository. Claims remove both limits |
56
57
57
58
X1 to X9 date from 2026-10-19, and X10 to X12 from 2026-10-29. A design pass later on 2026-10-29
58
59
settled XQ2 to XQ7, added X13 to X24, and changed X2 (wording), X10 and X11. A second design pass
Copy file name to clipboardExpand all lines: research/extension-types/02-current-problems.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -104,7 +104,7 @@ both paths. The cast rows stay until U1.
104
104
[#10566](https://github.com/vortex-data/vortex/pull/10566) and
105
105
[#10616](https://github.com/vortex-data/vortex/pull/10616) add separate aggregates, such as
106
106
`vortex.is_sorted[vortex.timestamp]`, that run on storage for `Date`, `Time`, `Timestamp` and
107
-
`Uuid`. Plain `is_sorted` and `min_max` do not use them. In phase 3, their computation becomes the
107
+
`Uuid`. Plain `is_sorted` and `min_max` do not use them. In SP4, their computation becomes the
108
108
default storage path of the plain aggregates, and the separate IDs go away (X10, X21). The open
109
109
draft
110
110
[#10408](https://github.com/vortex-data/vortex/pull/10408) adds yet another opt-in for `between`.
Copy file name to clipboardExpand all lines: research/extension-types/03-design.md
+12-5Lines changed: 12 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -190,6 +190,13 @@ on `values`. Every extension type inside one argument must hold the claim of tha
190
190
of an argument that needs another claim becomes a separate argument, for example
191
191
`sort_by(payload, get_item(row, "ts"))`.
192
192
193
+
Opacity lands before the claims (X35). Until the claims land, the functions of group 2 answer
194
+
`Specific` for hardcoded lists of types that match the claims of X16, and today's hand-written
195
+
storage paths do the work for those types. Tracking issue
196
+
[#11114](https://github.com/vortex-data/vortex/issues/11114) makes extension types opaque with
197
+
these lists, and [#11115](https://github.com/vortex-data/vortex/issues/11115) replaces them with
198
+
claims, `Storage` and the result mappings.
199
+
193
200
### Result mappings
194
201
195
202
`Storage` carries the result mapping, so the compiler forces every function that runs on storage
@@ -247,8 +254,8 @@ When `AggregateDTypes::try_new` meets an extension input, it asks the type first
247
254
A partial state stays in storage form in memory. It gets the logical form when it leaves the
248
255
accumulator: `finalize`, `finalize_scalar` and `to_scalar` wrap, and `partial_from_scalar`
249
256
unwraps. So consumers see the same partial dtype as today, and the cost is constant for each
250
-
partial. The composite IDs and `register_storage_ordered` go away. U9 adds the result mappings
251
-
first and uses them in today's adapter, so its results are right before the move.
257
+
partial. The composite IDs and `register_storage_ordered` go away. U9 first wraps the results of
258
+
today's adapter, so its results are right before the move.
252
259
253
260
### Operations across dtypes
254
261
@@ -345,9 +352,9 @@ The pushdown of functions through generic encodings is one shared mechanism, nex
345
352
346
353
Type-specific encodings are a separate question. `DateTimeParts` can move under the wrapper as an
347
354
`i64` encoding, so that its compare kernel works on storage. The compressor then chooses it from
348
-
the extension metadata. This has low priority. Until then, the compare PR of phase 3 adds a rule
349
-
that sends `ext_storage(DateTimeParts) < lit` to the existing `DateTimeParts` compare kernel (X32),
350
-
so the compare does not decode.
355
+
the extension metadata. This has low priority. Until then, the compare PR of #11115 (SP3) adds a
356
+
rule that sends `ext_storage(DateTimeParts) < lit` to the existing `DateTimeParts` compare kernel
357
+
(X32), so the compare does not decode.
351
358
352
359
[`agents/design-notes.md`](agents/design-notes.md), section 6, has the details.
(`vortex-array/src/aggregate_fn/storage_pushdown.rs:200`, `:299`, `:307`). An inner aggregate
154
+
without a fixed mapping is an error when the adapter is built.
155
+
- SP4 later replaces the fixed mapping with the result mappings of #11115, when it removes the
156
+
adapter.
157
157
- Partial states stay in storage form in memory. `to_scalar` gives them the logical form, and
158
158
`partial_from_scalar` removes it (X21). So a partial that leaves the adapter has the same dtype as
159
159
a partial of plain `min_max`.
160
160
- Remove the docs sentence that results are not converted back to `E` (`storage_pushdown.rs:57-59`).
161
161
162
162
Visible change: `vortex.min_max[vortex.timestamp]` returns `Timestamp` values, like plain `min_max`.
163
163
Only tests ask for the composite IDs today, so no other caller changes. The regression test compares
164
-
the result of the adapter with the result of plain `min_max`. Phase 3 later moves the body of the
164
+
the result of the adapter with the result of plain `min_max`. SP4 later moves the body of the
165
165
adapter into aggregate binding, and the composite IDs go away (X10, X21).
166
166
167
167
## Coordination with open work
@@ -210,13 +210,14 @@ Three PRs added or add mechanisms for extension semantics:
210
210
-[#10566](https://github.com/vortex-data/vortex/pull/10566) added `ExtensionStoragePushdown`, an
211
211
aggregate adapter with a composite ID, and
212
212
[#10616](https://github.com/vortex-data/vortex/pull/10616) registered it for the temporal types
213
-
and `Uuid`. Both merged on 2026-10-20. U9 fixes its results. In phase 3, its body becomes the
213
+
and `Uuid`. Both merged on 2026-10-20. U9 fixes its results. In SP4, its body becomes the
214
214
default storage path of aggregate binding, and its composite IDs go away (X10, X21).
215
215
-[#10408](https://github.com/vortex-data/vortex/pull/10408) adds `between` pushdown for
216
216
`AnyTemporal`. It is an open draft.
217
217
218
-
Phase 3 turns the `between` pushdown into a `Storage` declaration, and it replaces the
219
-
hand-written registrations of the adapter with the claims of the types.
218
+
TY1 gives the `between` pushdown the hardcoded declaration of X35, and SP3 turns it into a
219
+
`Storage` declaration. SP4 replaces the hand-written registrations of the adapter with the claims
220
+
of the types.
220
221
221
222
[#10769](https://github.com/vortex-data/vortex/pull/10769) (draft) moves the aggregate
222
223
implementations into a new `vortex-aggregates` crate, including the tests of the storage pushdown.
@@ -226,4 +227,5 @@ The new extension types `Duration` ([#10467](https://github.com/vortex-data/vort
226
227
and `Interval` ([#10468](https://github.com/vortex-data/vortex/pull/10468)) are open. Both edit
227
228
the `AnyTemporal` matcher and the Arrow, DataFusion and DuckDB converters. U1, U2 and phase 5
228
229
change the same code. `Duration` claims `Eq` and `Ord`, and `Interval` claims nothing until its PR
229
-
decides what interval equality means (X16).
230
+
decides what interval equality means (X16). Until the claims land, `Duration` joins both hardcoded
0 commit comments