Skip to content

Commit 45f0ed6

Browse files
committed
Keep the storage pushdown and wrap its results
ExtensionStoragePushdown stays as the way to run an aggregate on the storage of an extension type (X10). A declaration on the type registers it, in place of the register_storage_ordered calls that crates write by hand. Its results that are values of the input get the extension dtype back inside the adapter (X12). Finding EXT-A3 and the urgent change U9 record the current behavior and the fix. The lift rule of #10748 is not needed for correctness, so phase 4a has low priority, and WD5 no longer waits for its mechanism. Signed-off-by: Connor Tsui <[email protected]>
1 parent 053f392 commit 45f0ed6

10 files changed

Lines changed: 110 additions & 61 deletions

File tree

‎research/decimal/04-implementation.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -191,7 +191,7 @@ files. The column "Shared files" names the pairs that do not, so run those one a
191191
| 1 | FX1, FX3, WD1, CD1, CD2, CD3, CD4p | No dependencies. The next regular release after CD2 publishes the fixtures | CD2 calls `smallest_decimal_value_type`, which CD3 removes. The second of the two to merge adapts |
192192
| 2 | WD2, WD6, CD3b, CD7, CD5 | WD6 needs only WD1 (D35). CD7 waits for CD1 and CD3b, and it merges right after CD5. CD5 waits for the release with CD2 | CD5 and CD7: the crate doc in `encodings/decimal-byte-parts/src/lib.rs`, the tests in `assemble.rs`, and `benches/dbp_assemble.rs`. CD7 also changes the signature of `split_decimal`. WD6 and CD7: the golden snapshots of `vortex-btrblocks` |
193193
| 3 | WD3, then WD4 | One stack. Read the CodSpeed reports with the rule in the NR brief ("CodSpeed on stacked PRs") | None |
194-
| 4 | WD5, CD4 | FX3 is merged (`dcb4f50ae2`). Settle the mechanism question of #10748 before WD5, so that WD5 and phase 4a of the extension notes can share it | Both edit `vortex-array/src/arrays/widen/rules.rs` |
194+
| 4 | WD5, CD4 | FX3 is merged (`dcb4f50ae2`) | Both edit `vortex-array/src/arrays/widen/rules.rs` |
195195
| 5 | CD6 | Waits for FX1, CD4 and the release with CD2 | None |
196196
| 6 | CD8a | The last in-memory step that the wire change needs | None |
197197
| 7 | CD8b, CD8c, WR1 | Each needs only CD8a | CD8b adds a `ctx` parameter to `DecimalArray::new` and `try_new`, which CD8c and WR1 call. CD8b and CD8c both edit `vortex-array/src/builders/decimal.rs` and the same tests. Land CD8b first, or rebase the others on it |

‎research/decimal/agents/report.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1375,7 +1375,7 @@ needs an **optional** node.
13751375

13761376
| Id | Node | Kind | Notes |
13771377
| --- | --- | --- | --- |
1378-
| N1 | Extension opacity fixes | impl, urgent | The urgent changes U1 to U8 of the extension notes. They cover storage casts, lone-side storage compares, errors from reads, nullability, temporal validation, storage checks, zero values and debug output |
1378+
| N1 | Extension opacity fixes | impl, urgent | The urgent changes U1 to U9 of the extension notes. They cover storage casts, lone-side storage compares, errors from reads, nullability, temporal validation, storage checks, zero values, debug output and the results of the storage pushdown |
13791379
| N2 | Rewrite rules that the type owns | research, then impl | X4: a type implements its behavior as rewrite rules on arrays, registered with the type. There is no `ext_wrap`. X5 and X6 add the placement rule and the removal of nested extension layers. Recommendation (XQ2): the rules live on `ExtVTable`, written over `ReduceNode` |
13801380
| N3 | Derives for one dtype | impl | X2: `eq` and `ord` are properties of one dtype and never apply across dtypes. Operations across dtypes need a rule of the type (N2) or a cast by the engine (X3) |
13811381
| N4 | Cast rules (#8570) | impl | C1: a cast exists only if a rule for the pair exists. Extension types own cast rules (X4), in place of the `cast_to` and `cast_from` closures of #8570. CQ1, the default for inexact values, is open |
@@ -1594,12 +1594,12 @@ soft: the target works without the source, but it is cheaper, safer, or faster w
15941594
### 7.5 Mapping to the extension-type research
15951595

15961596
The extension-type notes in `../../extension-types/` use their own ids for urgent changes (U1 to
1597-
U8) and phases (0 to 6, with phase 4 split into 4a and 4b). The decisions file adds ids X1 to X11
1597+
U9) and phases (0 to 6, with phase 4 split into 4a and 4b). The decisions file adds ids X1 to X12
15981598
and XQ1 to XQ7. This table maps them onto the graph.
15991599

16001600
| Extension research item | Node here |
16011601
| --- | --- |
1602-
| U1 to U8, the urgent changes | N1, item P3 in section 8.3. U5 and U6 are dtype and storage checks, not the value validation of N5 |
1602+
| U1 to U9, the urgent changes | N1, item P3 in section 8.3. U5 and U6 are dtype and storage checks, not the value validation of N5 |
16031603
| U4 infallible nullability changes | Part of N1. It also removes the panic in the rebuild that edge 35 describes |
16041604
| Phase 1, contract and conformance harness | Not a separate node. Its decimal-like reference type over `i64` is an early acceptance test for N3 and N13 |
16051605
| Phase 2, typing, derives and rewrite rules (X2, X4) | N2 and N3. Part 5 of the extension notes lands them after phases 0 and 1, which the graph does not show |
@@ -1738,7 +1738,7 @@ Independent changes, in any order. The PR lists in #10701 and #10733 track their
17381738
part must land before step 4 of the in-memory change.
17391739
- **P2.** Dropped. The DBP nullability guard (section 3.3) closed without a merge as #10746,
17401740
because #10798 documents the push-down form of the rule as valid.
1741-
- **P3.** N1: the urgent extension changes U1 to U8.
1741+
- **P3.** N1: the urgent extension changes U1 to U9.
17421742
- **P4.** The integer `Widen` in memory. #10223, #10332 and #10333 closed on 2026-10-23. The
17431743
new PRs are written from scratch, with the local rewrite of #10223 and #10332 as a reference
17441744
(D38). WD1 adds `WidenArray::encode_values`, and WD6 moves the call sites to it (D28). The fix

‎research/decisions.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,11 @@ the maintainer has not confirmed yet.
2828
| X7 | `ExtensionArray::try_new` checks every non-null value. A new `ExtensionArray::new_unchecked` skips the check for kernels that keep legal values | Today arrays are never checked (EXT-V1). This is a bug |
2929
| X8 | API changes are acceptable where correctness needs them, for example fallible constructors and private enum variants | Correctness and clear abstractions matter more than the current API |
3030
| X9 | Whether a validity query raises the errors of values that it skips is out of scope here | Issue [#10313](https://github.com/vortex-data/vortex/issues/10313) owns this question, and it is not specific to extension types |
31-
| X10 | `ExtensionStoragePushdown` ([#10566](https://github.com/vortex-data/vortex/pull/10566)) and `register_storage_ordered` ([#10616](https://github.com/vortex-data/vortex/pull/10616)) are a temporary bridge. They give `is_sorted` and `min_max` the storage order of `Date`, `Time`, `Timestamp` and `Uuid` until the declarations and rules of phase 2 exist, and phase 2 replaces them | The pushdown was needed before the design existed. It differs from the design in where the opt-in lives, in the dtype of its results, and in who can register it ([part 3](extension-types/03-design.md#a-temporary-opt-in-for-aggregates)) |
31+
| X10 | `ExtensionStoragePushdown` ([#10566](https://github.com/vortex-data/vortex/pull/10566)) stays as the way to run an aggregate on the storage of an extension type. A type gets it automatically when the type declares the matching opt-in, for example order for `is_sorted` and `min_max`. This replaces the calls to `register_storage_ordered` ([#10616](https://github.com/vortex-data/vortex/pull/10616)) that crates write by hand | The adapter is correct when its conditions hold. The parts that differ from the design are the hand-written registrations, which nothing checks, and the dtype of the results (X12). [Part 3](extension-types/03-design.md#the-aggregate-opt-in-today) explains both |
3232
| X11 | An extension type author can opt in to common behavior, such as equality and order, with little code. This is the goal of derives. The `Derives` flags and the derive rules in the design notes are one sketch of a mechanism, not a decision | The mechanism is not designed yet. X2 still holds: an opt-in covers values of one dtype |
33+
| X12 | Each result of `ExtensionStoragePushdown` that is a value of the input gets the extension dtype back inside the adapter, before the adapter returns it. Today the adapter returns raw storage values | A storage result loses the metadata of the type, for example the time unit and time zone of a timestamp. Nothing wraps the result later, and plain `min_max` already wraps its results (EXT-A3) |
3334

34-
X1 to X9 date from 2026-10-19, and X10 and X11 from 2026-10-29.
35+
X1 to X9 date from 2026-10-19, and X10 to X12 from 2026-10-29.
3536

3637
### Open
3738

@@ -41,9 +42,9 @@ X1 to X9 date from 2026-10-19, and X10 and X11 from 2026-10-29.
4142
| XQ2 | Where do the rules of X4 live? | On the `ExtVTable`, so that they travel with the dtype and need no session lookup. Write them over `ReduceNode`, so that they also rewrite bound expressions for pruning |
4243
| XQ3 | Do file reads check extension values, or trust the file? | Check only what memory safety needs, trust semantic constraints, and never panic on an illegal value. Offer a strict read mode that checks values |
4344
| XQ4 | Which values can sit in null slots? | Any value. Accessors that take a child out of its parent mask it with the validity of the parent. This also allows a storage encoding to store nulls as a reserved value |
44-
| XQ5 | How does an aggregate declare its result? | As a value of its input, as a bound on its input, or as neither. Only a value gets wrapped again |
45-
| XQ6 | Does `Uuid` order by its bytes? | No recommendation yet. `register_storage_ordered(Uuid)` (#10616) already orders it by bytes in `is_sorted` and `min_max`, as a temporary bridge (X10) |
46-
| XQ7 | Which mechanism gives authors the opt-in of X11? | Start from the sketch in the design notes: flags on the dtype, which the typing check and the rules of the core functions read. Decide it together with XQ2, because both questions decide where the behavior of a type is declared |
45+
| XQ5 | How does an aggregate declare its result? | As a value of its input, as a bound on its input, or as neither. Only a value gets wrapped again. `ExtensionStoragePushdown` uses the same answer for X12 |
46+
| XQ6 | Does `Uuid` order by its bytes? | No recommendation yet. `register_storage_ordered(Uuid)` (#10616) already orders it by bytes in `is_sorted` and `min_max`, through a registration written by hand (X10) |
47+
| XQ7 | Which mechanism gives authors the opt-in of X11? | Start from the sketch in the design notes: flags on the dtype, which the typing check and the rules of the core functions read. Decide it together with XQ2, because both questions decide where the behavior of a type is declared. For aggregates, the declaration registers `ExtensionStoragePushdown` (X10), and plain `min_max` and `is_sorted` route to the registered adapter. Consumers such as zone maps and engine pushdown then need no composite IDs |
4748

4849
## Decimal and new physical types
4950

‎research/extension-types/02-current-problems.md‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,9 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors
55

66
# 2. Current problems on `develop`
77

8-
All findings are for `develop` at `5fc5cb8`. A check on 2026-10-29 found every finding still on
9-
`develop` at `1d09649bbd`. Only EXT-A1 changed, as described below. Each ID refers to an entry in
8+
All findings are for `develop` at `5fc5cb8`, except EXT-A3, which is for `develop` at
9+
`1d09649bbd`. A check on 2026-10-29 found every other finding still on `develop` at `1d09649bbd`.
10+
Only EXT-A1 changed, as described below. Each ID refers to an entry in
1011
[`agents/findings.md`](agents/findings.md), which has file and line references, reproductions and
1112
fix sketches. "Confirmed" means that a throwaway test ([`agents/probe.rs`](agents/probe.rs) or
1213
[`agents/probe_followup.rs`](agents/probe_followup.rs)) reproduced the behavior.
@@ -82,9 +83,12 @@ implementation.
8283
[#10566](https://github.com/vortex-data/vortex/pull/10566) and
8384
[#10616](https://github.com/vortex-data/vortex/pull/10616) add separate aggregates, such as
8485
`vortex.is_sorted[vortex.timestamp]`, that run on storage for `Date`, `Time`, `Timestamp` and
85-
`Uuid`. Plain `is_sorted` and `min_max` do not use them. This is a temporary bridge (X10). The
86-
open draft [#10408](https://github.com/vortex-data/vortex/pull/10408) adds yet another opt-in for
87-
`between`.
86+
`Uuid`. Plain `is_sorted` and `min_max` do not use them. The separate aggregates stay, and in
87+
phase 2 a declaration on the type registers them (X10). The open draft
88+
[#10408](https://github.com/vortex-data/vortex/pull/10408) adds yet another opt-in for `between`.
89+
- **Storage pushdown results (EXT-A3).** The separate aggregates return storage values. The min of a
90+
`Timestamp(ms)` column is a raw `i64`, while plain `min_max` returns a `Timestamp`. Nothing wraps
91+
the result later (X12, U9).
8892
- **Zone maps (EXT-A2).** When a statistic rejects an extension dtype, the zone map uses the storage
8993
dtype. JSON and WKB columns store exact, unbounded bounds, so WKB zone maps hold whole geometry
9094
blobs.

‎research/extension-types/03-design.md‎

Lines changed: 28 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -114,32 +114,34 @@ So `ts < 5i64` stays rejected, because only one side is an extension.
114114
| `Json`, WKB, native geometry | None |
115115
| Unknown type | None |
116116

117-
### A temporary opt-in for aggregates
117+
### The aggregate opt-in today
118118

119-
`develop` already has a narrower opt-in for aggregates, from
119+
`develop` already has an opt-in for aggregates, from
120120
[#10566](https://github.com/vortex-data/vortex/pull/10566) and
121121
[#10616](https://github.com/vortex-data/vortex/pull/10616). `ExtensionStoragePushdown<A, E>` runs
122-
aggregate `A` on the storage of extension type `E`, and `register_storage_ordered` registers it for
123-
`is_sorted` and `min_max`. The default session registers it for `Date`, `Time` and `Timestamp`, and
124-
`vortex_uuid::initialize` registers it for `Uuid`. It is a temporary bridge until phase 2 (X10),
125-
and it differs from this design in three ways:
126-
127-
- The opt-in lives in the aggregate session, under a separate aggregate ID such as
128-
`vortex.min_max[vortex.timestamp]`. Plain `min_max` and `is_sorted` do not consult it, so plain
129-
`min_max` still uses storage order for every extension type, and plain `is_sorted` still rejects
130-
every extension type. A consumer gets the opt-in only if it asks for the composite ID, and today
131-
only tests do. In this design, the type declares its behavior once, and every consumer reads
132-
that declaration through the typing check and the rules.
133-
- Its results are storage values. The min of a `Timestamp(ms)` column is a raw `i64`, without its
134-
time unit or time zone. In this design, a result that is a value of the input gets the extension
135-
dtype back (XQ5). Storage results are correct only for statistics, which store storage values
136-
(section "Statistics").
137-
- Any crate can register any pair of an aggregate and a type, and nothing checks the pair. The docs
138-
of the pushdown say that an invalid registration "can silently return incorrect results". In this
139-
design, only the type declares what it inherits.
140-
141-
Phase 2 replaces each registration with a declaration on the type, and the composite IDs then go
142-
away.
122+
aggregate `A` on the storage of extension type `E`, under a separate aggregate ID such as
123+
`vortex.min_max[vortex.timestamp]`. `register_storage_ordered` registers it for `is_sorted` and
124+
`min_max`. The default session calls it for `Date`, `Time` and `Timestamp`, and
125+
`vortex_uuid::initialize` calls it for `Uuid`.
126+
127+
The adapter fits this design, and it stays (X10). It is how a type with an order opt-in gets
128+
`is_sorted` and `min_max` on its storage. Two parts change:
129+
130+
- Registration. Today a crate calls `register_storage_ordered` by hand, and nothing checks the call.
131+
The docs of the adapter say that an invalid registration "can silently return incorrect results".
132+
In this design, a declaration on the type registers the adapter automatically. The type author
133+
then makes the claim once, as for any other opt-in. XQ7 decides which declaration does this.
134+
- Results. Today the results are storage values. The min of a `Timestamp(ms)` column is a raw
135+
`i64`, without its time unit or time zone. Nothing wraps the result later, and plain `min_max`
136+
already returns extension values. The adapter must give each result that is a value of the input
137+
the extension dtype back, before it returns the result (X12, U9). The `Bool` of `is_sorted` stays
138+
as it is.
139+
140+
One question stays open. Plain `min_max` and `is_sorted` do not consult the adapter, so a consumer
141+
gets the opt-in only if it asks for the composite ID, and today only tests do. Meanwhile plain
142+
`min_max` uses storage order for every extension type, and plain `is_sorted` rejects every extension
143+
type. These notes recommend that the plain aggregates route to the registered adapter, so that
144+
consumers such as zone maps and engine pushdown need no composite IDs (XQ7).
143145

144146
### Operations across dtypes
145147

@@ -220,9 +222,9 @@ get support from the same method.
220222

221223
The lift rule is the extension case of issue
222224
[#10748](https://github.com/vortex-data/vortex/issues/10748), which keeps generic structural
223-
encodings above every other encoding. `Widen` needs the same rule (D25), and WD5 plans it for
224-
`Widen` only. If the method above serves every wrapping encoding, `Widen`, `Extension` and later
225-
wrappers share one mechanism, and no wrapper needs its own copy of the seven rules.
225+
encodings above every other encoding. Nothing depends on it for correctness. Without it, fast paths
226+
that look only at the root of an array, such as dictionary export and `as_constant`, fall back to
227+
slower paths with the same results. It has low priority.
226228

227229
Type-specific encodings are a separate question. `DateTimeParts` moves under the wrapper as an
228230
`i64` encoding, so its compare kernel works on storage. The compressor chooses it from the

‎research/extension-types/04-urgent-changes.md‎

Lines changed: 23 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ fits in a small PR with a regression test. Finding IDs refer to
2020
| U6 | Check the storage dtype of arrays and scalars | EXT-V2, EXT-V7 | An array or a scalar with the wrong storage is rejected |
2121
| U7 | Stop invalid zero values | EXT-V3 | No invalid scalars |
2222
| U8 | Fix `Debug for ExtDTypeRef` | EXT-R1 | Correct debug output |
23+
| U9 | Wrap the results of the storage pushdown | EXT-A3 | `vortex.min_max[vortex.timestamp]` returns timestamps, not `i64` values |
2324

2425
## Test requirement
2526

@@ -128,6 +129,24 @@ when the dtype is built is the better place.
128129
- Format the metadata through a `Debug` adapter, like the existing `MetadataDisplay`
129130
(`vortex-array/src/dtype/extension/erased.rs:181-191`).
130131

132+
## U9. Storage pushdown results
133+
134+
- `ExtensionStoragePushdown` gives each result of `A` that is a value of the input the extension
135+
dtype `E` back, before it returns the result (X12). This changes `return_dtype`, `finalize` and
136+
`finalize_scalar` (`vortex-array/src/aggregate_fn/storage_pushdown.rs:200`, `:299`, `:307`). For
137+
`min_max`, the `min` and `max` fields become values of `E`. The `Bool` of `is_sorted` stays as
138+
it is.
139+
- The adapter must know which parts of a result are values of the input. A method on
140+
`AggregateFnVTable` can say this, with a default of "none". `MinMax` declares both of its fields.
141+
This is a narrow form of XQ5, so the method must leave room for the bound kind.
142+
- Partial states can stay in storage form. The docs of the adapter allow partials to combine only
143+
when the aggregate, the options and the full input dtype match.
144+
- Remove the docs sentence that results are not converted back to `E` (`storage_pushdown.rs:57-59`).
145+
146+
Visible change: `vortex.min_max[vortex.timestamp]` returns `Timestamp` values, like plain `min_max`.
147+
Only tests ask for the composite IDs today, so no other caller changes. The regression test compares
148+
the result of the adapter with the result of plain `min_max`.
149+
131150
## Coordination with open work
132151

133152
Three PRs added or add mechanisms for extension semantics:
@@ -137,12 +156,13 @@ Three PRs added or add mechanisms for extension semantics:
137156
- [#10566](https://github.com/vortex-data/vortex/pull/10566) added `ExtensionStoragePushdown`, an
138157
aggregate adapter with a composite ID, and
139158
[#10616](https://github.com/vortex-data/vortex/pull/10616) registered it for the temporal types
140-
and `Uuid`. Both merged on 2026-10-20. They are a temporary bridge until phase 2 (X10).
159+
and `Uuid`. Both merged on 2026-10-20. The adapter stays, and U9 fixes its results. In phase 2, a
160+
declaration on the type registers it (X10).
141161
- [#10408](https://github.com/vortex-data/vortex/pull/10408) adds `between` pushdown for
142162
`AnyTemporal`. It is an open draft.
143163

144-
Each one is a reasonable local fix, but each one fixes a different mechanism in place. Phase 2
145-
moves them onto the opt-in mechanism of part 3.
164+
Phase 2 moves the `between` pushdown onto the opt-in mechanism of part 3, and it replaces the
165+
hand-written registrations of the adapter with declarations on the types.
146166

147167
The new extension types `Duration` ([#10467](https://github.com/vortex-data/vortex/pull/10467))
148168
and `Interval` ([#10468](https://github.com/vortex-data/vortex/pull/10468)) are open. Both edit

0 commit comments

Comments
 (0)