Skip to content

Commit 9ee5b27

Browse files
committed
Record the storage pushdown and derive decisions in the type research
X10 records that ExtensionStoragePushdown (#10566, #10616) is a temporary bridge that phase 2 replaces, and part 3 explains how it differs from the design. X11 records that derives are a goal: authors opt in to common behavior with little code, and the Derives sketch is not a decision. XQ7 asks which mechanism gives that opt-in. The extension plan now adds ExtensionArray::new_unchecked in phase 1, because the rules of phases 2 and 3 build their results with it. Phase 4 splits into the lift rule (4a), which waits for nothing and is the extension case of #10748, and statistics with DateTimeParts (4b). Phase 2 starts with a design pass for XQ2, XQ5 and XQ7, and its typing covers lambda bodies. The type graph gets the missing N2 to N4 edge, and the reason for N2 to N3 now follows the design notes. The design notes rename their derive rules to DR1 to DR4 and their nullability steps to NU1 and NU2, so that they no longer collide with D1 to D4 and N1 and N2. Signed-off-by: Connor Tsui <[email protected]>
1 parent 8896923 commit 9ee5b27

6 files changed

Lines changed: 142 additions & 53 deletions

File tree

‎research/decimal/03-roadmap.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@ flowchart TD
8787
N21["N21 Duration and Interval"]
8888
8989
N2 --> N3
90+
N2 --> N4
9091
N1 --> N21
9192
N4 --> N21
9293
N28 --> N11
@@ -228,7 +229,7 @@ the bar of no compute regression applies to the final state (decided, D19).
228229
| --- | --- |
229230
| 0, independent | Strict enum decoding, the DecimalByteParts nullability fix, N1, the integer `Widen` in memory followed by the `encode_values` PR, and the decimal benchmarks and fixtures |
230231
| 1, one canonical form | N11 as two changes (D11). The in-memory change holds the width owner, N28 for decimals, DecimalByteParts decoding, the read plugin, the DecimalByteParts scheme on the canonical array and the strict `try_new`, each as its own PR. The wire change holds the `vortex.widen` wire format, the preview edition with DecimalByteParts lower parts, and the decimal `Narrow` scheme |
231-
| 2, foundations | N7 fixed-width kernels. N8 scaled-integer kernels, used first by `Timestamp` unit casts. N4 cast rules, then N21 `Duration` and `Interval`. N2 and then N3, with the decimal rules as acceptance tests. N23 engine plugins |
232+
| 2, foundations | N7 fixed-width kernels. N8 scaled-integer kernels, used first by `Timestamp` unit casts. N2, then N3 and N4, with the decimal rules as acceptance tests. N21 `Duration` and `Interval` after N1 and N4. N23 engine plugins |
232233
| 3, new types | N9 wide integers, user-visible only behind an edition (recommendation DQ5). N10 edition gates and the protobuf `oneof`. N17 FixedSizeBinary, after DQ7. N5 array validation |
233234
| 4, decimal kernels | N12 for precision 19 to 76. N13 rules over the view, which delete most decimal kernels. N14, which is optional. N18 and N27 |
234235
| 5, optional | N16, then N15 if the built-in type and an extension type become identical. N19 and N20 after N22 |

‎research/decimal/agents/report.md‎

Lines changed: 12 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1378,7 +1378,7 @@ needs an **optional** node.
13781378
| N1 | Extension opacity fixes | impl, urgent | The urgent changes U1 to U8 of the extension notes. They cover storage casts, lone-side storage compares, errors from reads, nullability, temporal validation, storage checks, zero values and debug output |
13791379
| N2 | Rewrite rules that the type owns | research, then impl | X4: a type implements its behavior as rewrite rules on arrays, registered with the type. There is no `ext_wrap`. X5 and X6 add the placement rule and the removal of nested extension layers. Recommendation (XQ2): the rules live on `ExtVTable`, written over `ReduceNode` |
13801380
| N3 | Derives for one dtype | impl | X2: `eq` and `ord` are properties of one dtype and never apply across dtypes. Operations across dtypes need a rule of the type (N2) or a cast by the engine (X3) |
1381-
| N4 | Cast rules (#8570) | impl | C1: a cast exists only if a rule for the pair exists. Extension types get `cast_to` and `cast_from` hooks. CQ1, the default for inexact values, is open |
1381+
| N4 | Cast rules (#8570) | impl | C1: a cast exists only if a rule for the pair exists. Extension types own cast rules (X4), in place of the `cast_to` and `cast_from` closures of #8570. CQ1, the default for inexact values, is open |
13821382
| N5 | Value checks in `try_new` | impl | X7: a checked `try_new` and a `new_unchecked` for kernels. `ExtensionArray::try_new` checks every non-null value. Checks: UTF-8 and FixedSizeBinary length. The decimal precision check moves into N11 (D14) |
13831383
| N6 | Nullability restructure | research, optional | Remove nullability duplication between wrapper and storage dtypes |
13841384
| N7 | Fixed-width structural layer | impl | Extend `FixedWidthArray` (PR #8864) to run-end, sparse, dict, constant, and builders, keyed by record width |
@@ -1450,6 +1450,7 @@ flowchart TD
14501450
N1 --> N21
14511451
N1 -.-> N18
14521452
N2 --> N3
1453+
N2 --> N4
14531454
N2 -.-> N13
14541455
N3 --> N15
14551456
N3 -.-> N21
@@ -1502,8 +1503,9 @@ soft: the target works without the source, but it is cheaper, safer, or faster w
15021503
3. N1 to N18 (soft): after N1, a UUID column no longer compares with a storage literal. The
15031504
engine literal paths then change once, before the storage changes. PR #9886 shows the
15041505
current workaround.
1505-
4. N2 to N3: a derive is one kind of rule that the type owns. It runs the storage operation for
1506-
operands of one dtype.
1506+
4. N2 to N3: the opt-in of N3 needs the typing check, which lands with N2 in phase 2 of the
1507+
extension notes. The opt-in itself need not be a rule that the type owns: the design notes
1508+
sketch derives as rules of the core functions. The mechanism is open (X11, XQ7).
15071509
5. N2 to N13 (soft): the decimal rules can reuse the rule mechanism of extension types. With
15081510
recommendation XQ2, rules over `ReduceNode` also rewrite bound expressions for pruning. The
15091511
built-in type can own its rules without N2.
@@ -1565,6 +1567,8 @@ soft: the target works without the source, but it is cheaper, safer, or faster w
15651567
35. N6 to N15 (soft): a wrapper repeats its nullability in the storage dtype, so every
15661568
nullability change rebuilds the extension dtype
15671569
(`vortex-array/src/dtype/extension/typed.rs:91-99`).
1570+
36. N2 to N4: cast rules are rules that the type owns (X4, C1), so they need the rule mechanism.
1571+
Phase 3 of the extension notes lands after phase 2 for the same reason.
15681572

15691573
### 7.4 Critical path and parallel work
15701574

@@ -1577,7 +1581,7 @@ soft: the target works without the source, but it is cheaper, safer, or faster w
15771581
- **Critical path for delegated decimal kernels (future work, D20):** N9, N12, N13. N8 must land
15781582
before N13. N7 is soft, but it shrinks N9, so do it first. N12 for precision 1 to 18 can land at
15791583
any point.
1580-
- **The optional move (N15):** after N13, with N1, N2 then N3, N4, and N16 joining at N15.
1584+
- **The optional move (N15):** after N13, with N1, N2 then N3 and N4, and N16 joining at N15.
15811585
- **Parallel tracks:**
15821586
- Extension correctness: N1, N2, N3, N4, N5.
15831587
- Decimal: N28, N11, N12, N7, N8, N9, N13, N14.
@@ -1590,8 +1594,8 @@ soft: the target works without the source, but it is cheaper, safer, or faster w
15901594
### 7.5 Mapping to the extension-type research
15911595

15921596
The extension-type notes in `../../extension-types/` use their own ids for urgent changes (U1 to
1593-
U8) and phases (0 to 6). The decisions file adds ids X1 to X9 and XQ1 to XQ6. This table maps them
1594-
onto the graph.
1597+
U8) and phases (0 to 6, with phase 4 split into 4a and 4b). The decisions file adds ids X1 to X11
1598+
and XQ1 to XQ7. This table maps them onto the graph.
15951599

15961600
| Extension research item | Node here |
15971601
| --- | --- |
@@ -1600,7 +1604,8 @@ onto the graph.
16001604
| Phase 1, contract and conformance harness | Not a separate node. Its decimal-like reference type over `i64` is an early acceptance test for N3 and N13 |
16011605
| Phase 2, typing, derives and rewrite rules (X2, X4) | N2 and N3. Part 5 of the extension notes lands them after phases 0 and 1, which the graph does not show |
16021606
| Phase 3, cast rules that the types own (C1) | N4 |
1603-
| Phase 4, the lift rule for structural arrays, and storage statistics (X5, XQ1) | The placement rule of section 3.3. DateTimeParts below the structural encodings is the temporal twin of DBP |
1607+
| Phase 4a, the lift rule for structural arrays (X5, XQ1, #10748) | The placement rule of section 3.3, shared with `Widen` (D25) |
1608+
| Phase 4b, storage statistics and `DateTimeParts` under the wrapper | DateTimeParts below the structural encodings is the temporal twin of DBP |
16041609
| Phase 5, value checks in `ExtensionArray::try_new`, and nullability (X7) | N5 and N6 |
16051610
| Phase 6, authoring and boundaries | N23 |
16061611

‎research/decisions.md‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,9 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors
66
# Decisions
77

88
This file records the decisions that the maintainer made while reviewing these notes on
9-
2026-10-19, 2026-10-22, 2026-10-23 and 2026-10-26. It also lists the questions that stay open. The
10-
other documents follow this file. If a document disagrees with it, this file wins.
9+
2026-10-19, 2026-10-22, 2026-10-23, 2026-10-26, 2026-10-28 and 2026-10-29. It also lists the
10+
questions that stay open. The other documents follow this file. If a document disagrees with it,
11+
this file wins.
1112

1213
"Settled" marks a choice of the maintainer. "Recommendation" marks a proposal of these notes that
1314
the maintainer has not confirmed yet.
@@ -27,6 +28,10 @@ the maintainer has not confirmed yet.
2728
| X7 | `ExtensionArray::try_new` checks every non-null value. A new `ExtensionArray::new_unchecked` skips the check for kernels that keep legal values | Today arrays are never checked (EXT-V1). This is a bug |
2829
| X8 | API changes are acceptable where correctness needs them, for example fallible constructors and private enum variants | Correctness and clear abstractions matter more than the current API |
2930
| X9 | Whether a validity query raises the errors of values that it skips is out of scope here | Issue [#10313](https://github.com/vortex-data/vortex/issues/10313) owns this question, and it is not specific to extension types |
31+
| X10 | `ExtensionStoragePushdown` ([#10566](https://github.com/vortex-data/vortex/pull/10566)) and `register_storage_ordered` ([#10616](https://github.com/vortex-data/vortex/pull/10616)) are a temporary bridge. They give `is_sorted` and `min_max` the storage order of `Date`, `Time`, `Timestamp` and `Uuid` until the declarations and rules of phase 2 exist, and phase 2 replaces them | The pushdown was needed before the design existed. It differs from the design in where the opt-in lives, in the dtype of its results, and in who can register it ([part 3](extension-types/03-design.md#a-temporary-opt-in-for-aggregates)) |
32+
| X11 | An extension type author can opt in to common behavior, such as equality and order, with little code. This is the goal of derives. The `Derives` flags and the derive rules in the design notes are one sketch of a mechanism, not a decision | The mechanism is not designed yet. X2 still holds: an opt-in covers values of one dtype |
33+
34+
X1 to X9 date from 2026-10-19, and X10 and X11 from 2026-10-29.
3035

3136
### Open
3237

@@ -37,7 +42,8 @@ the maintainer has not confirmed yet.
3742
| XQ3 | Do file reads check extension values, or trust the file? | Check only what memory safety needs, trust semantic constraints, and never panic on an illegal value. Offer a strict read mode that checks values |
3843
| XQ4 | Which values can sit in null slots? | Any value. Accessors that take a child out of its parent mask it with the validity of the parent. This also allows a storage encoding to store nulls as a reserved value |
3944
| XQ5 | How does an aggregate declare its result? | As a value of its input, as a bound on its input, or as neither. Only a value gets wrapped again |
40-
| XQ6 | Does `Uuid` order by its bytes? | No recommendation yet |
45+
| XQ6 | Does `Uuid` order by its bytes? | No recommendation yet. `register_storage_ordered(Uuid)` (#10616) already orders it by bytes in `is_sorted` and `min_max`, as a temporary bridge (X10) |
46+
| XQ7 | Which mechanism gives authors the opt-in of X11? | Start from the sketch in the design notes: flags on the dtype, which the typing check and the rules of the core functions read. Decide it together with XQ2, because both questions decide where the behavior of a type is declared |
4147

4248
## Decimal and new physical types
4349

‎research/extension-types/03-design.md‎

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,9 @@ compression keep working, because the rewritten operations are ordinary storage
7878
### Derives for one dtype
7979

8080
A derive says that a semantic operation on values of one extension dtype is the same operation on
81-
their storage values. It is a property of one dtype (X2):
81+
their storage values. It is a property of one dtype (X2). The goal is that an author can opt in to
82+
common behavior, such as equality and order, with little code. The mechanism in this section is a
83+
sketch of that goal, not a decision (X11, XQ7):
8284

8385
```rust
8486
/// Storage behavior that values of `dtype` inherit, like `#[derive]` on a newtype.
@@ -112,6 +114,33 @@ So `ts < 5i64` stays rejected, because only one side is an extension.
112114
| `Json`, WKB, native geometry | None |
113115
| Unknown type | None |
114116

117+
### A temporary opt-in for aggregates
118+
119+
`develop` already has a narrower opt-in for aggregates, from
120+
[#10566](https://github.com/vortex-data/vortex/pull/10566) and
121+
[#10616](https://github.com/vortex-data/vortex/pull/10616). `ExtensionStoragePushdown<A, E>` runs
122+
aggregate `A` on the storage of extension type `E`, and `register_storage_ordered` registers it for
123+
`is_sorted` and `min_max`. The default session registers it for `Date`, `Time` and `Timestamp`, and
124+
`vortex_uuid::initialize` registers it for `Uuid`. It is a temporary bridge until phase 2 (X10),
125+
and it differs from this design in three ways:
126+
127+
- The opt-in lives in the aggregate session, under a separate aggregate ID such as
128+
`vortex.min_max[vortex.timestamp]`. Plain `min_max` and `is_sorted` do not consult it, so plain
129+
`min_max` still uses storage order for every extension type, and plain `is_sorted` still rejects
130+
every extension type. A consumer gets the opt-in only if it asks for the composite ID, and today
131+
only tests do. In this design, the type declares its behavior once, and every consumer reads
132+
that declaration through the typing check and the rules.
133+
- Its results are storage values. The min of a `Timestamp(ms)` column is a raw `i64`, without its
134+
time unit or time zone. In this design, a result that is a value of the input gets the extension
135+
dtype back (XQ5). Storage results are correct only for statistics, which store storage values
136+
(section "Statistics").
137+
- Any crate can register any pair of an aggregate and a type, and nothing checks the pair. The docs
138+
of the pushdown say that an invalid registration "can silently return incorrect results". In this
139+
design, only the type declares what it inherits.
140+
141+
Phase 2 replaces each registration with a declaration on the type, and the composite IDs then go
142+
away.
143+
115144
### Operations across dtypes
116145

117146
Two dtypes with different metadata are different dtypes, for example `Timestamp(ms)` and
@@ -189,6 +218,12 @@ function. That method moves the wrapper, pushes `ext_storage` down, and decodes
189218
encoding that holds extension values. `RunEnd` and `Sparse`, which fail on extension values today,
190219
get support from the same method.
191220

221+
The lift rule is the extension case of issue
222+
[#10748](https://github.com/vortex-data/vortex/issues/10748), which keeps generic structural
223+
encodings above every other encoding. `Widen` needs the same rule (D25), and WD5 plans it for
224+
`Widen` only. If the method above serves every wrapping encoding, `Widen`, `Extension` and later
225+
wrappers share one mechanism, and no wrapper needs its own copy of the seven rules.
226+
192227
Type-specific encodings are a separate question. `DateTimeParts` moves under the wrapper as an
193228
`i64` encoding, so its compare kernel works on storage. The compressor chooses it from the
194229
extension metadata.

0 commit comments

Comments
 (0)