You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9ee5b27
Browse filesBrowse the repository at this point in the historyBrowse files
Record the storage pushdown and derive decisions in the type research
X10 records that ExtensionStoragePushdown (#10566, #10616) is a
temporary bridge that phase 2 replaces, and part 3 explains how it
differs from the design. X11 records that derives are a goal: authors
opt in to common behavior with little code, and the Derives sketch is
not a decision. XQ7 asks which mechanism gives that opt-in.
The extension plan now adds ExtensionArray::new_unchecked in phase 1,
because the rules of phases 2 and 3 build their results with it. Phase
4 splits into the lift rule (4a), which waits for nothing and is the
extension case of #10748, and statistics with DateTimeParts (4b).
Phase 2 starts with a design pass for XQ2, XQ5 and XQ7, and its typing
covers lambda bodies.
The type graph gets the missing N2 to N4 edge, and the reason for N2
to N3 now follows the design notes. The design notes rename their
derive rules to DR1 to DR4 and their nullability steps to NU1 and NU2,
so that they no longer collide with D1 to D4 and N1 and N2.
Signed-off-by: Connor Tsui <[email protected]>
Copy file name to clipboardExpand all lines: research/decimal/03-roadmap.md
+2-1Lines changed: 2 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -87,6 +87,7 @@ flowchart TD
87
87
N21["N21 Duration and Interval"]
88
88
89
89
N2 --> N3
90
+
N2 --> N4
90
91
N1 --> N21
91
92
N4 --> N21
92
93
N28 --> N11
@@ -228,7 +229,7 @@ the bar of no compute regression applies to the final state (decided, D19).
228
229
| --- | --- |
229
230
| 0, independent | Strict enum decoding, the DecimalByteParts nullability fix, N1, the integer `Widen` in memory followed by the `encode_values` PR, and the decimal benchmarks and fixtures |
230
231
| 1, one canonical form | N11 as two changes (D11). The in-memory change holds the width owner, N28 for decimals, DecimalByteParts decoding, the read plugin, the DecimalByteParts scheme on the canonical array and the strict `try_new`, each as its own PR. The wire change holds the `vortex.widen` wire format, the preview edition with DecimalByteParts lower parts, and the decimal `Narrow` scheme |
231
-
| 2, foundations | N7 fixed-width kernels. N8 scaled-integer kernels, used first by `Timestamp` unit casts. N4 cast rules, then N21 `Duration`and `Interval`. N2 and then N3, with the decimal rules as acceptance tests. N23 engine plugins |
232
+
| 2, foundations | N7 fixed-width kernels. N8 scaled-integer kernels, used first by `Timestamp` unit casts. N2, then N3 and N4, with the decimal rules as acceptance tests. N21 `Duration` and `Interval` after N1 and N4. N23 engine plugins |
232
233
| 3, new types | N9 wide integers, user-visible only behind an edition (recommendation DQ5). N10 edition gates and the protobuf `oneof`. N17 FixedSizeBinary, after DQ7. N5 array validation |
233
234
| 4, decimal kernels | N12 for precision 19 to 76. N13 rules over the view, which delete most decimal kernels. N14, which is optional. N18 and N27 |
234
235
| 5, optional | N16, then N15 if the built-in type and an extension type become identical. N19 and N20 after N22 |
Copy file name to clipboardExpand all lines: research/decimal/agents/report.md
+12-7Lines changed: 12 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1378,7 +1378,7 @@ needs an **optional** node.
1378
1378
| N1 | Extension opacity fixes | impl, urgent | The urgent changes U1 to U8 of the extension notes. They cover storage casts, lone-side storage compares, errors from reads, nullability, temporal validation, storage checks, zero values and debug output |
1379
1379
| N2 | Rewrite rules that the type owns | research, then impl | X4: a type implements its behavior as rewrite rules on arrays, registered with the type. There is no `ext_wrap`. X5 and X6 add the placement rule and the removal of nested extension layers. Recommendation (XQ2): the rules live on `ExtVTable`, written over `ReduceNode`|
1380
1380
| N3 | Derives for one dtype | impl | X2: `eq` and `ord` are properties of one dtype and never apply across dtypes. Operations across dtypes need a rule of the type (N2) or a cast by the engine (X3) |
1381
-
| N4 | Cast rules (#8570) | impl | C1: a cast exists only if a rule for the pair exists. Extension types get `cast_to` and `cast_from`hooks. CQ1, the default for inexact values, is open |
1381
+
| N4 | Cast rules (#8570) | impl | C1: a cast exists only if a rule for the pair exists. Extension types own cast rules (X4), in place of the `cast_to` and `cast_from`closures of #8570. CQ1, the default for inexact values, is open |
1382
1382
| N5 | Value checks in `try_new`| impl | X7: a checked `try_new` and a `new_unchecked` for kernels. `ExtensionArray::try_new` checks every non-null value. Checks: UTF-8 and FixedSizeBinary length. The decimal precision check moves into N11 (D14) |
1383
1383
| N6 | Nullability restructure | research, optional | Remove nullability duplication between wrapper and storage dtypes |
1384
1384
| N7 | Fixed-width structural layer | impl | Extend `FixedWidthArray` (PR #8864) to run-end, sparse, dict, constant, and builders, keyed by record width |
@@ -1450,6 +1450,7 @@ flowchart TD
1450
1450
N1 --> N21
1451
1451
N1 -.-> N18
1452
1452
N2 --> N3
1453
+
N2 --> N4
1453
1454
N2 -.-> N13
1454
1455
N3 --> N15
1455
1456
N3 -.-> N21
@@ -1502,8 +1503,9 @@ soft: the target works without the source, but it is cheaper, safer, or faster w
1502
1503
3. N1 to N18 (soft): after N1, a UUID column no longer compares with a storage literal. The
1503
1504
engine literal paths then change once, before the storage changes. PR #9886 shows the
1504
1505
current workaround.
1505
-
4. N2 to N3: a derive is one kind of rule that the type owns. It runs the storage operation for
1506
-
operands of one dtype.
1506
+
4. N2 to N3: the opt-in of N3 needs the typing check, which lands with N2 in phase 2 of the
1507
+
extension notes. The opt-in itself need not be a rule that the type owns: the design notes
1508
+
sketch derives as rules of the core functions. The mechanism is open (X11, XQ7).
1507
1509
5. N2 to N13 (soft): the decimal rules can reuse the rule mechanism of extension types. With
1508
1510
recommendation XQ2, rules over `ReduceNode` also rewrite bound expressions for pruning. The
1509
1511
built-in type can own its rules without N2.
@@ -1565,6 +1567,8 @@ soft: the target works without the source, but it is cheaper, safer, or faster w
1565
1567
35. N6 to N15 (soft): a wrapper repeats its nullability in the storage dtype, so every
36. N2 to N4: cast rules are rules that the type owns (X4, C1), so they need the rule mechanism.
1571
+
Phase 3 of the extension notes lands after phase 2 for the same reason.
1568
1572
1569
1573
### 7.4 Critical path and parallel work
1570
1574
@@ -1577,7 +1581,7 @@ soft: the target works without the source, but it is cheaper, safer, or faster w
1577
1581
-**Critical path for delegated decimal kernels (future work, D20):** N9, N12, N13. N8 must land
1578
1582
before N13. N7 is soft, but it shrinks N9, so do it first. N12 for precision 1 to 18 can land at
1579
1583
any point.
1580
-
-**The optional move (N15):** after N13, with N1, N2 then N3, N4, and N16 joining at N15.
1584
+
-**The optional move (N15):** after N13, with N1, N2 then N3 and N4, and N16 joining at N15.
1581
1585
-**Parallel tracks:**
1582
1586
- Extension correctness: N1, N2, N3, N4, N5.
1583
1587
- Decimal: N28, N11, N12, N7, N8, N9, N13, N14.
@@ -1590,8 +1594,8 @@ soft: the target works without the source, but it is cheaper, safer, or faster w
1590
1594
### 7.5 Mapping to the extension-type research
1591
1595
1592
1596
The extension-type notes in `../../extension-types/` use their own ids for urgent changes (U1 to
1593
-
U8) and phases (0 to 6). The decisions file adds ids X1 to X9 and XQ1 to XQ6. This table maps them
1594
-
onto the graph.
1597
+
U8) and phases (0 to 6, with phase 4 split into 4a and 4b). The decisions file adds ids X1 to X11
1598
+
and XQ1 to XQ7. This table maps them onto the graph.
1595
1599
1596
1600
| Extension research item | Node here |
1597
1601
| --- | --- |
@@ -1600,7 +1604,8 @@ onto the graph.
1600
1604
| Phase 1, contract and conformance harness | Not a separate node. Its decimal-like reference type over `i64` is an early acceptance test for N3 and N13 |
1601
1605
| Phase 2, typing, derives and rewrite rules (X2, X4) | N2 and N3. Part 5 of the extension notes lands them after phases 0 and 1, which the graph does not show |
1602
1606
| Phase 3, cast rules that the types own (C1) | N4 |
1603
-
| Phase 4, the lift rule for structural arrays, and storage statistics (X5, XQ1) | The placement rule of section 3.3. DateTimeParts below the structural encodings is the temporal twin of DBP |
1607
+
| Phase 4a, the lift rule for structural arrays (X5, XQ1, #10748) | The placement rule of section 3.3, shared with `Widen` (D25) |
1608
+
| Phase 4b, storage statistics and `DateTimeParts` under the wrapper | DateTimeParts below the structural encodings is the temporal twin of DBP |
1604
1609
| Phase 5, value checks in `ExtensionArray::try_new`, and nullability (X7) | N5 and N6 |
Copy file name to clipboardExpand all lines: research/decisions.md
+9-3Lines changed: 9 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,8 +6,9 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors
6
6
# Decisions
7
7
8
8
This file records the decisions that the maintainer made while reviewing these notes on
9
-
2026-10-19, 2026-10-22, 2026-10-23 and 2026-10-26. It also lists the questions that stay open. The
10
-
other documents follow this file. If a document disagrees with it, this file wins.
9
+
2026-10-19, 2026-10-22, 2026-10-23, 2026-10-26, 2026-10-28 and 2026-10-29. It also lists the
10
+
questions that stay open. The other documents follow this file. If a document disagrees with it,
11
+
this file wins.
11
12
12
13
"Settled" marks a choice of the maintainer. "Recommendation" marks a proposal of these notes that
13
14
the maintainer has not confirmed yet.
@@ -27,6 +28,10 @@ the maintainer has not confirmed yet.
27
28
| X7 |`ExtensionArray::try_new` checks every non-null value. A new `ExtensionArray::new_unchecked` skips the check for kernels that keep legal values | Today arrays are never checked (EXT-V1). This is a bug |
28
29
| X8 | API changes are acceptable where correctness needs them, for example fallible constructors and private enum variants | Correctness and clear abstractions matter more than the current API |
29
30
| X9 | Whether a validity query raises the errors of values that it skips is out of scope here | Issue [#10313](https://github.com/vortex-data/vortex/issues/10313) owns this question, and it is not specific to extension types |
31
+
| X10 |`ExtensionStoragePushdown` ([#10566](https://github.com/vortex-data/vortex/pull/10566)) and `register_storage_ordered` ([#10616](https://github.com/vortex-data/vortex/pull/10616)) are a temporary bridge. They give `is_sorted` and `min_max` the storage order of `Date`, `Time`, `Timestamp` and `Uuid` until the declarations and rules of phase 2 exist, and phase 2 replaces them | The pushdown was needed before the design existed. It differs from the design in where the opt-in lives, in the dtype of its results, and in who can register it ([part 3](extension-types/03-design.md#a-temporary-opt-in-for-aggregates)) |
32
+
| X11 | An extension type author can opt in to common behavior, such as equality and order, with little code. This is the goal of derives. The `Derives` flags and the derive rules in the design notes are one sketch of a mechanism, not a decision | The mechanism is not designed yet. X2 still holds: an opt-in covers values of one dtype |
33
+
34
+
X1 to X9 date from 2026-10-19, and X10 and X11 from 2026-10-29.
30
35
31
36
### Open
32
37
@@ -37,7 +42,8 @@ the maintainer has not confirmed yet.
37
42
| XQ3 | Do file reads check extension values, or trust the file? | Check only what memory safety needs, trust semantic constraints, and never panic on an illegal value. Offer a strict read mode that checks values |
38
43
| XQ4 | Which values can sit in null slots? | Any value. Accessors that take a child out of its parent mask it with the validity of the parent. This also allows a storage encoding to store nulls as a reserved value |
39
44
| XQ5 | How does an aggregate declare its result? | As a value of its input, as a bound on its input, or as neither. Only a value gets wrapped again |
40
-
| XQ6 | Does `Uuid` order by its bytes? | No recommendation yet |
45
+
| XQ6 | Does `Uuid` order by its bytes? | No recommendation yet. `register_storage_ordered(Uuid)` (#10616) already orders it by bytes in `is_sorted` and `min_max`, as a temporary bridge (X10) |
46
+
| XQ7 | Which mechanism gives authors the opt-in of X11? | Start from the sketch in the design notes: flags on the dtype, which the typing check and the rules of the core functions read. Decide it together with XQ2, because both questions decide where the behavior of a type is declared |
aggregate `A` on the storage of extension type `E`, and `register_storage_ordered` registers it for
123
+
`is_sorted` and `min_max`. The default session registers it for `Date`, `Time` and `Timestamp`, and
124
+
`vortex_uuid::initialize` registers it for `Uuid`. It is a temporary bridge until phase 2 (X10),
125
+
and it differs from this design in three ways:
126
+
127
+
- The opt-in lives in the aggregate session, under a separate aggregate ID such as
128
+
`vortex.min_max[vortex.timestamp]`. Plain `min_max` and `is_sorted` do not consult it, so plain
129
+
`min_max` still uses storage order for every extension type, and plain `is_sorted` still rejects
130
+
every extension type. A consumer gets the opt-in only if it asks for the composite ID, and today
131
+
only tests do. In this design, the type declares its behavior once, and every consumer reads
132
+
that declaration through the typing check and the rules.
133
+
- Its results are storage values. The min of a `Timestamp(ms)` column is a raw `i64`, without its
134
+
time unit or time zone. In this design, a result that is a value of the input gets the extension
135
+
dtype back (XQ5). Storage results are correct only for statistics, which store storage values
136
+
(section "Statistics").
137
+
- Any crate can register any pair of an aggregate and a type, and nothing checks the pair. The docs
138
+
of the pushdown say that an invalid registration "can silently return incorrect results". In this
139
+
design, only the type declares what it inherits.
140
+
141
+
Phase 2 replaces each registration with a declaration on the type, and the composite IDs then go
142
+
away.
143
+
115
144
### Operations across dtypes
116
145
117
146
Two dtypes with different metadata are different dtypes, for example `Timestamp(ms)` and
@@ -189,6 +218,12 @@ function. That method moves the wrapper, pushes `ext_storage` down, and decodes
189
218
encoding that holds extension values. `RunEnd` and `Sparse`, which fail on extension values today,
190
219
get support from the same method.
191
220
221
+
The lift rule is the extension case of issue
222
+
[#10748](https://github.com/vortex-data/vortex/issues/10748), which keeps generic structural
223
+
encodings above every other encoding. `Widen` needs the same rule (D25), and WD5 plans it for
224
+
`Widen` only. If the method above serves every wrapping encoding, `Widen`, `Extension` and later
225
+
wrappers share one mechanism, and no wrapper needs its own copy of the seven rules.
226
+
192
227
Type-specific encodings are a separate question. `DateTimeParts` moves under the wrapper as an
193
228
`i64` encoding, so its compare kernel works on storage. The compressor chooses it from the
0 commit comments