Skip to content

Commit 38b06df

Browse files
committed
Record that U9 is dropped in favor of SP4
The maintainer closed #11188 on 2026-11-04, because naming the value fields by index is a half-measure and SP4 removes the adapter anyway. Issue #11243 tracks the bug until SP4. X12, part 4, the plan, and the findings now say so, and part 3 asks whether SP1 needs a positional `Fields` mapping at all. Signed-off-by: Connor Tsui <[email protected]>
1 parent 463c40d commit 38b06df

7 files changed

Lines changed: 46 additions & 29 deletions

File tree

‎research/decisions.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ on `ct/decimal-research`.
3434
| X9 | Whether a validity query raises the errors of values that it skips is out of scope here | Issue [#10313](https://github.com/vortex-data/vortex/issues/10313) owns this question, and it is not specific to extension types |
3535
| X10 | `ExtensionStoragePushdown` ([#10566](https://github.com/vortex-data/vortex/pull/10566)) computes an aggregate on storage correctly, but it does not stay a separate aggregate. Its body becomes the default storage path of aggregate binding (X21). Its composite IDs and `register_storage_ordered` ([#10616](https://github.com/vortex-data/vortex/pull/10616)) go away | Consumers such as zone maps and engine pushdown ask for the plain IDs, so an opt-in that needs a composite ID reaches only tests. The registrations are written by hand, and nothing checks them. [Part 3](extension-types/03-design.md#aggregates) explains the change |
3636
| X11 | An extension type author can opt in to common behavior, such as equality and order, with little code. Claims (X13) are the mechanism. They replace the `Derives` flags sketch | One line in the vtable of the type is enough. X2 still holds: a claim covers values of one dtype |
37-
| X12 | Each result of `ExtensionStoragePushdown` that is a value of the input gets the extension dtype back inside the adapter, before the adapter returns it. Today the adapter returns raw storage values | A storage result loses the metadata of the type, for example the time unit and time zone of a timestamp. Nothing wraps the result later, and plain `min_max` already wraps its results (EXT-A3) |
37+
| X12 | Each result of `ExtensionStoragePushdown` that is a value of the input gets the extension dtype back inside the adapter, before the adapter returns it. Today the adapter returns raw storage values. Superseded on 2026-11-04: the maintainer dropped the fix in the adapter (U9, #11188), because naming the value fields by index is a half-measure. SP4 gives the results their extension dtype when it moves the adapter into aggregate binding (X21), and [#11243](https://github.com/vortex-data/vortex/issues/11243) tracks the bug until then | A storage result loses the metadata of the type, for example the time unit and time zone of a timestamp. Nothing wraps the result later, and plain `min_max` already wraps its results (EXT-A3) |
3838
| X13 | A claim is a promise that an extension type makes about its storage: one kind of operation gives the same answer on the values and on their storage values, in both directions. `Eq` means that two values are equal exactly when their storage values are equal. `Ord` means that the value order is the storage order, and it implies `Eq`. An extension type is opaque unless it makes a claim. Each type lists its claims by hand in `ExtVTable::claims`, and the default is no claims. Claims are never derived from which `=` or `<` kernels exist. Built-in dtypes hold claims by a fixed table: `Null`, `Bool`, `Primitive`, `Decimal`, `Utf8` and `Binary` hold both. `Struct`, `List`, `FixedSizeList` and `Map` hold a claim when their children hold it. `Union` and `Variant` hold neither. A type stored as floats inherits the float semantics of Vortex: a total order and bitwise equality | Kernels are registered per encoding, not per dtype, so a missing kernel would silently remove a claim. A type with its own `=` rule still cannot claim `Eq`, because `group by` and hashing run on storage. The built-in table follows what compare supports today (`vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:206-219` at `c56ee8803b`) |
3939
| X14 | The claims are `#[non_exhaustive] pub enum Claim { Eq, Ord }`, with `Claim::implies` and `DType::holds(Claim)`. A claim defined outside Vortex needs a struct with statics later. That switch is a mechanical change of interfaces, and the logic stays the same | Start simple. Only a third-party claim needs the open form, and none exists yet |
4040
| X15 | `ExtDType::try_new` checks that the declared storage dtype holds every claim of the type (X27). The claims must not depend on the top-level nullability (X25), so the check runs once. Typing then checks only the top-level dtype of an argument with `DType::holds` | A claim is valid only if the storage holds it. The check is linear in the size of the dtype tree and stops at nested extension dtypes, which were checked when they were built. So nested extension types need no extra work in typing |

‎research/extension-types/02-current-problems.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -112,7 +112,7 @@ both paths. The cast rows stay until U1.
112112
`between`, but it closed without a merge on 2026-11-03.
113113
- **Storage pushdown results (EXT-A3).** The separate aggregates return storage values. The min of a
114114
`Timestamp(ms)` column is a raw `i64`, while plain `min_max` returns a `Timestamp`. Nothing wraps
115-
the result later (X12, U9).
115+
the result later (X12). Issue #11243 tracks this bug until SP4 fixes it.
116116
- **Zone maps (EXT-A2).** When a statistic rejects an extension dtype, the zone map uses the storage
117117
dtype. JSON and WKB columns store exact, unbounded bounds, so WKB zone maps hold whole geometry
118118
blobs. Writers have stored storage-order min and max for every extension type for at least a

‎research/extension-types/03-design.md‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -231,6 +231,12 @@ and uses `ExtensionArray::new_unchecked`, and debug builds run the value check.
231231
`Rewrap` must guarantee legal results. `Eq` and `Ord` do, because their results are input values. A
232232
future arithmetic claim can overflow, so it needs a checked mapping.
233233

234+
Open question: `Fields` is a positional list next to the dtype of the result, for example the two
235+
`Rewrap`s of `min_max`. U9 (#11188) named such fields by index, and the maintainer rejected it as a
236+
half-measure, because the indices can drift from the real fields. Each aggregate already builds its
237+
dtypes from an element dtype, for example `make_minmax_dtype`. So SP1 can find out whether an
238+
aggregate that builds its dtypes over the extension dtype needs a mapping at all.
239+
234240
### Aggregates
235241

236242
`develop` has a narrower opt-in for aggregates, from
@@ -254,8 +260,9 @@ When `AggregateDTypes::try_new` meets an extension input, it asks the type first
254260
A partial state stays in storage form in memory. It gets the logical form when it leaves the
255261
accumulator: `finalize`, `finalize_scalar` and `to_scalar` wrap, and `partial_from_scalar`
256262
unwraps. So consumers see the same partial dtype as today, and the cost is constant for each
257-
partial. The composite IDs and `register_storage_ordered` go away. U9 first wraps the results of
258-
today's adapter, so its results are right before the move.
263+
partial. The composite IDs and `register_storage_ordered` go away. Until then, the adapter
264+
returns storage values, and
265+
[#11243](https://github.com/vortex-data/vortex/issues/11243) tracks this bug.
259266

260267
### Operations across dtypes
261268

‎research/extension-types/04-urgent-changes.md‎

Lines changed: 18 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,11 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors
66
# 4. Urgent changes
77

88
These changes fix wrong results and panics. U1 to U8 do not depend on the design in part 3, and
9-
each one fits in a small PR with a regression test. U9 does not depend on the design either, and it
10-
can land at any time (X35). Finding IDs refer to [`agents/findings.md`](agents/findings.md). On
11-
2026-10-29, every change was still needed on `develop` at `1d09649bbd`, and the line numbers in this
12-
part are for that commit. U5 merged in #11015 on 2026-10-30. U2, U3, U4, U6, U7 and U8 merged on
13-
2026-11-02. U1 has no PR yet, and U9 is the open draft #11188.
9+
each one fits in a small PR with a regression test. The maintainer dropped U9 on 2026-11-04, and
10+
SP4 fixes its finding instead (see the U9 section). Finding IDs refer to
11+
[`agents/findings.md`](agents/findings.md). On 2026-10-29, every change was still needed on
12+
`develop` at `1d09649bbd`, and the line numbers in this part are for that commit. U5 merged in
13+
#11015 on 2026-10-30. U2, U3, U4, U6, U7 and U8 merged on 2026-11-02. U1 has no PR yet.
1414

1515
| ID | Change | Fixes | Visible effect |
1616
| --- | --- | --- | --- |
@@ -22,7 +22,7 @@ part are for that commit. U5 merged in #11015 on 2026-10-30. U2, U3, U4, U6, U7
2222
| U6 | Check the storage dtype of arrays and scalars | EXT-V2, EXT-V7 | An array or a scalar with the wrong storage is rejected |
2323
| U7 | Stop invalid zero values | EXT-V3 | No invalid scalars |
2424
| U8 | Fix `Debug for ExtDTypeRef` | EXT-R1 | Correct debug output |
25-
| U9 | Wrap the results of the storage pushdown | EXT-A3 | `vortex.min_max[vortex.timestamp]` returns timestamps, not `i64` values |
25+
| ~~U9~~ | ~~Wrap the results of the storage pushdown~~ Dropped, SP4 fixes it (#11243) | EXT-A3 | ~~`vortex.min_max[vortex.timestamp]` returns timestamps, not `i64` values~~ |
2626

2727
## Test requirement
2828

@@ -142,6 +142,13 @@ caller. The storage declaration of phase 1 replaces this interim form (X27).
142142

143143
## U9. Storage pushdown results
144144

145+
The maintainer dropped U9 on 2026-11-04 and closed its PR #11188. The PR named the value fields of
146+
each partial and result by index (`[0, 1]` for `min_max`, `[2, 3]` for `is_sorted`), with a new
147+
trait that also limited what `finalize` can do. The indices can drift from the real fields, and
148+
SP4 removes the adapter anyway, so the fix was a half-measure. SP4 fixes EXT-A3 instead, and issue
149+
[#11243](https://github.com/vortex-data/vortex/issues/11243) tracks the bug until then. The rest
150+
of this section records the original plan.
151+
145152
U9 does not wait for `ResultMapping` (X35). `register_storage_ordered` wraps only `MinMax` and
146153
`IsSorted` (`vortex-array/src/aggregate_fn/session.rs:195-198` at `c7449d567a`), and each of them
147154
maps its result in a fixed way:
@@ -180,7 +187,7 @@ On 2026-11-04, the PRs for part 4 stand as follows:
180187
the other operand, so `Uuid` and `Json` filters still push down. Its tests cover dynamic filter
181188
updates. It also makes the export of a non-temporal extension scalar as a bare DataFusion
182189
`ScalarValue` an error. So item 1 of "Order constraints" in part 5 holds.
183-
- U9 is the open draft #11188.
190+
- U9 closed without a merge as #11188 on 2026-11-04. Issue #11243 tracks EXT-A3 until SP4.
184191
- U1 has no PR yet. The Arrow export that lands before it (AX) is the open draft #11224.
185192

186193
Three more PRs touch this work:
@@ -265,16 +272,17 @@ Three PRs added or proposed mechanisms for extension semantics:
265272
- [#10566](https://github.com/vortex-data/vortex/pull/10566) added `ExtensionStoragePushdown`, an
266273
aggregate adapter with a composite ID, and
267274
[#10616](https://github.com/vortex-data/vortex/pull/10616) registered it for the temporal types
268-
and `Uuid`. Both merged on 2026-10-20. U9 fixes its results. In SP4, its body becomes the
269-
default storage path of aggregate binding, and its composite IDs go away (X10, X21).
275+
and `Uuid`. Both merged on 2026-10-20. In SP4, its body becomes the default storage path of
276+
aggregate binding, its composite IDs go away, and its results get their extension dtype (X10,
277+
X21, #11243).
270278
- [#10408](https://github.com/vortex-data/vortex/pull/10408) proposed a `between` pushdown for
271279
`AnyTemporal`. It closed without a merge on 2026-11-03.
272280

273281
SP4 replaces the hand-written registrations of the adapter with the claims of the types.
274282

275283
[#10769](https://github.com/vortex-data/vortex/pull/10769) (draft) moves the aggregate
276284
implementations into a new `vortex-aggregates` crate, including the tests of the storage pushdown.
277-
U9 changes the same code, so the order of the two needs agreement.
285+
SP4 changes the same code, so the order of the two needs agreement.
278286

279287
The new extension types `Duration` ([#10467](https://github.com/vortex-data/vortex/pull/10467))
280288
and `Interval` ([#10468](https://github.com/vortex-data/vortex/pull/10468)) are open. Both edit

‎research/extension-types/05-implementation.md‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ phases 2 to 4 use hardcoded lists of types (X35).
1818
| --- | --- | --- |
1919
| 0 | Urgent fixes U1 to U8, the path-consistency test helper, and four fixes outside part 4: the `Timestamp` range (#10953), the `Date` display (EXT-V10), the spatial nested nullability (EXT-N2), and imports that fail on nested nulls (X29) | Nothing |
2020
| 1 | The storage declaration: `storage_dtype`, `validate_metadata`, strict reads, the move of storage facts into the metadata, and `ExtensionArray::new_unchecked` | U4 and U5, both merged, and the spatial validator fix SN1 |
21-
| 2 | Declarations with hardcoded opt-ins: `ExtensionUse` with `None`, `Structural` and `Specific`, `extension_use` on both vtables, and the hardcoded lists of X35. Then the declarations of the built-in functions, by family. Nothing reads them yet, so behavior does not change. U9 can land at any time | Nothing |
21+
| 2 | Declarations with hardcoded opt-ins: `ExtensionUse` with `None`, `Structural` and `Specific`, `extension_use` on both vtables, and the hardcoded lists of X35. Then the declarations of the built-in functions, by family. Nothing reads them yet, so behavior does not change. U9 was dropped (#11243) | Nothing |
2222
| 3 | Typing: the check that rejects every call that answers `None` | 0 (U1, U2), 2 |
2323
| 4 | Statistics: readers ignore min and max of types outside the order list, and writers drop the storage fallback | 2 for the readers, 3 for the writers |
2424
| 5 | Type rules and casts: `ExtVTable::rules` over `ReduceNode`, exact unit casts, and the named lossy unit function | 3 |
@@ -30,8 +30,8 @@ Phases 0 and 1 can run in parallel. Phase 1 builds on U4 and U5, which merged on
3030
2026-10-30. Its storage declaration includes the strict reads, so it lands after the spatial
3131
validator fix SN1, which merged on 2026-11-04 (see "Order constraints", items 3 and 6). The Arrow
3232
export with metadata is part of phase 7, but it lands in phase 0, before U1. Phase 2 adds
33-
declarations that nothing reads yet, so it can start at any time. U9 changes only the results of
34-
the composite aggregates, which only tests use. Phase 3 is the critical step: it turns on typing,
33+
declarations that nothing reads yet, so it can start at any time. U9 was dropped, and SP4 fixes the
34+
results of the composite aggregates (#11243). Phase 3 is the critical step: it turns on typing,
3535
and from then on extension types are opaque. Phases 4 and 5 build on it. Phase 6 runs in parallel
3636
with phases 2 to 5.
3737

@@ -54,7 +54,7 @@ dotted line without an arrow links an open PR that changes the same code, so the
5454
coordination. Each node is one PR, and its ID is the ID of that PR in the PR plan of
5555
[#11114](https://github.com/vortex-data/vortex/issues/11114), or of
5656
[#11115](https://github.com/vortex-data/vortex/issues/11115) for the claims (CL) and the storage
57-
paths (SP).
57+
paths (SP). A gray node was dropped.
5858

5959
```mermaid
6060
flowchart TD
@@ -97,7 +97,7 @@ flowchart TD
9797
TY3["TY3 structural functions declare Structural"]
9898
TY4["TY4 list_contains declares Specific"]
9999
TY5["TY5 type-specific functions declare Specific"]
100-
U9["U9 wrap storage pushdown results<br/>#11188"]:::open
100+
U9["U9 dropped, SP4 fixes EXT-A3<br/>#11243"]:::dropped
101101
end
102102
FD --> TY1
103103
FD --> TY2
@@ -195,7 +195,6 @@ flowchart TD
195195
PT["Comparisons proven true from statistics<br/>#10686"]:::open
196196
OS["optimize only with a session<br/>#10793"]:::open
197197
end
198-
AG -.- U9
199198
AG -.- TY2
200199
AG -.- SP4
201200
BB -.- FD
@@ -208,6 +207,7 @@ flowchart TD
208207
209208
classDef merged fill:#d7f0dc,stroke:#2f7a45,color:#10261a
210209
classDef open fill:#fbefc7,stroke:#a87c0f,color:#2b2105
210+
classDef dropped fill:#e6e6e6,stroke:#7a7a7a,color:#333333,stroke-dasharray:4 3
211211
```
212212

213213
## Order constraints
@@ -253,14 +253,14 @@ breaks something that works today. Those parts must land together, or in a fixed
253253
The check rejects every call that answers `None`, so a family without its declaration stops
254254
working. For example, a compare of two `Timestamp` columns would fail when it is built.
255255

256-
TY2, U9 and SP4 also need coordination with
256+
TY2 and SP4 also need coordination with
257257
[#10769](https://github.com/vortex-data/vortex/pull/10769), which moves the aggregate
258258
implementations that they change into `vortex-aggregates`.
259259

260260
## Phase 0: urgent fixes
261261

262262
- Write the path-consistency helper first, because U1 and U2 use it. It merged as #11013.
263-
- Land U1 to U8 as separate PRs. U9 is listed in phase 2, but it can land at any time.
263+
- Land U1 to U8 as separate PRs. U9 was dropped on 2026-11-04 (#11243).
264264
- The maintainer finished [#10953](https://github.com/vortex-data/vortex/pull/10953) with commits
265265
instead of waiting on its author. It makes `Timestamp` accept every `i64` (X31) and fixes the
266266
`DateTimeParts` days wrap (EXT-S3). The maintainer commits of 2026-10-30 merge `develop`, update
@@ -369,8 +369,8 @@ implementations that they change into `vortex-aggregates`.
369369
- The hand-written paths of today stay, and they do the work for the types on the lists. For
370370
example, the extension compare kernel still compares storage. After phase 3, typing rejects every
371371
other call before it reaches them.
372-
- U9 wraps the results of today's `ExtensionStoragePushdown`. Part 4 has the details. It needs no
373-
declaration, so it can land at any time.
372+
- U9 was dropped on 2026-11-04. SP4 gives the results of today's `ExtensionStoragePushdown` their
373+
extension dtype instead, and #11243 tracks the bug until then. Part 4 has the details.
374374
- Coordinate with [#10769](https://github.com/vortex-data/vortex/pull/10769), which moves the
375375
aggregate implementations into a new `vortex-aggregates` crate, and with
376376
[#10837](https://github.com/vortex-data/vortex/pull/10837), which changes
@@ -551,7 +551,7 @@ Every finding of [`agents/findings.md`](agents/findings.md) has a fix in this pl
551551
| EXT-Q2 | TY1 and the typing check of phase 3, with the hardcoded lists, and then SP3 with the claims |
552552
| EXT-A1 | SP4 and SP5 |
553553
| EXT-A2 | Phase 4, and SP6 for the claims |
554-
| EXT-A3 | U9 |
554+
| EXT-A3 | SP4, tracked in #11243 (U9 was dropped) |
555555
| EXT-V1 | U3 for the panics, merged in #11014, and phase 6 for the value check of `try_new` |
556556
| EXT-V2, EXT-V7 | U6, merged in #11014 |
557557
| EXT-V3 | U7, merged in #11014 |

‎research/extension-types/agents/design-notes.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -589,8 +589,9 @@ sees the same partial dtype as today. For example, a zone map of a `Timestamp` c
589589
already wraps its result after each batch today
590590
(`vortex-array/src/aggregate_fn/fns/min_max/extension.rs:28-29` at `1d09649bbd`).
591591

592-
U9 comes first (part 4). It wraps the results of today's adapter with a fixed mapping for `MinMax`
593-
and `IsSorted`, so that the composite aggregates return logical values before SP4 removes them.
592+
U9 was meant to come first (part 4), with a fixed mapping for `MinMax` and `IsSorted` in today's
593+
adapter. The maintainer dropped it on 2026-11-04, so the composite aggregates return storage values
594+
until SP4 removes them (#11243).
594595

595596
## 5. Wrapping storage
596597

‎research/extension-types/agents/findings.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -318,8 +318,9 @@ Terms:
318318
`storage_pushdown.rs`, only the registrations (`vortex-array/src/aggregate_fn/session.rs:134-136`
319319
and `:192-194`, `encodings/uuid/src/lib.rs:46`) and the `Uuid` tests name the adapter. The test
320320
`pushes_down_into_storage` (`storage_pushdown.rs:350`) asserts the storage result.
321-
- **Fix:** U9. The adapter gives each result that is a value of the input the extension dtype back
322-
(X12).
321+
- **Fix:** SP4, tracked in [#11243](https://github.com/vortex-data/vortex/issues/11243). U9
322+
(#11188) gave each result that is a value of the input the extension dtype back inside the
323+
adapter (X12), but the maintainer dropped it on 2026-11-04.
323324

324325
### EXT-V1: Arrays are never value-checked, and reads panic on bad values
325326

0 commit comments

Comments
 (0)