You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 38b06df
Browse filesBrowse the repository at this point in the historyBrowse files
The maintainer closed#11188 on 2026-11-04, because naming the value fields
by index is a half-measure and SP4 removes the adapter anyway. Issue #11243
tracks the bug until SP4. X12, part 4, the plan, and the findings now say
so, and part 3 asks whether SP1 needs a positional `Fields` mapping at all.
Signed-off-by: Connor Tsui <[email protected]>
Copy file name to clipboardExpand all lines: research/decisions.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -34,7 +34,7 @@ on `ct/decimal-research`.
34
34
| X9 | Whether a validity query raises the errors of values that it skips is out of scope here | Issue [#10313](https://github.com/vortex-data/vortex/issues/10313) owns this question, and it is not specific to extension types |
35
35
| X10 |`ExtensionStoragePushdown` ([#10566](https://github.com/vortex-data/vortex/pull/10566)) computes an aggregate on storage correctly, but it does not stay a separate aggregate. Its body becomes the default storage path of aggregate binding (X21). Its composite IDs and `register_storage_ordered` ([#10616](https://github.com/vortex-data/vortex/pull/10616)) go away | Consumers such as zone maps and engine pushdown ask for the plain IDs, so an opt-in that needs a composite ID reaches only tests. The registrations are written by hand, and nothing checks them. [Part 3](extension-types/03-design.md#aggregates) explains the change |
36
36
| X11 | An extension type author can opt in to common behavior, such as equality and order, with little code. Claims (X13) are the mechanism. They replace the `Derives` flags sketch | One line in the vtable of the type is enough. X2 still holds: a claim covers values of one dtype |
37
-
| X12 | Each result of `ExtensionStoragePushdown` that is a value of the input gets the extension dtype back inside the adapter, before the adapter returns it. Today the adapter returns raw storage values | A storage result loses the metadata of the type, for example the time unit and time zone of a timestamp. Nothing wraps the result later, and plain `min_max` already wraps its results (EXT-A3) |
37
+
| X12 | Each result of `ExtensionStoragePushdown` that is a value of the input gets the extension dtype back inside the adapter, before the adapter returns it. Today the adapter returns raw storage values. Superseded on 2026-11-04: the maintainer dropped the fix in the adapter (U9, #11188), because naming the value fields by index is a half-measure. SP4 gives the results their extension dtype when it moves the adapter into aggregate binding (X21), and [#11243](https://github.com/vortex-data/vortex/issues/11243) tracks the bug until then| A storage result loses the metadata of the type, for example the time unit and time zone of a timestamp. Nothing wraps the result later, and plain `min_max` already wraps its results (EXT-A3) |
38
38
| X13 | A claim is a promise that an extension type makes about its storage: one kind of operation gives the same answer on the values and on their storage values, in both directions. `Eq` means that two values are equal exactly when their storage values are equal. `Ord` means that the value order is the storage order, and it implies `Eq`. An extension type is opaque unless it makes a claim. Each type lists its claims by hand in `ExtVTable::claims`, and the default is no claims. Claims are never derived from which `=` or `<` kernels exist. Built-in dtypes hold claims by a fixed table: `Null`, `Bool`, `Primitive`, `Decimal`, `Utf8` and `Binary` hold both. `Struct`, `List`, `FixedSizeList` and `Map` hold a claim when their children hold it. `Union` and `Variant` hold neither. A type stored as floats inherits the float semantics of Vortex: a total order and bitwise equality | Kernels are registered per encoding, not per dtype, so a missing kernel would silently remove a claim. A type with its own `=` rule still cannot claim `Eq`, because `group by` and hashing run on storage. The built-in table follows what compare supports today (`vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:206-219` at `c56ee8803b`) |
39
39
| X14 | The claims are `#[non_exhaustive] pub enum Claim { Eq, Ord }`, with `Claim::implies` and `DType::holds(Claim)`. A claim defined outside Vortex needs a struct with statics later. That switch is a mechanical change of interfaces, and the logic stays the same | Start simple. Only a third-party claim needs the open form, and none exists yet |
40
40
| X15 |`ExtDType::try_new` checks that the declared storage dtype holds every claim of the type (X27). The claims must not depend on the top-level nullability (X25), so the check runs once. Typing then checks only the top-level dtype of an argument with `DType::holds`| A claim is valid only if the storage holds it. The check is linear in the size of the dtype tree and stops at nested extension dtypes, which were checked when they were built. So nested extension types need no extra work in typing |
|U9| Wrap the results of the storage pushdown| EXT-A3 |`vortex.min_max[vortex.timestamp]` returns timestamps, not `i64` values |
25
+
|~~U9~~|~~Wrap the results of the storage pushdown~~ Dropped, SP4 fixes it (#11243) | EXT-A3 |~~`vortex.min_max[vortex.timestamp]` returns timestamps, not `i64` values~~|
26
26
27
27
## Test requirement
28
28
@@ -142,6 +142,13 @@ caller. The storage declaration of phase 1 replaces this interim form (X27).
142
142
143
143
## U9. Storage pushdown results
144
144
145
+
The maintainer dropped U9 on 2026-11-04 and closed its PR #11188. The PR named the value fields of
146
+
each partial and result by index (`[0, 1]` for `min_max`, `[2, 3]` for `is_sorted`), with a new
147
+
trait that also limited what `finalize` can do. The indices can drift from the real fields, and
148
+
SP4 removes the adapter anyway, so the fix was a half-measure. SP4 fixes EXT-A3 instead, and issue
149
+
[#11243](https://github.com/vortex-data/vortex/issues/11243) tracks the bug until then. The rest
150
+
of this section records the original plan.
151
+
145
152
U9 does not wait for `ResultMapping` (X35). `register_storage_ordered` wraps only `MinMax` and
146
153
`IsSorted` (`vortex-array/src/aggregate_fn/session.rs:195-198` at `c7449d567a`), and each of them
147
154
maps its result in a fixed way:
@@ -180,7 +187,7 @@ On 2026-11-04, the PRs for part 4 stand as follows:
180
187
the other operand, so `Uuid` and `Json` filters still push down. Its tests cover dynamic filter
181
188
updates. It also makes the export of a non-temporal extension scalar as a bare DataFusion
182
189
`ScalarValue` an error. So item 1 of "Order constraints" in part 5 holds.
183
-
- U9 is the open draft #11188.
190
+
- U9 closed without a merge as #11188 on 2026-11-04. Issue #11243 tracks EXT-A3 until SP4.
184
191
- U1 has no PR yet. The Arrow export that lands before it (AX) is the open draft #11224.
185
192
186
193
Three more PRs touch this work:
@@ -265,16 +272,17 @@ Three PRs added or proposed mechanisms for extension semantics:
265
272
-[#10566](https://github.com/vortex-data/vortex/pull/10566) added `ExtensionStoragePushdown`, an
266
273
aggregate adapter with a composite ID, and
267
274
[#10616](https://github.com/vortex-data/vortex/pull/10616) registered it for the temporal types
268
-
and `Uuid`. Both merged on 2026-10-20. U9 fixes its results. In SP4, its body becomes the
269
-
default storage path of aggregate binding, and its composite IDs go away (X10, X21).
275
+
and `Uuid`. Both merged on 2026-10-20. In SP4, its body becomes the default storage path of
276
+
aggregate binding, its composite IDs go away, and its results get their extension dtype (X10,
277
+
X21, #11243).
270
278
-[#10408](https://github.com/vortex-data/vortex/pull/10408) proposed a `between` pushdown for
271
279
`AnyTemporal`. It closed without a merge on 2026-11-03.
272
280
273
281
SP4 replaces the hand-written registrations of the adapter with the claims of the types.
274
282
275
283
[#10769](https://github.com/vortex-data/vortex/pull/10769) (draft) moves the aggregate
276
284
implementations into a new `vortex-aggregates` crate, including the tests of the storage pushdown.
277
-
U9 changes the same code, so the order of the two needs agreement.
285
+
SP4 changes the same code, so the order of the two needs agreement.
278
286
279
287
The new extension types `Duration` ([#10467](https://github.com/vortex-data/vortex/pull/10467))
280
288
and `Interval` ([#10468](https://github.com/vortex-data/vortex/pull/10468)) are open. Both edit
Copy file name to clipboardExpand all lines: research/extension-types/05-implementation.md
+11-11Lines changed: 11 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,7 +18,7 @@ phases 2 to 4 use hardcoded lists of types (X35).
18
18
| --- | --- | --- |
19
19
| 0 | Urgent fixes U1 to U8, the path-consistency test helper, and four fixes outside part 4: the `Timestamp` range (#10953), the `Date` display (EXT-V10), the spatial nested nullability (EXT-N2), and imports that fail on nested nulls (X29) | Nothing |
20
20
| 1 | The storage declaration: `storage_dtype`, `validate_metadata`, strict reads, the move of storage facts into the metadata, and `ExtensionArray::new_unchecked`| U4 and U5, both merged, and the spatial validator fix SN1 |
21
-
| 2 | Declarations with hardcoded opt-ins: `ExtensionUse` with `None`, `Structural` and `Specific`, `extension_use` on both vtables, and the hardcoded lists of X35. Then the declarations of the built-in functions, by family. Nothing reads them yet, so behavior does not change. U9 can land at any time| Nothing |
21
+
| 2 | Declarations with hardcoded opt-ins: `ExtensionUse` with `None`, `Structural` and `Specific`, `extension_use` on both vtables, and the hardcoded lists of X35. Then the declarations of the built-in functions, by family. Nothing reads them yet, so behavior does not change. U9 was dropped (#11243)| Nothing |
22
22
| 3 | Typing: the check that rejects every call that answers `None`| 0 (U1, U2), 2 |
23
23
| 4 | Statistics: readers ignore min and max of types outside the order list, and writers drop the storage fallback | 2 for the readers, 3 for the writers |
24
24
| 5 | Type rules and casts: `ExtVTable::rules` over `ReduceNode`, exact unit casts, and the named lossy unit function | 3 |
@@ -30,8 +30,8 @@ Phases 0 and 1 can run in parallel. Phase 1 builds on U4 and U5, which merged on
30
30
2026-10-30. Its storage declaration includes the strict reads, so it lands after the spatial
31
31
validator fix SN1, which merged on 2026-11-04 (see "Order constraints", items 3 and 6). The Arrow
32
32
export with metadata is part of phase 7, but it lands in phase 0, before U1. Phase 2 adds
33
-
declarations that nothing reads yet, so it can start at any time. U9 changes only the results of
34
-
the composite aggregates, which only tests use. Phase 3 is the critical step: it turns on typing,
33
+
declarations that nothing reads yet, so it can start at any time. U9 was dropped, and SP4 fixes the
34
+
results of the composite aggregates (#11243). Phase 3 is the critical step: it turns on typing,
35
35
and from then on extension types are opaque. Phases 4 and 5 build on it. Phase 6 runs in parallel
36
36
with phases 2 to 5.
37
37
@@ -54,7 +54,7 @@ dotted line without an arrow links an open PR that changes the same code, so the
54
54
coordination. Each node is one PR, and its ID is the ID of that PR in the PR plan of
55
55
[#11114](https://github.com/vortex-data/vortex/issues/11114), or of
56
56
[#11115](https://github.com/vortex-data/vortex/issues/11115) for the claims (CL) and the storage
@@ -253,14 +253,14 @@ breaks something that works today. Those parts must land together, or in a fixed
253
253
The check rejects every call that answers `None`, so a family without its declaration stops
254
254
working. For example, a compare of two `Timestamp` columns would fail when it is built.
255
255
256
-
TY2, U9 and SP4 also need coordination with
256
+
TY2 and SP4 also need coordination with
257
257
[#10769](https://github.com/vortex-data/vortex/pull/10769), which moves the aggregate
258
258
implementations that they change into `vortex-aggregates`.
259
259
260
260
## Phase 0: urgent fixes
261
261
262
262
- Write the path-consistency helper first, because U1 and U2 use it. It merged as #11013.
263
-
- Land U1 to U8 as separate PRs. U9 is listed in phase 2, but it can land at any time.
263
+
- Land U1 to U8 as separate PRs. U9 was dropped on 2026-11-04 (#11243).
264
264
- The maintainer finished [#10953](https://github.com/vortex-data/vortex/pull/10953) with commits
265
265
instead of waiting on its author. It makes `Timestamp` accept every `i64` (X31) and fixes the
266
266
`DateTimeParts` days wrap (EXT-S3). The maintainer commits of 2026-10-30 merge `develop`, update
@@ -369,8 +369,8 @@ implementations that they change into `vortex-aggregates`.
369
369
- The hand-written paths of today stay, and they do the work for the types on the lists. For
370
370
example, the extension compare kernel still compares storage. After phase 3, typing rejects every
371
371
other call before it reaches them.
372
-
- U9 wraps the results of today's `ExtensionStoragePushdown`. Part 4 has the details. It needs no
373
-
declaration, so it can land at any time.
372
+
- U9 was dropped on 2026-11-04. SP4 gives the results of today's `ExtensionStoragePushdown` their
373
+
extension dtype instead, and #11243 tracks the bug until then. Part 4 has the details.
374
374
- Coordinate with [#10769](https://github.com/vortex-data/vortex/pull/10769), which moves the
375
375
aggregate implementations into a new `vortex-aggregates` crate, and with
376
376
[#10837](https://github.com/vortex-data/vortex/pull/10837), which changes
@@ -551,7 +551,7 @@ Every finding of [`agents/findings.md`](agents/findings.md) has a fix in this pl
551
551
| EXT-Q2 | TY1 and the typing check of phase 3, with the hardcoded lists, and then SP3 with the claims |
552
552
| EXT-A1 | SP4 and SP5 |
553
553
| EXT-A2 | Phase 4, and SP6 for the claims |
554
-
| EXT-A3 |U9|
554
+
| EXT-A3 |SP4, tracked in #11243 (U9 was dropped)|
555
555
| EXT-V1 | U3 for the panics, merged in #11014, and phase 6 for the value check of `try_new`|
0 commit comments