Skip to content

Commit fa50f1f

Browse files
committed
Add order constraints and a findings table to the extension plan
The plan now lists the five places where a change that lands alone breaks working behavior, and maps every finding to the change or phase that fixes it. Signed-off-by: Connor Tsui <[email protected]>
1 parent 99c93c7 commit fa50f1f

1 file changed

Lines changed: 77 additions & 5 deletions

File tree

‎research/extension-types/05-implementation.md‎

Lines changed: 77 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -38,19 +38,55 @@ flowchart LR
3838
L2["Low priority: DateTimeParts under the wrapper"]
3939
```
4040

41-
Phases 0 and 1 can run in parallel, and so can the Arrow part of phase 7. Phase 2 adds types and
42-
methods but changes no behavior, so it is small. U9 ends phase 2, because it needs the result
43-
mappings. Phase 3 is the critical step: it turns on typing and removes the hand-written extension
44-
paths. Phases 4 and 5 build on it. Phase 6 runs in parallel with phases 2 to 5.
41+
Phases 0 and 1 can run in parallel, except that phase 1 builds on U4 and U5 (see "Order
42+
constraints"). The Arrow part of phase 7 can start at any time. Phase 2 adds types and methods that
43+
nothing reads yet, so it is small. U9 ends phase 2, because it needs the result mappings. It
44+
changes only the results of the composite aggregates, which only tests use. Phase 3 is the critical
45+
step: it turns on typing and removes the hand-written extension paths. Phases 4 and 5 build on it.
46+
Phase 6 runs in parallel with phases 2 to 5.
4547

4648
The default storage paths of phase 3 and the rules of phase 5 build their results with
4749
`ExtensionArray::new_unchecked`. So phase 1 adds that constructor, and phase 6 only adds the value
4850
check to `try_new`.
4951

52+
## Order constraints
53+
54+
Most PRs can land in any order that the graph allows. In five places, a change that lands alone
55+
breaks something that works today. Those parts must land together, or in a fixed order:
56+
57+
1. U2 lands together with the DataFusion literal conversion (EXT-B1), the DataFusion dynamic filter
58+
path ([#10865](https://github.com/vortex-data/vortex/pull/10865),
59+
[#10869](https://github.com/vortex-data/vortex/pull/10869),
60+
[#10895](https://github.com/vortex-data/vortex/pull/10895)), and a check of the DuckDB literal
61+
conversion. Otherwise a filter that compares a `Uuid` or `Json` column with a storage literal
62+
fails when it is built, instead of being pushed down.
63+
2. The Arrow export of phase 7 lands before U1, or together with it. U1 makes a storage-like Arrow
64+
target type an error, for example `Int64` for a timestamp column in the Python
65+
`to_arrow_array(arrow_type=...)`. The export with metadata gives those callers a path that loses
66+
nothing.
67+
3. Phase 1 builds on U4 and U5. U4 changes `ExtDType::try_with_vtable`, where phase 1 adds the check
68+
of the claims. U5 changes the `Date`, `Time` and `Timestamp` vtables, which get their claims in
69+
phase 1. So phase 1 waits for both to merge, or it is based on them.
70+
4. The aggregates PR of phase 3 includes two more changes:
71+
- The writers' part of phase 4. Once `Min` and `Max` reject types without `Ord`, the storage
72+
fallback of `aggregate_state_dtype` would write a storage-typed state for those columns. No
73+
version wrote that form before.
74+
- DuckDB aggregate pushdown reads the claims. Today it excludes only `Uuid`, so `max` over a
75+
`Json` column would fail in DuckDB. Whether DuckDB falls back to its own aggregation when
76+
binding fails is not checked.
77+
5. A change of `PartialOrd` for `Scalar` (XQ8) lands with the aggregates PR of phase 3, or after it.
78+
Today `min` and `max` merge partial results with `vortex_expect`, so an earlier change can panic.
79+
80+
Phase 2 and U9 also need coordination with
81+
[#10769](https://github.com/vortex-data/vortex/pull/10769), which moves the aggregate
82+
implementations that they change into `vortex-aggregates`.
83+
5084
## Phase 0: urgent fixes
5185

5286
- Write the path-consistency helper first, because U1 and U2 use it.
5387
- Land U1 to U8 as separate PRs. U9 needs the declaration types, so it lands in phase 2.
88+
- Land the Arrow part of phase 7 before U1, and change the DataFusion literal path in the PR of U2
89+
("Order constraints", items 1 and 2).
5490
- Some existing code depends on a leak, for example a comparison of an extension column with a
5591
storage literal. The full test suite finds these callers. Fix each one with an explicit
5692
`ext_storage`.
@@ -137,7 +173,8 @@ check to `try_new`.
137173
only phase 1, so it can land early.
138174
- Writers drop the storage fallback of `aggregate_state_dtype`, so types without `Ord` get no min
139175
and max. Types with `Ord` keep their logical `Min` and `Max`, so the format does not change. This
140-
needs phase 3, because `Min` and `Max` reject types without `Ord` only after it.
176+
lands in the aggregates PR of phase 3, because `Min` and `Max` reject types without `Ord` only
177+
after it ("Order constraints", item 4).
141178
- Pruning compares logical statistics through the claims, or unwraps them to storage bounds. The
142179
stats rewrite rules live in `vortex-expr`
143180
([#10734](https://github.com/vortex-data/vortex/pull/10734)).
@@ -196,6 +233,41 @@ check to `try_new`.
196233
- Out of scope: the lossy primitive casts (the known issue in `decisions.md`), and the Rust traits
197234
of `Scalar` (XQ8).
198235

236+
## Findings by phase
237+
238+
Every finding of [`agents/findings.md`](agents/findings.md) has a fix in this plan:
239+
240+
| Finding | Fixed by |
241+
| --- | --- |
242+
| EXT-P1 | U1 and U2. Phase 3 keeps the paths consistent, because its rewrite runs on every path |
243+
| EXT-C1, EXT-C2, EXT-C3 | U1 |
244+
| EXT-Q1, EXT-Q3 | U2 |
245+
| EXT-Q2 | The compare PR of phase 3, with the claims of phase 1 |
246+
| EXT-A1 | The aggregates PR and the row encoding PR of phase 3 |
247+
| EXT-A2 | Phase 4 |
248+
| EXT-A3 | U9, at the end of phase 2 |
249+
| EXT-V1 | U3 for the panics, and phase 6 for the value check of `try_new` |
250+
| EXT-V2, EXT-V7 | U6 |
251+
| EXT-V3 | U7 |
252+
| EXT-V4, EXT-V5, EXT-V8 | U5 |
253+
| EXT-V6 | Phase 6. The trait allows a type without a value check, so a type adds one only where its values have constraints |
254+
| EXT-N1 | U4. NU2 in phase 6 is the long-term form |
255+
| EXT-S1 | The low-priority lift rule. Code handles both shapes today, so no result is wrong |
256+
| EXT-S2 | The low-priority move of `DateTimeParts` under the wrapper. U2 fixes its compare kernel |
257+
| EXT-B1 | U2 for literals, and phase 7 for schema and scalar conversion |
258+
| EXT-B2 | U1 for the storage leak, and phase 7 for the export with metadata |
259+
| EXT-B3 | U1 for `vx.scalar`, and phase 7 for `as_py` |
260+
| EXT-B4 | The aggregates PR of phase 3 for pushdown, and phase 7 for the converter hook |
261+
| EXT-B5 | Phase 7 |
262+
| EXT-B6 | U1 removes the casts out of extension types that this lead is about. Check it again after U1 |
263+
| EXT-R1 | U8 |
264+
| EXT-R2 | Any time, as its own small PR |
265+
| EXT-R3 | Phase 1 gives `ForeignExtDType` no claims, and phase 3 rejects semantic operations on it |
266+
| EXT-R4 | Phase 1 for the contract on `ExtVTable`, and phase 7 for the developer guide |
267+
268+
The lossy primitive casts in `decisions.md` are not an extension finding, and they stay out of
269+
scope.
270+
199271
## Test strategy
200272

201273
- **Path consistency.** Every semantic operation gives the same answer on the whole array, on one

0 commit comments

Comments
 (0)