Skip to content

Commit d869412

Browse files
committed
Add the SN3, TY5, and DS2 follow-ups to the extension type plan
SN3 drops masked nulls below every nested level after SN2. TY5 declares the type-specific functions, which the typing check needs. DS2 rejects DataFusion columns that mix an extension type and plain storage. #11114 now lists all three. Signed-off-by: Connor Tsui <[email protected]>
1 parent 8ee0926 commit d869412

1 file changed

Lines changed: 23 additions & 1 deletion

File tree

‎research/extension-types/05-implementation.md‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,12 +72,14 @@ flowchart TD
7272
SN1["SN1 spatial validators reject nullable nested levels<br/>#11183"]:::merged
7373
DP["DP dictionary peel before the Arrow importers<br/>#11185"]:::merged
7474
SN2["SN2 Arrow importers of known types convert or fail<br/>#11223"]:::open
75+
SN3["SN3 Arrow import drops masked nulls below every nested level"]
7576
end
7677
PC --> U2
7778
U2 -->|"item 1, lands together"| DF
7879
PC --> U1
7980
AX -.->|"item 2"| U1
8081
DP --> SN2
82+
SN2 --> SN3
8183
8284
subgraph PH1["Phase 1"]
8385
SD["SD storage declaration and strict reads<br/>storage_dtype, validate_metadata, facts in metadata<br/>#11184"]:::open
@@ -94,12 +96,14 @@ flowchart TD
9496
TY2["TY2 aggregates declare Specific"]
9597
TY3["TY3 structural functions declare Structural"]
9698
TY4["TY4 list_contains declares Specific"]
99+
TY5["TY5 type-specific functions declare Specific"]
97100
U9["U9 wrap storage pushdown results<br/>#11188"]:::open
98101
end
99102
FD --> TY1
100103
FD --> TY2
101104
FD --> TY3
102105
FD --> TY4
106+
FD --> TY5
103107
104108
subgraph PH3["Phase 3"]
105109
TY["TY typing check, with the phase 4 writers and DuckDB pushdown"]
@@ -108,6 +112,7 @@ flowchart TD
108112
TY2 -->|"item 7"| TY
109113
TY3 -->|"item 7"| TY
110114
TY4 -->|"item 7"| TY
115+
TY5 -->|"item 7"| TY
111116
U1 --> TY
112117
DF --> TY
113118
TY -.->|"items 4 and 5"| PO["Any PartialOrd change for Scalar, XQ8"]
@@ -138,7 +143,9 @@ flowchart TD
138143
BD1["BD1 converters for DuckDB, Python and JNI"]
139144
BD3["BD3 typed array view and developer guide"]
140145
DS["DS DataFusion physical schema for extension dtypes<br/>#11225"]:::open
146+
DS2["DS2 reject columns that mix an extension type and plain storage"]
141147
end
148+
DS --> DS2
142149
143150
subgraph CLAIMS["Claims, #11115"]
144151
CL1["CL1 Claim, DType::holds, ExtVTable::claims"]
@@ -242,7 +249,7 @@ breaks something that works today. Those parts must land together, or in a fixed
242249
have one. The maintainer accepted that such files stop reading, so no read plugin is needed.
243250
SN1 ([#11183](https://github.com/vortex-data/vortex/pull/11183)) merged on 2026-11-04 as
244251
`956c1ea188`, so this order holds.
245-
7. The declarations of phase 2 (TY1 to TY4) land before the typing check of phase 3, or with it.
252+
7. The declarations of phase 2 (TY1 to TY5) land before the typing check of phase 3, or with it.
246253
The check rejects every call that answers `None`, so a family without its declaration stops
247254
working. For example, a compare of two `Timestamp` columns would fail when it is built.
248255

@@ -285,6 +292,12 @@ implementations that they change into `vortex-aggregates`.
285292
declines it first, and the session then retries with the value type. DP peels the dictionary
286293
before the importers run, so importers never see dictionaries and can fail. The result stays
287294
`Dict(codes, Ext(values))`.
295+
- SN3 builds on SN2, and the import then drops masked nulls below every nested level. SN2 drops
296+
them through structs, lists, large lists, and fixed-size lists. List views, maps, run-end arrays,
297+
and dictionaries do not pass their visible rows to their children. So a masked null below one of
298+
them in a non-nullable level fails the import. A masked dictionary key can also point outside
299+
the values, and `new_null_array` builds such keys. SN3 gives these keys a valid target. The
300+
maintainer agreed that the import always drops masked nulls.
288301
- V10 gives the `Display` of `Date` the fallback that #10953 gives `Timestamp`, in its own PR on
289302
`develop` (EXT-V10). It merged as #11187 on 2026-11-04.
290303
- U4 is the interim form of the storage declaration: `with_nullability` stops validating again, and
@@ -350,6 +363,9 @@ implementations that they change into `vortex-aggregates`.
350363
3. TY3: the structural functions answer `Structural`.
351364
4. TY4: `list_contains` answers `Specific` for elements on the equality list. The row encoding
352365
rejects extension types today, so it stays as it is until SP5.
366+
5. TY5: the type-specific functions answer `Specific` (X18). These are `replace_time_zone`, the
367+
spatial and tensor functions, `json_to_variant`, and the `GeometryAabb` aggregate. Without
368+
these declarations, the typing check of phase 3 rejects these functions.
353369
- The hand-written paths of today stay, and they do the work for the types on the lists. For
354370
example, the extension compare kernel still compares storage. After phase 3, typing rejects every
355371
other call before it reaches them.
@@ -439,6 +455,12 @@ implementations that they change into `vortex-aggregates`.
439455
ListView and Utf8 storage (`vortex-datafusion/src/convert/schema.rs`), which probably breaks
440456
`Vector` columns. DS fixes it, once a regression test confirms the failure. A regression test
441457
confirmed it, and DS is the open draft #11225.
458+
- DS2: the DataFusion physical schema rejects a table column whose files mix an extension type and
459+
its plain storage. Today such a table labels the plain rows with the extension type, with no
460+
error. `Json` and `Vector` columns have this bug, and AX adds every type without a plugin. The
461+
proposed fix copies the `ARROW:extension:*` metadata onto the physical field of a file only when
462+
the column in that file has the extension dtype. Otherwise, the schema fails. The final review of
463+
AX found this bug.
442464
- Add converter traits for DuckDB, Python and JNI, modeled on the Arrow registry. Move the
443465
hard-coded type lists into converters, and add converters where the implicit conversions of today
444466
stop.

0 commit comments

Comments
 (0)