You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d869412
Browse filesBrowse the repository at this point in the historyBrowse files
Add the SN3, TY5, and DS2 follow-ups to the extension type plan
SN3 drops masked nulls below every nested level after SN2. TY5 declares the
type-specific functions, which the typing check needs. DS2 rejects DataFusion
columns that mix an extension type and plain storage. #11114 now lists all
three.
Signed-off-by: Connor Tsui <[email protected]>
TY["TY typing check, with the phase 4 writers and DuckDB pushdown"]
@@ -108,6 +112,7 @@ flowchart TD
108
112
TY2 -->|"item 7"| TY
109
113
TY3 -->|"item 7"| TY
110
114
TY4 -->|"item 7"| TY
115
+
TY5 -->|"item 7"| TY
111
116
U1 --> TY
112
117
DF --> TY
113
118
TY -.->|"items 4 and 5"| PO["Any PartialOrd change for Scalar, XQ8"]
@@ -138,7 +143,9 @@ flowchart TD
138
143
BD1["BD1 converters for DuckDB, Python and JNI"]
139
144
BD3["BD3 typed array view and developer guide"]
140
145
DS["DS DataFusion physical schema for extension dtypes<br/>#11225"]:::open
146
+
DS2["DS2 reject columns that mix an extension type and plain storage"]
141
147
end
148
+
DS --> DS2
142
149
143
150
subgraph CLAIMS["Claims, #11115"]
144
151
CL1["CL1 Claim, DType::holds, ExtVTable::claims"]
@@ -242,7 +249,7 @@ breaks something that works today. Those parts must land together, or in a fixed
242
249
have one. The maintainer accepted that such files stop reading, so no read plugin is needed.
243
250
SN1 ([#11183](https://github.com/vortex-data/vortex/pull/11183)) merged on 2026-11-04 as
244
251
`956c1ea188`, so this order holds.
245
-
7. The declarations of phase 2 (TY1 to TY4) land before the typing check of phase 3, or with it.
252
+
7. The declarations of phase 2 (TY1 to TY5) land before the typing check of phase 3, or with it.
246
253
The check rejects every call that answers `None`, so a family without its declaration stops
247
254
working. For example, a compare of two `Timestamp` columns would fail when it is built.
248
255
@@ -285,6 +292,12 @@ implementations that they change into `vortex-aggregates`.
285
292
declines it first, and the session then retries with the value type. DP peels the dictionary
286
293
before the importers run, so importers never see dictionaries and can fail. The result stays
287
294
`Dict(codes, Ext(values))`.
295
+
- SN3 builds on SN2, and the import then drops masked nulls below every nested level. SN2 drops
296
+
them through structs, lists, large lists, and fixed-size lists. List views, maps, run-end arrays,
297
+
and dictionaries do not pass their visible rows to their children. So a masked null below one of
298
+
them in a non-nullable level fails the import. A masked dictionary key can also point outside
299
+
the values, and `new_null_array` builds such keys. SN3 gives these keys a valid target. The
300
+
maintainer agreed that the import always drops masked nulls.
288
301
- V10 gives the `Display` of `Date` the fallback that #10953 gives `Timestamp`, in its own PR on
289
302
`develop` (EXT-V10). It merged as #11187 on 2026-11-04.
290
303
- U4 is the interim form of the storage declaration: `with_nullability` stops validating again, and
@@ -350,6 +363,9 @@ implementations that they change into `vortex-aggregates`.
350
363
3. TY3: the structural functions answer `Structural`.
351
364
4. TY4: `list_contains` answers `Specific` for elements on the equality list. The row encoding
352
365
rejects extension types today, so it stays as it is until SP5.
366
+
5. TY5: the type-specific functions answer `Specific` (X18). These are `replace_time_zone`, the
367
+
spatial and tensor functions, `json_to_variant`, and the `GeometryAabb` aggregate. Without
368
+
these declarations, the typing check of phase 3 rejects these functions.
353
369
- The hand-written paths of today stay, and they do the work for the types on the lists. For
354
370
example, the extension compare kernel still compares storage. After phase 3, typing rejects every
355
371
other call before it reaches them.
@@ -439,6 +455,12 @@ implementations that they change into `vortex-aggregates`.
439
455
ListView and Utf8 storage (`vortex-datafusion/src/convert/schema.rs`), which probably breaks
440
456
`Vector` columns. DS fixes it, once a regression test confirms the failure. A regression test
441
457
confirmed it, and DS is the open draft #11225.
458
+
- DS2: the DataFusion physical schema rejects a table column whose files mix an extension type and
459
+
its plain storage. Today such a table labels the plain rows with the extension type, with no
460
+
error. `Json` and `Vector` columns have this bug, and AX adds every type without a plugin. The
461
+
proposed fix copies the `ARROW:extension:*` metadata onto the physical field of a file only when
462
+
the column in that file has the extension dtype. Otherwise, the schema fails. The final review of
463
+
AX found this bug.
442
464
- Add converter traits for DuckDB, Python and JNI, modeled on the Arrow registry. Move the
443
465
hard-coded type lists into converters, and add converters where the implicit conversions of today
0 commit comments