You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9748b74
Browse filesBrowse the repository at this point in the historyBrowse files
Scan extension columns over list, map, and string storage in DataFusion (#11225)
## Summary
- Tracking Issue: #11114
DataFusion can't scan a `Vector` column today. For a logical `List`,
`FixedSizeList`, `Map`, `ListView`, `Utf8`, or `Binary` type, the
physical schema compares the layout with the extension dtype instead of
its storage, so the scan fails before it reads a row. The same bug
breaks a `Json` column that the table schema declares as `Utf8`.
## Changes
The layout checks now look through extension types, as the struct check
already did. An extension column keeps its logical layout rather than
the type that `ArrowSession` would export for it, because the V2 source
has no schema adapter, so its batches must match the plan schema
exactly. The export type would make the V2 source emit `Utf8View`
batches for a `Json` column that the table schema declares as `Utf8`.
This only helps types with an Arrow export plugin, such as `Vector` and
`Json`. A type without one still can't be scanned until Vortex can
export it as its storage with extension metadata (AX in #11114).
The first commit fixes a panic that this change would otherwise expose.
The Arrow map export panicked on any dtype other than a map, and an
extension column over map storage can now reach it. It returns an error
instead, as the list exporters already do.
Tests cover a SQL scan of a file with a `Vector` column, a V2 projection
over a `Json` column, each layout check of the physical schema, and the
map export of a non-map array.
---------
Signed-off-by: Connor Tsui <[email protected]>
0 commit comments