Skip to content

Scan extension columns over list, map, and string storage in DataFusion - #11225

Merged
connortsui20 merged 2 commits into
developfrom
ct/datafusion-ext-physical-schema
Nov 5, 2026
Merged

connortsui20 merged 2 commits into
developfrom
ct/datafusion-ext-physical-schema

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Summary

DataFusion can't scan a Vector column today. Arrow carries an extension column as its storage plus field metadata, but the physical schema compares the table's Arrow layout with the extension dtype instead of its storage, so any extension column over list, map, or string storage fails before the scan reads a row. The same bug breaks a Json column that the table schema declares as Utf8.

Changes

The layout checks now look through extension types, as the struct check already did. An extension column keeps the table's layout (e.g., Utf8 rather than Utf8View for Json) instead of the type that ArrowSession would export for it, because the V2 source has no schema adapter, so its batches must match the plan schema exactly.

The Arrow map export returns an error instead of panicking on a dtype other than a map, as the list exporters already do, since an extension column over map storage can reach it now.

Note that this only helps types with an Arrow export plugin, such as Vector and Json, until #11224 exports the other types as their storage. A table field without the extension name now reads a Json or Vector column as plain storage (against X24), and #11268 makes that fail, so the two PRs should merge together.

Tests cover a SQL scan of a Vector column, a V2 projection over a Json column, each layout check of the physical schema, and the map export of a non-map array.

@connortsui20 connortsui20 added the changelog/fix A bug fix label Nov 4, 2026
@connortsui20 connortsui20 mentioned this pull request Nov 4, 2026
12 of 49 tasks
connortsui20 added a commit that referenced this pull request Nov 4, 2026
DP (#11185) merged on 2026-11-04. SN2, AX and DS are open drafts as

Signed-off-by: Connor Tsui <[email protected]>
#11223, #11224 and #11225, so the graph and parts 4 and 5 link them.
@connortsui20
connortsui20 marked this pull request as ready for review November 5, 2026 15:47
Comment thread vortex-datafusion/src/convert/schema.rs Resolved
Comment thread vortex-datafusion/src/convert/schema.rs Resolved
@connortsui20
connortsui20 force-pushed the ct/datafusion-ext-physical-schema branch from 4060fe0 to 055c280 Compare November 5, 2026 16:24
@connortsui20

Copy link
Copy Markdown
Member Author

whoops claude pushed a bunch of unnecessary changes

@connortsui20
connortsui20 force-pushed the ct/datafusion-ext-physical-schema branch from 9d81df5 to 1ed554d Compare November 5, 2026 17:49
`to_arrow_map` executed every input to a `MapArray`, and
`Canonical::into_map` panics when the dtype is not a map, for example
for an extension type over map storage. The list, fixed-size list, and
list view exporters already check the dtype first, so the map exporter
now does the same.

Signed-off-by: Connor Tsui <[email protected]>
The layout arms of `calculate_physical_field_type` matched the extension
dtype itself instead of its storage, so they rejected every extension
column whose logical type was a `List`, `FixedSizeList`, `Map`,
`ListView`, `Utf8`, or `Binary` layout. A `Vector` column failed the scan
before it read a row. The struct arm already looked through extension
types, so that walk now runs before the match and every layout arm uses
it.

Signed-off-by: Connor Tsui <[email protected]>
@connortsui20
connortsui20 force-pushed the ct/datafusion-ext-physical-schema branch from 1ed554d to f65ff0d Compare November 5, 2026 18:47
@connortsui20
connortsui20 merged commit 9748b74 into develop Nov 5, 2026
70 of 71 checks passed
@connortsui20
connortsui20 deleted the ct/datafusion-ext-physical-schema branch November 5, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants