Skip to content

Make Arrow imports of known extension types strict - #11223

Draft
connortsui20 wants to merge 6 commits into
developfrom
ct/arrow-import-strict
Draft

connortsui20 wants to merge 6 commits into
developfrom
ct/arrow-import-strict

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Summary

The Arrow importers of known extension types decline input that they can't handle, and a declined column silently imports as plain storage. For example, pyarrow marks list elements nullable, so a pyarrow vector column loses its Vector type. Every importer of a known type (the spatial types, Vector, Uuid, and Json) now converts such input into the form that its type declares, or fails when the data falls outside the domain of the type (X29).

Changes

Arrow lets a child hold anything under a null row of its parent, since nothing reads it, so the import drops those masked nulls, while a null in a valid row of a non-nullable level fails. The canonical import passes the visible rows of each level to its children, so a known-type column under a null struct row (e.g., from an outer join) imports as its type instead of failing. The spatial importers lose their name-only fallback, because it guessed the dimension and dropped the CRS.

List views, maps, run-end arrays, and dictionaries don't pass their visible rows down yet. SN3 in #11114 covers them.

API Changes

  • Malformed Arrow input for a known extension type fails instead of importing as plain storage, e.g., an arrow.uuid field over FixedSizeBinary(8).
  • A vortex.tensor.vector field with nullable elements imports as Vector, and GeoArrow input in the LargeList layout or with nullable-marked coordinates imports as its spatial type.
  • A known extension name on the outer field of a run-end encoded field fails, since Arrow puts it on the values field.
  • A null in a valid row of a non-nullable struct field, or of a non-nullable level below one, fails where it used to import as a placeholder value. This includes arrays that arrow-rs already validated, when the schema passed to the import is stricter than the array, e.g., ArrowSession::from_arrow_record_batch.
  • Adds ArrowSession::from_arrow_storage.

@connortsui20 connortsui20 added the changelog/break A breaking API change label Nov 4, 2026
@connortsui20 connortsui20 mentioned this pull request Nov 4, 2026
12 of 49 tasks
connortsui20 added a commit that referenced this pull request Nov 4, 2026
DP (#11185) merged on 2026-11-04. SN2, AX and DS are open drafts as

Signed-off-by: Connor Tsui <[email protected]>
#11223, #11224 and #11225, so the graph and parts 4 and 5 link them.
ArrowSession::from_arrow_storage imports an array whose Arrow layout is the
storage layout of an extension type. Producers such as pyarrow mark nested
levels nullable even when they hold no nulls, or hold nulls only under null
rows. The helper drops the nulls that no row reads, fails on a null in a
valid row, and checks field names, order, and types first. Its errors name
the extension type.

The canonical import applies the same rule while it converts. Each struct
and list level passes the visibility of its slots down to its children, and
each level drops the nulls that a null row above it hides. An importer
never sees the rows above its column, so the import rebuilds a known-type
column under a null row in the form that its dtype declares before the
importer runs. So a known type imports under a null row of a struct, list,
or fixed-size list that hides nulls in the levels that the type declares
non-nullable. The import also drops the nulls that a null fixed-size list
hides in its non-nullable elements, which used to fail.

This replaces the strip of non-nullable struct children, which dropped
every null there without checking that a null row hid it. So a null in a
valid row of a non-nullable struct field, or of a non-nullable level below
one, now fails where it used to import as a placeholder value. arrow-rs
checks an array only against its own type, so arrays that arrow-rs
validated reach this failure too, when the field or schema of the import
marks a level non-nullable that the type of the array marks nullable. A
non-nullable dictionary of fixed-size list elements now drops the key nulls
under a null list, as a dictionary struct field already did, so it fails in
the same way when a key under them points outside its values, as in a
dictionary of only nulls.

The ArrowImportVTable docs now state when an importer may defer: only for
an input that another importer under the same name owns. Importer errors
for a field now carry the field name.

Signed-off-by: Connor Tsui <[email protected]>
The Vector importer used to decline a field that was not a FixedSizeList of
floats with non-nullable elements, so the column imported as plain storage.
It now fails on a field that cannot store vectors, and accepts nullable
elements, such as the ones pyarrow writes. The array import drops the nulls
under null vectors and fails on a null element of a valid vector.

Signed-off-by: Connor Tsui <[email protected]>
The arrow.uuid and arrow.json importers declined a field that broke the
canonical spec of its type, so the column imported as plain storage. They
now fail on such a field and build their arrays with try_new instead of
panicking on a storage mismatch.

The arrow.uuid importer uses the check that arrow-rs implements, which
matches the spec. The arrow.json check of arrow-rs is stricter than the
spec, which reserves the fields of the metadata object for later additions,
and Arrow C++ reads a missing metadata key as empty metadata. So the
arrow.json importer accepts a missing key, an empty string, and any JSON
object, and it still checks the storage type. The arrow.json exporter now
checks its target field by the same rule, so a JSON column exports back to
the field that it imports from.

Signed-off-by: Connor Tsui <[email protected]>
When GeoArrow rejected a field, for example a list view layout or metadata
that is not JSON, the spatial importers fell back to the extension name
alone. The fallback guessed the dimension from the field names and dropped
the CRS, and the array import then declined the same field, so the column
became plain storage under an extension dtype. A field that names a GeoArrow
type now has to parse as that type, and the WKB importer fails instead of
declining.

Signed-off-by: Connor Tsui <[email protected]>
The spatial importers declined the LargeList layout, which GeoArrow
accepts, and kept the nullability that Arrow gives coordinates, so a column
with nullable-marked coordinates failed to match its own extension dtype.
They now import through ArrowSession::from_arrow_storage. Nullable-marked
levels import as the non-nullable levels that GeoArrow declares, nulls under
null geometries are dropped, and a null coordinate of a valid geometry fails
with an error that names the type.

Signed-off-by: Connor Tsui <[email protected]>
A literal whose field no importer accepts already stays in DataFusion. Now
that importers fail instead of declining, an importer can accept a field
and still reject its value, for example a vector literal with a null
element. That failure used to fail the whole conversion. It now leaves the
literal to DataFusion as well.

Signed-off-by: Connor Tsui <[email protected]>
@connortsui20
connortsui20 force-pushed the ct/arrow-import-strict branch from d09769e to cf64b5c Compare November 9, 2026 18:40

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant