Repository navigation
Make Arrow imports of known extension types strict - #11223
Draft
connortsui20 wants to merge 6 commits into
Draft
connortsui20 wants to merge 6 commits into
connortsui20 wants to merge 6 commits into
Conversation
12 of 49 tasks
connortsui20
added a commit
that referenced
this pull request
Nov 4, 2026
DP (#11185) merged on 2026-11-04. SN2, AX and DS are open drafts as Signed-off-by: Connor Tsui <[email protected]> #11223, #11224 and #11225, so the graph and parts 4 and 5 link them.
ArrowSession::from_arrow_storage imports an array whose Arrow layout is the storage layout of an extension type. Producers such as pyarrow mark nested levels nullable even when they hold no nulls, or hold nulls only under null rows. The helper drops the nulls that no row reads, fails on a null in a valid row, and checks field names, order, and types first. Its errors name the extension type. The canonical import applies the same rule while it converts. Each struct and list level passes the visibility of its slots down to its children, and each level drops the nulls that a null row above it hides. An importer never sees the rows above its column, so the import rebuilds a known-type column under a null row in the form that its dtype declares before the importer runs. So a known type imports under a null row of a struct, list, or fixed-size list that hides nulls in the levels that the type declares non-nullable. The import also drops the nulls that a null fixed-size list hides in its non-nullable elements, which used to fail. This replaces the strip of non-nullable struct children, which dropped every null there without checking that a null row hid it. So a null in a valid row of a non-nullable struct field, or of a non-nullable level below one, now fails where it used to import as a placeholder value. arrow-rs checks an array only against its own type, so arrays that arrow-rs validated reach this failure too, when the field or schema of the import marks a level non-nullable that the type of the array marks nullable. A non-nullable dictionary of fixed-size list elements now drops the key nulls under a null list, as a dictionary struct field already did, so it fails in the same way when a key under them points outside its values, as in a dictionary of only nulls. The ArrowImportVTable docs now state when an importer may defer: only for an input that another importer under the same name owns. Importer errors for a field now carry the field name. Signed-off-by: Connor Tsui <[email protected]>
The Vector importer used to decline a field that was not a FixedSizeList of floats with non-nullable elements, so the column imported as plain storage. It now fails on a field that cannot store vectors, and accepts nullable elements, such as the ones pyarrow writes. The array import drops the nulls under null vectors and fails on a null element of a valid vector. Signed-off-by: Connor Tsui <[email protected]>
The arrow.uuid and arrow.json importers declined a field that broke the canonical spec of its type, so the column imported as plain storage. They now fail on such a field and build their arrays with try_new instead of panicking on a storage mismatch. The arrow.uuid importer uses the check that arrow-rs implements, which matches the spec. The arrow.json check of arrow-rs is stricter than the spec, which reserves the fields of the metadata object for later additions, and Arrow C++ reads a missing metadata key as empty metadata. So the arrow.json importer accepts a missing key, an empty string, and any JSON object, and it still checks the storage type. The arrow.json exporter now checks its target field by the same rule, so a JSON column exports back to the field that it imports from. Signed-off-by: Connor Tsui <[email protected]>
When GeoArrow rejected a field, for example a list view layout or metadata that is not JSON, the spatial importers fell back to the extension name alone. The fallback guessed the dimension from the field names and dropped the CRS, and the array import then declined the same field, so the column became plain storage under an extension dtype. A field that names a GeoArrow type now has to parse as that type, and the WKB importer fails instead of declining. Signed-off-by: Connor Tsui <[email protected]>
The spatial importers declined the LargeList layout, which GeoArrow accepts, and kept the nullability that Arrow gives coordinates, so a column with nullable-marked coordinates failed to match its own extension dtype. They now import through ArrowSession::from_arrow_storage. Nullable-marked levels import as the non-nullable levels that GeoArrow declares, nulls under null geometries are dropped, and a null coordinate of a valid geometry fails with an error that names the type. Signed-off-by: Connor Tsui <[email protected]>
A literal whose field no importer accepts already stays in DataFusion. Now that importers fail instead of declining, an importer can accept a field and still reject its value, for example a vector literal with a null element. That failure used to fail the whole conversion. It now leaves the literal to DataFusion as well. Signed-off-by: Connor Tsui <[email protected]>
connortsui20
force-pushed
the
ct/arrow-import-strict
branch
from
November 9, 2026 18:40
d09769e to
cf64b5c
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Arrow importers of known extension types decline input that they can't handle, and a declined column silently imports as plain storage. For example, pyarrow marks list elements nullable, so a pyarrow vector column loses its
Vectortype. Every importer of a known type (the spatial types,Vector,Uuid, andJson) now converts such input into the form that its type declares, or fails when the data falls outside the domain of the type (X29).Changes
Arrow lets a child hold anything under a null row of its parent, since nothing reads it, so the import drops those masked nulls, while a null in a valid row of a non-nullable level fails. The canonical import passes the visible rows of each level to its children, so a known-type column under a null struct row (e.g., from an outer join) imports as its type instead of failing. The spatial importers lose their name-only fallback, because it guessed the dimension and dropped the CRS.
List views, maps, run-end arrays, and dictionaries don't pass their visible rows down yet. SN3 in #11114 covers them.
API Changes
arrow.uuidfield overFixedSizeBinary(8).vortex.tensor.vectorfield with nullable elements imports asVector, and GeoArrow input in theLargeListlayout or with nullable-marked coordinates imports as its spatial type.ArrowSession::from_arrow_record_batch.ArrowSession::from_arrow_storage.