You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 19ae853
Browse filesBrowse the repository at this point in the historyBrowse files
Compare nested extension types by storage in Vortex ID exports (#11351)
## Summary
- Tracking Issue: #11114
Follow-up to #11301, which checks that the Arrow type of a field that
names a Vortex ID imports as the storage of the array. That import can
lose information about nested extension types: the UUID plugin imports
every `arrow.uuid` field without a version, and a session that does not
know a nested Vortex ID imports it as plain storage. So since #11301,
some valid exports fail. For example, an extension type whose storage is
a struct with a `Uuid(v4)` field can no longer export to its own
`to_arrow_storage_field`, which worked before.
## Changes
The check now compares nested extension types by their storage types,
and keeps the exact comparison at the top level. Nested fields keep
their checks, because the exporter checks each nested field when it
reaches it, and the temporal executor still rejects a time unit that
does not match.
The alternative was to require every import plugin to give back exactly
the type that it exported. `ArrowImportVTable` does not require that
today, and a plugin for a canonical Arrow type such as `arrow.uuid` has
no standard place for Vortex-only metadata such as the UUID version.
Exact round trips for the plugins that Vortex owns are still worth
having, but as a separate goal with its own property tests, not as an
assumption of this check.
This PR also does the comment pass from the review of #11301 on the
export dispatch, the map import, and the map test. The map import still
runs import plugins for the extension name of the entries field, which
`from_arrow_datatype` ignores. The comment now says so instead of
claiming that the two match, and a test relies on those plugins running,
so the behavior is unchanged. The metadata check that the import and the
export both copied is now one helper. New tests cover the nested case
(it fails without the fix), an alias of a Vortex ID, and a plain array
that exports to a field that names a Vortex ID.
Signed-off-by: Connor Tsui <[email protected]>
0 commit comments