You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d7aa24e
Browse filesBrowse the repository at this point in the historyBrowse files
Make successful Arrow plugin imports final (#11464)
## Summary
TurboQuant stores its Arrow metadata as base64 without an encoding
marker. Vortex's fallback for registered Vortex IDs decodes base64 only
when marked, so it reads TurboQuant's metadata as literal text and
rejects valid arrays, even after a plugin has imported them
successfully. An exporter that declines and delegates to storage
conversion can hit the same check.
## Changes
Successful plugin imports are final because the existing dtype and
length checks already validate their result. Export fallback asks the
registered importer to decode the field before comparing types, since
the same metadata can use different byte encodings. The complete dtype
check, apart from outer nullability, and the physical storage check
remain. Without a matching importer, the field must contain the source
type's serialized metadata.
Map entries import as a plain struct because the map's dtype comes from
its key and value fields. Running a plugin on the entries field could
produce a different dtype from schema inference. Key and value plugins
still run.
Tests cover native metadata imports and export round trips, invalid
metadata, and physical storage mismatches. A byte-collision case checks
that equal metadata bytes cannot hide different type parameters. Map
tests cover ignored entries plugins and length checks on value imports.
## API Changes
Import plugins no longer run on a map's entries field. Extension types
on the key and value fields still reach their plugins.
Signed-off-by: Connor Tsui <[email protected]>
0 commit comments