Repository navigation
Commit 128b1a1
authored
Round-trip extension types without an Arrow plugin through Arrow (#11224)
## Summary
- Tracking Issue: #11114
Right now, an extension type without an Arrow plugin can't leave Vortex
through Arrow at all. That covers `FixedShapeTensor`, types from other
crates, and unknown types read from a file, so DataFusion can't even
infer the schema of a file that holds one, and the Python, JNI, and FFI
schema exports fail the same way. A plugin for every type doesn't scale,
and an unknown type can't have one. The import has the same gap, since a
field whose extension name is a Vortex ID imports as plain storage.
So a type without a plugin now exports its storage plus Arrow extension
metadata, with its Vortex ID in `ARROW:extension:name` and its metadata
in `ARROW:extension:metadata`. When the session knows the Vortex ID, the
import uses that metadata to rebuild the type, so the type survives a
round trip through Arrow, as X24 requires of every boundary.
## Changes
The metadata is base64, since Arrow field metadata must be UTF-8 and
Vortex metadata is arbitrary bytes.
An Arrow extension name that Vortex doesn't know still imports as plain
storage (X29), unlike the closed #10651, which imported it as a foreign
type. Under the opacity plan, a foreign type only supports structural
operations, so a pyarrow column with an unknown extension would lose
filters and comparisons, and a file that holds it would need
`allow_unknown` to read.
This has to land before U1, which makes a storage-like target an error.
A target whose extension name is the array's own Vortex ID still gets
the storage, as long as it has the same metadata. So a caller keeps a
lossless way to export, e.g., a timestamp column as `Int64`.
An export without a target field (`execute_arrow` with no target,
`Datum`, `ArrowSession::to_arrow_datatype`, and the Python array
exports) has no Arrow field to hold the extension metadata, so it fails
for a top-level column of these types instead of returning storage that
has lost its type. All of them build their field with
`ArrowSession::default_arrow_export_field`, so the check lives in one
place and adds no work to an export. A nested column works, because its
child field keeps the metadata. The types with a plugin still lose their
name at the top level, which #11286 tracks.
Note that until #11268, a table that mixes one of these columns with
plain storage labels the plain rows with the extension type, as it
already does for `Json`. Only the `ArrowSession` that
`vortex_arrow::initialize` registers knows the extension types of its
session, so a session created lazily imports these types as plain
storage. The tests in `vortex-arrow/tests/vortex_id.rs` pin the field
format and the round trip at the top level, in a struct, in a list, and
over a timestamp, and a DataFusion test copies a `FixedShapeTensor`
column through `COPY`.
## API Changes
- `ArrowSession::to_arrow_field` and `to_arrow_schema` export these
types instead of failing.
- The new `ArrowSession::default_arrow_export_field` returns the field
that an export without a target uses, and fails for a top-level type
without a plugin.
- When no importer accepts a field, but the session knows its Vortex ID,
the import rebuilds the type. It fails if the metadata or the storage
doesn't fit the type. For example, an `Int64` field named
`vortex.timestamp` now imports as a `Timestamp`, and a
`vortex.tensor.vector` field that the `Vector` importer declines now
fails instead of importing as plain storage.
---------
Signed-off-by: Connor Tsui <[email protected]>1 parent 7a67257 commit 128b1a1
11 files changed
Lines changed: 701 additions & 27 deletions
File tree
- vortex-arrow
- src
- executor
- tests
- vortex-datafusion/src/persistent
- vortex-python/src
- arrays
- dtype
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
119 | 119 | | |
120 | 120 | | |
121 | 121 | | |
| 122 | + | |
122 | 123 | | |
123 | 124 | | |
124 | 125 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
20 | 25 | | |
21 | 26 | | |
22 | 27 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
183 | 183 | | |
184 | 184 | | |
185 | 185 | | |
186 | | - | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
187 | 196 | | |
188 | 197 | | |
189 | 198 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
| 19 | + | |
18 | 20 | | |
| 21 | + | |
19 | 22 | | |
20 | 23 | | |
21 | 24 | | |
| |||
45 | 48 | | |
46 | 49 | | |
47 | 50 | | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
48 | 54 | | |
49 | | - | |
| 55 | + | |
50 | 56 | | |
0 commit comments