Skip to content

Commit 128b1a1

Browse files
authored
Round-trip extension types without an Arrow plugin through Arrow (#11224)
## Summary - Tracking Issue: #11114 Right now, an extension type without an Arrow plugin can't leave Vortex through Arrow at all. That covers `FixedShapeTensor`, types from other crates, and unknown types read from a file, so DataFusion can't even infer the schema of a file that holds one, and the Python, JNI, and FFI schema exports fail the same way. A plugin for every type doesn't scale, and an unknown type can't have one. The import has the same gap, since a field whose extension name is a Vortex ID imports as plain storage. So a type without a plugin now exports its storage plus Arrow extension metadata, with its Vortex ID in `ARROW:extension:name` and its metadata in `ARROW:extension:metadata`. When the session knows the Vortex ID, the import uses that metadata to rebuild the type, so the type survives a round trip through Arrow, as X24 requires of every boundary. ## Changes The metadata is base64, since Arrow field metadata must be UTF-8 and Vortex metadata is arbitrary bytes. An Arrow extension name that Vortex doesn't know still imports as plain storage (X29), unlike the closed #10651, which imported it as a foreign type. Under the opacity plan, a foreign type only supports structural operations, so a pyarrow column with an unknown extension would lose filters and comparisons, and a file that holds it would need `allow_unknown` to read. This has to land before U1, which makes a storage-like target an error. A target whose extension name is the array's own Vortex ID still gets the storage, as long as it has the same metadata. So a caller keeps a lossless way to export, e.g., a timestamp column as `Int64`. An export without a target field (`execute_arrow` with no target, `Datum`, `ArrowSession::to_arrow_datatype`, and the Python array exports) has no Arrow field to hold the extension metadata, so it fails for a top-level column of these types instead of returning storage that has lost its type. All of them build their field with `ArrowSession::default_arrow_export_field`, so the check lives in one place and adds no work to an export. A nested column works, because its child field keeps the metadata. The types with a plugin still lose their name at the top level, which #11286 tracks. Note that until #11268, a table that mixes one of these columns with plain storage labels the plain rows with the extension type, as it already does for `Json`. Only the `ArrowSession` that `vortex_arrow::initialize` registers knows the extension types of its session, so a session created lazily imports these types as plain storage. The tests in `vortex-arrow/tests/vortex_id.rs` pin the field format and the round trip at the top level, in a struct, in a list, and over a timestamp, and a DataFusion test copies a `FixedShapeTensor` column through `COPY`. ## API Changes - `ArrowSession::to_arrow_field` and `to_arrow_schema` export these types instead of failing. - The new `ArrowSession::default_arrow_export_field` returns the field that an export without a target uses, and fails for a top-level type without a plugin. - When no importer accepts a field, but the session knows its Vortex ID, the import rebuilds the type. It fails if the metadata or the storage doesn't fit the type. For example, an `Int64` field named `vortex.timestamp` now imports as a `Timestamp`, and a `vortex.tensor.vector` field that the `Vector` importer declines now fails instead of importing as plain storage. --------- Signed-off-by: Connor Tsui <[email protected]>
1 parent 7a67257 commit 128b1a1

11 files changed

Lines changed: 701 additions & 27 deletions

File tree

‎Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,7 @@ async-lock = "3.4"
119119
async-stream = "0.3.6"
120120
async-trait = "0.1.89"
121121
base16ct = "1.0.0"
122+
base64 = "0.23.1"
122123
bigdecimal = "0.4.8"
123124
bindgen = "0.72.0"
124125
bit-vec = "0.10.1"

‎vortex-arrow/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ arrow-buffer = { workspace = true }
2424
arrow-cast = { workspace = true }
2525
arrow-data = { workspace = true }
2626
arrow-schema = { workspace = true, features = ["canonical_extension_types"] }
27+
base64 = { workspace = true }
2728
itertools = { workspace = true }
2829
num-traits = { workspace = true }
2930
simdutf8 = { workspace = true }

‎vortex-arrow/src/datum.rs‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,11 @@ use vortex_error::vortex_panic;
1717
use crate::ArrowSessionExt;
1818

1919
/// A wrapper around a generic Arrow array that can be used as a Datum in Arrow compute.
20+
///
21+
/// A `Datum` has no Arrow field to hold extension metadata, so it fails for the same types that
22+
/// [`ArrowSession::default_arrow_export_field`] rejects.
23+
///
24+
/// [`ArrowSession::default_arrow_export_field`]: crate::ArrowSession::default_arrow_export_field
2025
#[derive(Debug)]
2126
pub struct Datum {
2227
array: ArrowArrayRef,

‎vortex-arrow/src/executor/struct_.rs‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -183,7 +183,16 @@ fn create_from_fields(
183183
let mut arrow_fields = Vec::with_capacity(vortex_fields.len());
184184
for (name, vx_field) in names.iter().zip_eq(vortex_fields.iter()) {
185185
let inferred = infer_nearest_arrow_field(vx_field, name.as_ref(), ctx)?;
186-
let arrow_array = exporter.execute_arrow(vx_field.clone(), None, ctx)?;
186+
187+
// A child with an Arrow extension name exports to its inferred field. With `None`,
188+
// the export would fail for a type without an Arrow plugin, because an export
189+
// without a target has no field to hold its extension metadata. Other children keep
190+
// `None`, so that their Arrow type still follows their encoding.
191+
let target = inferred
192+
.extension_type_name()
193+
.is_some()
194+
.then_some(&inferred);
195+
let arrow_array = exporter.execute_arrow(vx_field.clone(), target, ctx)?;
187196
// The executed array is authoritative for the physical type; only the metadata is
188197
// taken from the inferred field.
189198
arrow_fields.push(Arc::new(

‎vortex-arrow/src/lib.rs‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,10 @@
1515
//!
1616
//! The [`ArrowSession`] session variable is created lazily on first use of
1717
//! [`ArrowSessionExt::arrow`], so no explicit registration is required for plain conversions.
18+
//! To import extension types that have no Arrow plugin, register the session with [`initialize`]
19+
//! (see [`ArrowSession`]).
1820
21+
use vortex_array::dtype::session::DTypeSessionExt;
1922
use vortex_session::VortexSession;
2023

2124
pub mod convert;
@@ -45,6 +48,9 @@ pub use session::*;
4548
/// Register vortex-arrow's session extensions into `session`.
4649
///
4750
/// This eagerly registers the [`ArrowSession`], which is otherwise created lazily on first use.
51+
/// Only a session registered here can look up the extension types of `session` by their Vortex IDs.
52+
/// A session that is created lazily on first use cannot, so it imports an extension type without an
53+
/// Arrow plugin as plain storage.
4854
pub fn initialize(session: &VortexSession) {
49-
session.register(ArrowSession::default());
55+
session.register(ArrowSession::new(&session.dtypes()));
5056
}

0 commit comments

Comments
 (0)