Require matching extension types in the DataFusion physical schema - #11268

)

## Summary

- Tracking Issue: #11114

Right now, an extension type without an Arrow plugin can't leave Vortex
through Arrow at all. That covers `FixedShapeTensor`, types from other
crates, and unknown types read from a file, so DataFusion can't even
infer the schema of a file that holds one, and the Python, JNI, and FFI
schema exports fail the same way. A plugin for every type doesn't scale,
and an unknown type can't have one. The import has the same gap, since a
field whose extension name is a Vortex ID imports as plain storage.

So a type without a plugin now exports its storage plus Arrow extension
metadata, with its Vortex ID in `ARROW:extension:name` and its metadata
in `ARROW:extension:metadata`. When the session knows the Vortex ID, the
import uses that metadata to rebuild the type, so the type survives a
round trip through Arrow, as X24 requires of every boundary.

## Changes

The metadata is base64, since Arrow field metadata must be UTF-8 and
Vortex metadata is arbitrary bytes.

An Arrow extension name that Vortex doesn't know still imports as plain
storage (X29), unlike the closed #10651, which imported it as a foreign
type. Under the opacity plan, a foreign type only supports structural
operations, so a pyarrow column with an unknown extension would lose
filters and comparisons, and a file that holds it would need
`allow_unknown` to read.

This has to land before U1, which makes a storage-like target an error.
A target whose extension name is the array's own Vortex ID still gets
the storage, as long as it has the same metadata. So a caller keeps a
lossless way to export, e.g., a timestamp column as `Int64`.

An export without a target field (`execute_arrow` with no target,
`Datum`, `ArrowSession::to_arrow_datatype`, and the Python array
exports) has no Arrow field to hold the extension metadata, so it fails
for a top-level column of these types instead of returning storage that
has lost its type. All of them build their field with
`ArrowSession::default_arrow_export_field`, so the check lives in one
place and adds no work to an export. A nested column works, because its
child field keeps the metadata. The types with a plugin still lose their
name at the top level, which #11286 tracks.

Note that until #11268, a table that mixes one of these columns with
plain storage labels the plain rows with the extension type, as it
already does for `Json`. Only the `ArrowSession` that
`vortex_arrow::initialize` registers knows the extension types of its
session, so a session created lazily imports these types as plain
storage. The tests in `vortex-arrow/tests/vortex_id.rs` pin the field
format and the round trip at the top level, in a struct, in a list, and
over a timestamp, and a DataFusion test copies a `FixedShapeTensor`
column through `COPY`.

## API Changes

- `ArrowSession::to_arrow_field` and `to_arrow_schema` export these
types instead of failing.
- The new `ArrowSession::default_arrow_export_field` returns the field
that an export without a target uses, and fails for a top-level type
without a plugin.
- When no importer accepts a field, but the session knows its Vortex ID,
the import rebuilds the type. It fails if the metadata or the storage
doesn't fit the type. For example, an `Int64` field named
`vortex.timestamp` now imports as a `Timestamp`, and a
`vortex.tensor.vector` field that the `Vector` importer declines now
fails instead of importing as plain storage.

---------

Signed-off-by: Connor Tsui <[email protected]>