Work with Arrow fields instead of Arrow data types #11286

Problem or motivation

A Vortex DType corresponds to an Arrow Field, not to an Arrow DataType, but vortex-arrow and the bindings convert through both. Arrow keeps the top-level nullability and the extension type (ARROW:extension:name and ARROW:extension:metadata) on the Field, and arrow-rs follows this exactly: a DataType has no extension variant, and an array has no field. So every API that only takes or returns a DataType (or an array without its field) silently loses the top-level extension type, in both directions:

// Export: `to_arrow_field` gives `FixedSizeBinary(16)` named `arrow.uuid`, but:
session.arrow().to_arrow_datatype(&uuid_dtype)?;            // FixedSizeBinary(16)
uuid_array.execute_arrow(None, &mut ctx)?;                   // builds the field internally, then drops it
Datum::try_new(&json_array, &mut ctx)?;                      // a string array, so Arrow `eq` compares JSON as text

// Import: nothing here can name an extension type, so `arrow.uuid` data imports as plain storage.
session.arrow().from_arrow_datatype(&DataType::FixedSizeBinary(16), Nullable)?;
session.arrow().from_arrow_array(uuid_arrow_array, true)?;

The Python bindings do the same, since DType.to_arrow_type(), to_arrow_array(), DType.from_arrow(), and the arrow_type argument of to_arrow_array() all convert through an arrow-rs DataType. So a Uuid column arrives in pyarrow as fixed_size_binary(16) instead of arrow.uuid, even though pyarrow itself can represent extension types.

This breaks X24 (every boundary keeps the extension type or fails), and #11224 widens it to every extension type, because the types without an Arrow plugin can now export too. Nested types aren't affected, since the child fields of a List or Struct data type keep their metadata.

Proposed solution

Ideally, Vortex only talks to Arrow in fields (and schemas), so every API that only takes or returns a DataType is deprecated in favor of a field version:

1. ArrowSession::to_arrow_datatype and from_arrow_datatype are deprecated in favor of to_arrow_field and from_arrow_field.
2. An export without a target returns the field that describes its result, e.g., (Field, ArrowArrayRef), instead of array.execute_arrow(None, ctx). The struct exporter already does this by hand for each child, and it builds each child field twice.
3. Datum::try_new_with_target_datatype and the nullability form of from_arrow_array take a Field instead. The export already does, since Export to Arrow with the session of the execution context #11304 made the executor that takes a DataType private.
4. Datum rejects every non-temporal extension dtype, as the scalar ToArrowDatum already does, because it feeds arrow-rs compute kernels that only take arrays. A caller that wants the storage calls ext_storage first. Nothing in the repository has called Datum since Implement compare in vortex instead of falling back to arrow #8661, Remove arrow usage from VarBin::compare, VarBinView::filter and BoolArray::from_iter #8695, and Implement LIKE in vortex instead of falling back to arrow #8709, so deleting it is also an option.
5. The Python bindings exchange fields with pyarrow, which rebuilds the extension types it knows, such as arrow.uuid and arrow.json. pyarrow turns a name it doesn't know (e.g., a Vortex ID) into plain storage, so an export fails in that case until BD1 in Tracking Issue: Fix Extension Types #11114 registers the Vortex types with pyarrow.

Additional context

DataFusion made the same move: it deprecated TypePlanner::plan_type (53) for plan_type_field and data_type_and_nullable (51) for to_field, Expr::Cast holds a target FieldRef, and functions receive arg_fields and return_field next to their arrays.

The DataFusion scans, FFI, and JNI already build their schema once and export every batch into it, so they don't change. Steps 4 and 5 do change behavior for the types with a plugin, since Uuid, Json, Vector, and the spatial types stop leaving through Datum and the Python exports as plain storage.

This is a temporary issue, and a tracking issue will replace it.