@@ -31,7 +31,7 @@ FooRef thin Arc wrapper, delegates to DynFoo
3131** Data struct** ` Foo<V> ` is generic over the vtable type ` V ` . It holds the vtable instance
3232(which may be zero-sized for native implementations or non-zero-sized for language bindings),
3333common fields, and a VTable-specific associated type for concept-specific data (e.g.
34- ` V::Metadata ` for ExtDType, ` V::Array ` for Array). ` Foo<V> ` is not Arc-wrapped — it lives
34+ ` V::Metadata ` for ExtDType, ` V::Array ` for Array). ` Foo<V> ` is not Arc-wrapped, it lives
3535behind ` Arc ` inside ` FooRef ` . ` Foo<V> ` has inherent methods for all operations, with
3636pre/post-condition enforcement. These delegate to ` FooVTable ` methods.
3737
@@ -41,7 +41,7 @@ the result of downcasting from `&FooRef`), while common methods on `Foo<V>` rema
4141via normal method resolution.
4242
4343** Sealed trait** ` DynFoo ` is object-safe and has a blanket ` impl<V: FooVTable> DynFoo for Foo<V> ` .
44- This blanket impl is a thin forwarder to ` Foo<V> ` inherent methods — no logic of its own.
44+ This blanket impl is a thin forwarder to ` Foo<V> ` inherent methods, no logic of its own.
4545Its purpose is to enable dynamic dispatch from ` FooRef ` through to the typed ` Foo<V> ` .
4646` DynFoo ` must stay private (` pub(super) ` ) because it exposes internal plumbing
4747(` as_any ` , ` metadata_any ` ) that should not be part of the public API.
@@ -57,6 +57,9 @@ FooRef::as_::<V>(&self) -> &Foo<V> borrow, free after type check
5757FooRef::downcast::<V>(self) -> Arc<Foo<V>> owned, free after type check
5858```
5959
60+ Extension dtypes provide borrowed metadata access through ` metadata::<V>() ` and owned typed access
61+ through ` downcast::<V>() ` , as shown in the extension dtype example below.
62+
6063** Plugin** ` FooPlugin ` is a separate trait for registry-based deserialization. It knows how to
6164reconstruct a ` FooRef ` from serialized bytes without knowing ` V ` at compile time. Plugins are
6265registered in the session by their ID.
@@ -86,16 +89,25 @@ For Array encodings, each encoding has its own module (e.g. `arrays/primitive/`)
8689Extension dtypes follow this pattern:
8790
8891``` rust
89- trait ExtDTypeVTable : Sized + Send + Sync + Clone + Debug {
92+ trait ExtVTable : Sized + Send + Sync + Clone + Debug {
9093 type Metadata : Send + Sync + Clone + Debug + Display + Eq + Hash ;
94+ type StoredMetadata ;
9195
9296 fn id (& self ) -> ExtId ;
93- fn serialize (& self , metadata : & Self :: Metadata ) -> VortexResult <Vec <u8 >>;
94- fn deserialize (& self , metadata : & [u8 ]) -> VortexResult <Self :: Metadata >;
95- fn validate (& self , metadata : & Self :: Metadata , storage_dtype : & DType ) -> VortexResult <()>;
97+ fn serialize_metadata (& self , metadata : & Self :: Metadata ) -> VortexResult <Vec <u8 >>;
98+ fn deserialize_metadata (& self , bytes : & [u8 ]) -> VortexResult <Self :: StoredMetadata >;
99+ fn metadata_from_storage (
100+ & self ,
101+ stored_metadata : Self :: StoredMetadata ,
102+ storage_dtype : & DType ,
103+ ) -> VortexResult <Self :: Metadata >;
104+ fn storage_dtype (metadata : & Self :: Metadata ) -> DType ;
105+ fn validate_metadata (_metadata : & Self :: Metadata ) -> VortexResult <()> {
106+ Ok (())
107+ }
96108}
97109
98- struct ExtDType <V : ExtDTypeVTable > {
110+ struct ExtDType <V : ExtVTable > {
99111 vtable : V ,
100112 metadata : V :: Metadata ,
101113 storage_dtype : DType ,
@@ -107,11 +119,47 @@ struct ExtDTypeRef(Arc<dyn DynExtDType>);
107119Downcasting from the erased form recovers the typed data struct:
108120
109121``` rust
110- let ext : & ExtDTypeRef = dtype . ext ();
111- let ts : & ExtDType <Timestamp > = ext . as_ :: <Timestamp >();
112- let unit : & TimeUnit = & ts . metadata; // V::Metadata is concrete
122+ let ext : & ExtDTypeRef = dtype . as_extension ();
123+ let metadata = ext . metadata :: <Timestamp >();
124+ let unit : & TimeUnit = & metadata . unit; // V::Metadata is concrete.
125+ let ts : Arc <ExtDType <Timestamp >> = ext . clone (). downcast :: <Timestamp >();
113126```
114127
128+ ## Storage and nullability
129+
130+ ` ExtDType::try_new(metadata, nullability) ` uses a default vtable. ` try_with_vtable ` accepts an
131+ explicit vtable instance. Both validate metadata before deriving storage through
132+ ` ExtVTable::storage_dtype ` .
133+
134+ The in-memory metadata determines the storage structure, element types, and nested nullability.
135+ ` ExtDType ` applies the requested top-level nullability. For example, a vector's metadata holds its
136+ float element type and dimension, and its storage is a fixed-size list of non-nullable elements.
137+ The outer list can be nullable independently of its elements.
138+
139+ ` with_nullability ` keeps the metadata and rebuilds storage from the declaration with the requested
140+ outer nullability. It does not validate metadata again. Storage of ` DType::Null ` , including an
141+ extension over it, is always nullable. ` with_nullability ` panics if asked to make that storage
142+ non-nullable, while fallible constructors return an error.
143+
144+ ` StoredMetadata ` is the in-memory form of the metadata stored alongside the storage dtype.
145+ Callers can also construct it directly. Complete ` Metadata ` adds facts inferred from storage.
146+ For a vector, stored metadata is ` () ` , and storage supplies the element type and dimension.
147+ Stored tensor metadata supplies shape, dimension names, and permutation, while storage supplies
148+ the element type. A timestamp uses the same type for both, because storage cannot determine its
149+ unit or time zone.
150+
151+ ` ExtDType::try_from_storage(stored_metadata, storage_dtype) ` reconstructs metadata through
152+ ` metadata_from_storage ` . It passes a nullable outer storage level, validates the metadata, and
153+ requires its declaration to match the supplied storage exactly, including nested nullability.
154+ The resulting dtype keeps the supplied outer nullability. ` try_from_storage_with_vtable ` accepts
155+ an explicit vtable. ` ExtensionArray::try_from_storage ` and
156+ ` Scalar::try_extension_from_storage ` use the actual storage array or scalar's dtype. The scalar
157+ constructor also validates the value.
158+
159+ On read, ` deserialize_metadata ` decodes bytes into ` StoredMetadata ` . The dtype plugin then uses the
160+ same checked storage constructor. Serialized bytes can omit facts represented by storage, and a
161+ storage mismatch returns an error naming both dtypes.
162+
115163## Why Not ` dyn Trait `
116164
117165Rust's ` dyn Trait ` vtables have several limitations that make them unsuitable for Vortex's
@@ -147,7 +195,7 @@ on the same type triggers `clippy::same_name_method`. The correct handling is:
147195 delegation to ` FooVTable ` all happen here.
1481962 . ** ` DynFoo ` blanket impl is a thin forwarder.** Each method body is just ` self.method() ` .
149197 Rust's method resolution picks inherent methods over trait methods, so this calls the
150- inherent impl — no infinite recursion.
198+ inherent impl, no infinite recursion.
1511993 . ** ` #[allow(clippy::same_name_method)] ` ** on the ` Foo<V> ` inherent impl block
152200 acknowledges the intentional shadowing.
153201
@@ -169,7 +217,7 @@ impl<V: FooVTable> DynFoo for Foo<V> {
169217` DynFoo ` must stay ** private** (` pub(super) ` ) because it exposes internal plumbing
170218(` as_any ` , ` metadata_any ` ) that external callers should never reach. Making it public
171219or implementing it for ` FooRef ` would leak these internals. Instead, ` FooRef ` has its
172- own inherent methods that delegate to ` DynFoo ` — providing a clean public API without
220+ own inherent methods that delegate to ` DynFoo ` , providing a clean public API without
173221exposing the dispatch machinery.
174222
175223Methods that exist only for erased dispatch (e.g. ` as_any ` , ` metadata_any ` ,
@@ -189,19 +237,19 @@ consistent across all vtable-backed types in Vortex.
189237
190238## Migration Status
191239
192- All four vtable-backed types are converging on the pattern described above.
240+ All four vtable-backed types are converging on the pattern described above. The extension dtype
241+ status reflects the API above. The remaining subsections retain the broader migration plan.
193242
194- ### ExtDType -- Partially done
243+ ### ExtDType -- Done
195244
196- File split already matches conventions: ` vtable.rs ` , ` typed.rs ` , ` erased.rs ` ,
197- ` plugin.rs ` , ` matcher.rs ` . Remaining:
245+ The file split matches the conventions: ` vtable.rs ` , ` typed.rs ` , ` erased.rs ` , ` plugin.rs ` , and
246+ ` matcher.rs ` .
198247
199- - ** Drop internal Arc from ` ExtDType<V> ` .** Currently wraps
200- ` Arc<ExtDTypeAdapter<V>> ` . Should become the data struct itself, with
201- ` ExtDTypeRef(Arc<dyn DynExtDType>) ` holding the Arc. Remove ` ExtDTypeAdapter ` .
202- - ** Rename** ` ExtDTypeImpl ` → ` DynExtDType ` , ` DynExtVTable ` → ` ExtDTypePlugin ` .
203- - ** Remove ` ExtDTypeMetadata ` ** erased wrapper. Its methods (` serialize ` , ` Display ` ,
204- ` Debug ` , ` PartialEq ` , ` Hash ` ) should move to ` ExtDTypeRef ` .
248+ - ` ExtDType<V> ` is the data struct. ` ExtDTypeRef(Arc<dyn DynExtDType>) ` owns the shared allocation.
249+ - ` ExtVTable ` is the public implementation trait, ` DynExtDType ` provides private erased dispatch,
250+ and ` ExtDTypePlugin ` provides registry-based deserialization.
251+ - Metadata stays typed inside ` ExtDType<V> ` . ` ExtDTypeRef ` exposes serialization, ` Display ` ,
252+ ` Debug ` , equality, and hashing through the private dispatch layer.
205253
206254### Expr -- Not started
207255
@@ -254,7 +302,7 @@ with pre/post-condition enforcement. These delegate to `ArrayVTable` methods.
254302the associated type are callable directly on ` &Array<V> ` . Common methods
255303(` dtype() ` , ` len() ` ) are inherent on ` Array<V> ` and resolve first.
256304
257- Children live in ` V::Array ` — each encoding owns its child representation.
305+ Children live in ` V::Array ` , each encoding owns its child representation.
258306The VTable provides ` nchildren() ` / ` child(i) ` for generic traversal.
259307
260308Constructors live on the vtable ZST: ` Primitive::new(...) -> ArrayRef ` .
0 commit comments