Skip to content

Commit 3c109bf

Browse files
committed
Show the storage declaration in the vtables guide
The guide described obsolete extension APIs and migration plans. Explain the implemented typed and erased dispatch path, metadata reconstruction, nullability, and strict storage checks using the current APIs. Signed-off-by: Connor Tsui <[email protected]>
1 parent d907a0f commit 3c109bf

1 file changed

Lines changed: 71 additions & 23 deletions

File tree

‎docs/developer-guide/internals/vtables.md‎

Lines changed: 71 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ FooRef thin Arc wrapper, delegates to DynFoo
3131
**Data struct** `Foo<V>` is generic over the vtable type `V`. It holds the vtable instance
3232
(which may be zero-sized for native implementations or non-zero-sized for language bindings),
3333
common fields, and a VTable-specific associated type for concept-specific data (e.g.
34-
`V::Metadata` for ExtDType, `V::Array` for Array). `Foo<V>` is not Arc-wrapped — it lives
34+
`V::Metadata` for ExtDType, `V::Array` for Array). `Foo<V>` is not Arc-wrapped, it lives
3535
behind `Arc` inside `FooRef`. `Foo<V>` has inherent methods for all operations, with
3636
pre/post-condition enforcement. These delegate to `FooVTable` methods.
3737

@@ -41,7 +41,7 @@ the result of downcasting from `&FooRef`), while common methods on `Foo<V>` rema
4141
via normal method resolution.
4242

4343
**Sealed trait** `DynFoo` is object-safe and has a blanket `impl<V: FooVTable> DynFoo for Foo<V>`.
44-
This blanket impl is a thin forwarder to `Foo<V>` inherent methods — no logic of its own.
44+
This blanket impl is a thin forwarder to `Foo<V>` inherent methods, no logic of its own.
4545
Its purpose is to enable dynamic dispatch from `FooRef` through to the typed `Foo<V>`.
4646
`DynFoo` must stay private (`pub(super)`) because it exposes internal plumbing
4747
(`as_any`, `metadata_any`) that should not be part of the public API.
@@ -57,6 +57,9 @@ FooRef::as_::<V>(&self) -> &Foo<V> borrow, free after type check
5757
FooRef::downcast::<V>(self) -> Arc<Foo<V>> owned, free after type check
5858
```
5959

60+
Extension dtypes provide borrowed metadata access through `metadata::<V>()` and owned typed access
61+
through `downcast::<V>()`, as shown in the extension dtype example below.
62+
6063
**Plugin** `FooPlugin` is a separate trait for registry-based deserialization. It knows how to
6164
reconstruct a `FooRef` from serialized bytes without knowing `V` at compile time. Plugins are
6265
registered in the session by their ID.
@@ -86,16 +89,25 @@ For Array encodings, each encoding has its own module (e.g. `arrays/primitive/`)
8689
Extension dtypes follow this pattern:
8790

8891
```rust
89-
trait ExtDTypeVTable: Sized + Send + Sync + Clone + Debug {
92+
trait ExtVTable: Sized + Send + Sync + Clone + Debug {
9093
type Metadata: Send + Sync + Clone + Debug + Display + Eq + Hash;
94+
type StoredMetadata;
9195

9296
fn id(&self) -> ExtId;
93-
fn serialize(&self, metadata: &Self::Metadata) -> VortexResult<Vec<u8>>;
94-
fn deserialize(&self, metadata: &[u8]) -> VortexResult<Self::Metadata>;
95-
fn validate(&self, metadata: &Self::Metadata, storage_dtype: &DType) -> VortexResult<()>;
97+
fn serialize_metadata(&self, metadata: &Self::Metadata) -> VortexResult<Vec<u8>>;
98+
fn deserialize_metadata(&self, bytes: &[u8]) -> VortexResult<Self::StoredMetadata>;
99+
fn metadata_from_storage(
100+
&self,
101+
stored_metadata: Self::StoredMetadata,
102+
storage_dtype: &DType,
103+
) -> VortexResult<Self::Metadata>;
104+
fn storage_dtype(metadata: &Self::Metadata) -> DType;
105+
fn validate_metadata(_metadata: &Self::Metadata) -> VortexResult<()> {
106+
Ok(())
107+
}
96108
}
97109

98-
struct ExtDType<V: ExtDTypeVTable> {
110+
struct ExtDType<V: ExtVTable> {
99111
vtable: V,
100112
metadata: V::Metadata,
101113
storage_dtype: DType,
@@ -107,11 +119,47 @@ struct ExtDTypeRef(Arc<dyn DynExtDType>);
107119
Downcasting from the erased form recovers the typed data struct:
108120

109121
```rust
110-
let ext: & ExtDTypeRef = dtype.ext();
111-
let ts: & ExtDType<Timestamp> = ext.as_::<Timestamp>();
112-
let unit: & TimeUnit = & ts.metadata; // V::Metadata is concrete
122+
let ext: &ExtDTypeRef = dtype.as_extension();
123+
let metadata = ext.metadata::<Timestamp>();
124+
let unit: &TimeUnit = &metadata.unit; // V::Metadata is concrete.
125+
let ts: Arc<ExtDType<Timestamp>> = ext.clone().downcast::<Timestamp>();
113126
```
114127

128+
## Storage and nullability
129+
130+
`ExtDType::try_new(metadata, nullability)` uses a default vtable. `try_with_vtable` accepts an
131+
explicit vtable instance. Both validate metadata before deriving storage through
132+
`ExtVTable::storage_dtype`.
133+
134+
The in-memory metadata determines the storage structure, element types, and nested nullability.
135+
`ExtDType` applies the requested top-level nullability. For example, a vector's metadata holds its
136+
float element type and dimension, and its storage is a fixed-size list of non-nullable elements.
137+
The outer list can be nullable independently of its elements.
138+
139+
`with_nullability` keeps the metadata and rebuilds storage from the declaration with the requested
140+
outer nullability. It does not validate metadata again. Storage of `DType::Null`, including an
141+
extension over it, is always nullable. `with_nullability` panics if asked to make that storage
142+
non-nullable, while fallible constructors return an error.
143+
144+
`StoredMetadata` is the in-memory form of the metadata stored alongside the storage dtype.
145+
Callers can also construct it directly. Complete `Metadata` adds facts inferred from storage.
146+
For a vector, stored metadata is `()`, and storage supplies the element type and dimension.
147+
Stored tensor metadata supplies shape, dimension names, and permutation, while storage supplies
148+
the element type. A timestamp uses the same type for both, because storage cannot determine its
149+
unit or time zone.
150+
151+
`ExtDType::try_from_storage(stored_metadata, storage_dtype)` reconstructs metadata through
152+
`metadata_from_storage`. It passes a nullable outer storage level, validates the metadata, and
153+
requires its declaration to match the supplied storage exactly, including nested nullability.
154+
The resulting dtype keeps the supplied outer nullability. `try_from_storage_with_vtable` accepts
155+
an explicit vtable. `ExtensionArray::try_from_storage` and
156+
`Scalar::try_extension_from_storage` use the actual storage array or scalar's dtype. The scalar
157+
constructor also validates the value.
158+
159+
On read, `deserialize_metadata` decodes bytes into `StoredMetadata`. The dtype plugin then uses the
160+
same checked storage constructor. Serialized bytes can omit facts represented by storage, and a
161+
storage mismatch returns an error naming both dtypes.
162+
115163
## Why Not `dyn Trait`
116164

117165
Rust's `dyn Trait` vtables have several limitations that make them unsuitable for Vortex's
@@ -147,7 +195,7 @@ on the same type triggers `clippy::same_name_method`. The correct handling is:
147195
delegation to `FooVTable` all happen here.
148196
2. **`DynFoo` blanket impl is a thin forwarder.** Each method body is just `self.method()`.
149197
Rust's method resolution picks inherent methods over trait methods, so this calls the
150-
inherent impl — no infinite recursion.
198+
inherent impl, no infinite recursion.
151199
3. **`#[allow(clippy::same_name_method)]`** on the `Foo<V>` inherent impl block
152200
acknowledges the intentional shadowing.
153201

@@ -169,7 +217,7 @@ impl<V: FooVTable> DynFoo for Foo<V> {
169217
`DynFoo` must stay **private** (`pub(super)`) because it exposes internal plumbing
170218
(`as_any`, `metadata_any`) that external callers should never reach. Making it public
171219
or implementing it for `FooRef` would leak these internals. Instead, `FooRef` has its
172-
own inherent methods that delegate to `DynFoo` — providing a clean public API without
220+
own inherent methods that delegate to `DynFoo`, providing a clean public API without
173221
exposing the dispatch machinery.
174222

175223
Methods that exist only for erased dispatch (e.g. `as_any`, `metadata_any`,
@@ -189,19 +237,19 @@ consistent across all vtable-backed types in Vortex.
189237

190238
## Migration Status
191239

192-
All four vtable-backed types are converging on the pattern described above.
240+
All four vtable-backed types are converging on the pattern described above. The extension dtype
241+
status reflects the API above. The remaining subsections retain the broader migration plan.
193242

194-
### ExtDType -- Partially done
243+
### ExtDType -- Done
195244

196-
File split already matches conventions: `vtable.rs`, `typed.rs`, `erased.rs`,
197-
`plugin.rs`, `matcher.rs`. Remaining:
245+
The file split matches the conventions: `vtable.rs`, `typed.rs`, `erased.rs`, `plugin.rs`, and
246+
`matcher.rs`.
198247

199-
- **Drop internal Arc from `ExtDType<V>`.** Currently wraps
200-
`Arc<ExtDTypeAdapter<V>>`. Should become the data struct itself, with
201-
`ExtDTypeRef(Arc<dyn DynExtDType>)` holding the Arc. Remove `ExtDTypeAdapter`.
202-
- **Rename** `ExtDTypeImpl` → `DynExtDType`, `DynExtVTable` → `ExtDTypePlugin`.
203-
- **Remove `ExtDTypeMetadata`** erased wrapper. Its methods (`serialize`, `Display`,
204-
`Debug`, `PartialEq`, `Hash`) should move to `ExtDTypeRef`.
248+
- `ExtDType<V>` is the data struct. `ExtDTypeRef(Arc<dyn DynExtDType>)` owns the shared allocation.
249+
- `ExtVTable` is the public implementation trait, `DynExtDType` provides private erased dispatch,
250+
and `ExtDTypePlugin` provides registry-based deserialization.
251+
- Metadata stays typed inside `ExtDType<V>`. `ExtDTypeRef` exposes serialization, `Display`,
252+
`Debug`, equality, and hashing through the private dispatch layer.
205253

206254
### Expr -- Not started
207255

@@ -254,7 +302,7 @@ with pre/post-condition enforcement. These delegate to `ArrayVTable` methods.
254302
the associated type are callable directly on `&Array<V>`. Common methods
255303
(`dtype()`, `len()`) are inherent on `Array<V>` and resolve first.
256304

257-
Children live in `V::Array` — each encoding owns its child representation.
305+
Children live in `V::Array`, each encoding owns its child representation.
258306
The VTable provides `nchildren()` / `child(i)` for generic traversal.
259307

260308
Constructors live on the vtable ZST: `Primitive::new(...) -> ArrayRef`.

0 commit comments

Comments
 (0)