Skip to content

Commit 47daf16

Browse files
authored
Move array serde from VTable to explicit ArrayPlugin impls (#11285)
## Summary Removes the blanket `impl<V: VTable> ArrayPlugin for V` and the `serialize` / `deserialize` methods on `VTable`. Every serializable encoding now implements `ArrayPlugin` explicitly on its vtable type, in a `plugin.rs` child module next to its `impl VTable`. Redo of #10111 on current develop. ## Changes - `VTable` covers only in-memory behaviour; serde is `impl ArrayPlugin for X`. - Serde bodies are ported verbatim, so wire formats are unchanged. - `read_ids` / `write_ids` default to `vec![self.id()]`, what the blanket impl declared. Plugins with one wire ID no longer spell them out, including `ALPPatchedPlugin`, `ScalarFnArrayPlugin` and the Python stub. - Inside an impl the wire ID is `ArrayPlugin::id(self)`; only the `id` method itself delegates to `VTable::id`. - `session.arrays().register(X)` is unchanged for every encoding that was serializable. - In-memory-only vtables lose the two stub methods and get no plugin: Filter, Slice, Interleave, ScalarFn, ScalarFnValidity, ForeignArray, PiecewiseSequence, Widen, Shared, test-harness vtables. - `DecimalPlugin` (#10995) becomes `impl ArrayPlugin for Decimal`: one wire ID, same legacy-node reads, no separate type. Its module, `StoredDecimal` and tests stay where they are. - Multi-ID plugins (`BitPackedPlugin`, `FoRPlugin`, `DecimalBytePartsPlugin`) and `WidenPlugin` are untouched; `ALPPatchedPlugin` now delegates to `ArrayPlugin::deserialize(&ALP, ..)`. - `PythonVTable` keeps a stub plugin so the Python `register` API still compiles. - Vtable files that gained a `plugin.rs` child are directory modules (`array.rs` becomes `array/mod.rs`, 14 pure renames). ALP's own impl and `ALPPatchedPlugin` share `alp/array/plugin.rs`. - Tests that called the vtable serde directly (ALP, ALPRD, Dict, List, RunEnd, VarBin, RLE, serde.rs, unary, zstd buffers) go through `ArrayPlugin` instead. | | count | |---|---| | new `impl ArrayPlugin` (vortex-array) | 20 | | new `impl ArrayPlugin` (encodings + python) | 17 | | vtables that only lose the stub methods | 14 | ## API changes - `VTable` loses `serialize` and `deserialize`. An external vtable that was registered through the blanket impl must now `impl ArrayPlugin` to be registrable. - `ArrayPlugin::read_ids` / `write_ids` gain defaults of `vec![self.id()]`. - `vortex_array::arrays::DecimalPlugin` is removed; register `Decimal` instead. ## Behaviour changes - `PiecewiseSequence` is no longer registered by `ArraySession::default()`; it never serialized, and its error is now "is not registered for serialization". - The writer seeds the footer's encoding list from the session registry, so dropping `vortex.piecewise-sequence` makes every default-session file 48 bytes smaller. `test_stock_ticker_file_size` expectations move accordingly (70060 → 70012, 69996 → 69948, 55144 → 55096). - `DecimalByteParts`, `BitPacked` and `FoR` can no longer be registered bare, so `bare_vtable_refuses_serde` and `vtable_serde_requires_plugin` are removed. ## Testing - `cargo clippy --all-targets --all-features -- -D warnings` and `cargo +nightly-2026-09-10 fmt --check` on every touched crate (vortex-array, vortex-file, vortex-layout, vortex-python and the 13 encodings crates). - `cargo nextest run` on all touched crates plus vortex-ipc and vortex-scan: 8981 passed, 0 failed; `cargo test --doc` on vortex-array, fastlanes, decimal-byte-parts and alp. - `cargo nextest run -p vortex-file -E 'test(test_stock_ticker_file_size)'` with and without `--features zstd`, to derive the new sizes. ## Follow-up - Pre-existing, unchanged here: the Patched, RLE, OnPair, ALPRD, FSST and Zstd deserializers index children or buffers before checking their counts, so a malformed node panics in `SerializedArray::child` instead of returning an error. Worth hoisting the count checks in a separate PR. - Fold `FoRPlugin` / `BitPackedPlugin` / `DecimalBytePartsPlugin` into `impl ArrayPlugin for FoR` etc. if wanted. - Overlaps with #11298, which makes the same change with inline impls and trait defaults for `read_ids` / `write_ids`. --------- Signed-off-by: Matt Katz <[email protected]>
1 parent 7854a19 commit 47daf16

107 files changed

Lines changed: 4372 additions & 3095 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/developer-guide/internals/vtables.md‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,9 @@ needs:
121121
But operations like deserialization need to construct a new instance from raw bytes -- there
122122
is no instance yet. Vortex vtables colocate instance logic (e.g. `serialize`) with type-level
123123
logic (e.g. `deserialize`) in the same struct, since the vtable itself is a lightweight value
124-
(often zero-sized) that can be default-constructed.
124+
(often zero-sized) that can be default-constructed. Array vtables are the exception: their serde
125+
lives on a separate `ArrayPlugin` implementation, usually on the vtable type itself, so a vtable
126+
without one is in-memory only.
125127

126128
**No associated type access.** A `dyn Trait` erases all associated types. With Vortex vtables,
127129
the typed form `Foo<V>` preserves full access to `V::Metadata` and other associated types,
@@ -176,10 +178,11 @@ Methods that exist only for erased dispatch (e.g. `as_any`, `metadata_any`,
176178

177179
## Registration and Deserialization
178180

179-
Vtables are registered in the session by their ID. When a serialized value is encountered
180-
(e.g. an extension dtype in a file footer), the session's registry resolves the ID to the
181-
`FooPlugin` and calls `deserialize` to reconstruct the value. The result is returned as the
182-
erased form (`FooRef`) so it can be stored without generic parameters.
181+
Vtables are registered in the session by their ID. An array vtable registers through its
182+
`ArrayPlugin` implementation, and a vtable without one cannot be registered. When a serialized
183+
value is encountered (e.g. an extension dtype in a file footer), the session's registry resolves
184+
the ID to the `FooPlugin` and calls `deserialize` to reconstruct the value. The result is returned
185+
as the erased form (`FooRef`) so it can be stored without generic parameters.
183186

184187
This pattern -- register plugin by ID, deserialize via plugin, store as erased ref -- is
185188
consistent across all vtable-backed types in Vortex.
Lines changed: 3 additions & 77 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,6 @@ use std::fmt::Formatter;
77
use std::hash::Hash;
88
use std::hash::Hasher;
99

10-
use prost::Message;
1110
use vortex_array::Array;
1211
use vortex_array::ArrayEq;
1312
use vortex_array::ArrayHash;
@@ -21,7 +20,6 @@ use vortex_array::ExecutionCtx;
2120
use vortex_array::ExecutionResult;
2221
use vortex_array::IntoArray;
2322
use vortex_array::TypedArrayRef;
24-
use vortex_array::VortexSessionExecute;
2523
use vortex_array::array_slots;
2624
use vortex_array::arrays::Primitive;
2725
use vortex_array::buffer::BufferHandle;
@@ -34,7 +32,6 @@ use vortex_array::patches::PatchesData;
3432
use vortex_array::patches::PatchesMetadata;
3533
use vortex_array::require_child;
3634
use vortex_array::require_patches;
37-
use vortex_array::serde::ArrayChildren;
3835
use vortex_array::smallvec::smallvec;
3936
use vortex_array::vtable::VTable;
4037
use vortex_array::vtable::ValidityChild;
@@ -45,14 +42,16 @@ use vortex_error::vortex_bail;
4542
use vortex_error::vortex_ensure;
4643
use vortex_error::vortex_ensure_eq;
4744
use vortex_error::vortex_panic;
48-
use vortex_session::VortexSession;
4945
use vortex_session::registry::CachedId;
5046

5147
use crate::ALPFloat;
5248
use crate::alp::Exponents;
5349
use crate::alp::decompress::execute_decompress;
5450
use crate::alp::rules::RULES;
5551

52+
mod plugin;
53+
pub(crate) use plugin::ALPPatchedPlugin;
54+
5655
/// A [`ALP`]-encoded Vortex array.
5756
pub type ALPArray = Array<ALP>;
5857

@@ -113,79 +112,6 @@ impl VTable for ALP {
113112
vortex_array::vtable::with_empty_buffers(self, array, buffers)
114113
}
115114

116-
fn serialize(
117-
array: ArrayView<'_, Self>,
118-
_session: &VortexSession,
119-
) -> VortexResult<Option<Vec<u8>>> {
120-
let exponents = array.exponents();
121-
Ok(Some(
122-
ALPMetadata {
123-
exp_e: exponents.e as u32,
124-
exp_f: exponents.f as u32,
125-
patches: array
126-
.patches()
127-
.map(|p| p.to_metadata(array.len(), array.dtype()))
128-
.transpose()?,
129-
}
130-
.encode_to_vec(),
131-
))
132-
}
133-
134-
fn deserialize(
135-
&self,
136-
dtype: &DType,
137-
len: usize,
138-
metadata: &[u8],
139-
_buffers: &[BufferHandle],
140-
children: &dyn ArrayChildren,
141-
session: &VortexSession,
142-
) -> VortexResult<ArrayParts<Self>> {
143-
let metadata = ALPMetadata::decode(metadata)?;
144-
let encoded_ptype = match &dtype {
145-
DType::Primitive(PType::F32, n) => DType::Primitive(PType::I32, *n),
146-
DType::Primitive(PType::F64, n) => DType::Primitive(PType::I64, *n),
147-
d => vortex_bail!("expected type: f32 or f64 but instead got {}", d),
148-
};
149-
let encoded = children.get(0, &encoded_ptype, len)?;
150-
151-
let patches = metadata
152-
.patches
153-
.map(|p| {
154-
let indices = children.get(1, &p.indices_dtype()?, p.len()?)?;
155-
let values = children.get(2, dtype, p.len()?)?;
156-
let chunk_offsets = p
157-
.chunk_offsets_dtype()?
158-
.map(|dtype| children.get(3, &dtype, usize::try_from(p.chunk_offsets_len())?))
159-
.transpose()?;
160-
161-
Patches::from_metadata(
162-
len,
163-
&p,
164-
indices,
165-
values,
166-
chunk_offsets,
167-
&mut session.create_execution_ctx(),
168-
)
169-
})
170-
.transpose()?;
171-
172-
let slots = ALPData::make_slots(&encoded, patches.as_ref());
173-
let data = ALPData::new(
174-
Exponents {
175-
e: u8::try_from(metadata.exp_e)?,
176-
f: u8::try_from(metadata.exp_f)?,
177-
},
178-
patches,
179-
);
180-
Ok(ArrayParts::new(
181-
self.clone(),
182-
dtype.clone(),
183-
len,
184-
data,
185-
slots,
186-
))
187-
}
188-
189115
fn slot_name(_array: ArrayView<'_, Self>, idx: usize) -> String {
190116
ALPSlots::NAMES[idx].to_string()
191117
}
Lines changed: 120 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,137 @@
11
// SPDX-License-Identifier: Apache-2.0
22
// SPDX-FileCopyrightText: Copyright the Vortex contributors
33

4-
//! A custom [`ArrayPlugin`] that lets you load in and deserialize an `ALP` array with interior
5-
//! patches as a `PatchedArray` that wraps a patchless `ALP` array.
4+
//! [`ArrayPlugin`] implementations for [`ALP`].
65
//!
7-
//! This enables zero-cost backward compatibility with previously written datasets.
6+
//! [`ALP`] itself reads and writes the current `vortex.alp` form. [`ALPPatchedPlugin`] reads an
7+
//! ALP node with interior patches as a `PatchedArray` wrapping a patchless ALP array, which keeps
8+
//! previously written datasets readable at no cost.
89
10+
use prost::Message;
911
use vortex_array::Array;
1012
use vortex_array::ArrayDeserialization;
1113
use vortex_array::ArrayId;
14+
use vortex_array::ArrayParts;
1215
use vortex_array::ArrayPlugin;
1316
use vortex_array::ArrayRef;
1417
use vortex_array::ArraySerialization;
15-
use vortex_array::ArrayVTable;
1618
use vortex_array::IntoArray;
19+
use vortex_array::VTable;
1720
use vortex_array::VortexSessionExecute;
1821
use vortex_array::arrays::Patched;
22+
use vortex_array::dtype::DType;
23+
use vortex_array::dtype::PType;
24+
use vortex_array::patches::Patches;
1925
use vortex_error::VortexResult;
26+
use vortex_error::vortex_bail;
2027
use vortex_error::vortex_ensure_eq;
2128
use vortex_error::vortex_err;
2229
use vortex_session::VortexSession;
2330

24-
use crate::ALP;
25-
use crate::ALPArrayExt;
26-
use crate::ALPArrayOwnedExt;
31+
use super::ALP;
32+
use super::ALPArrayExt;
33+
use super::ALPArrayOwnedExt;
34+
use super::ALPData;
35+
use super::ALPMetadata;
36+
use crate::alp::Exponents;
37+
38+
impl ArrayPlugin for ALP {
39+
fn id(&self) -> ArrayId {
40+
VTable::id(self)
41+
}
42+
43+
fn serialize(
44+
&self,
45+
array: &ArrayRef,
46+
_session: &VortexSession,
47+
) -> VortexResult<Option<ArraySerialization>> {
48+
vortex_ensure_eq!(
49+
array.encoding_id(),
50+
ArrayPlugin::id(self),
51+
"ALP plugin cannot serialize an array of another encoding",
52+
);
53+
let array = array.as_::<ALP>();
54+
let exponents = array.exponents();
55+
let metadata = ALPMetadata {
56+
exp_e: exponents.e as u32,
57+
exp_f: exponents.f as u32,
58+
patches: array
59+
.patches()
60+
.map(|p| p.to_metadata(array.len(), array.dtype()))
61+
.transpose()?,
62+
}
63+
.encode_to_vec();
64+
Ok(Some(ArraySerialization::from_array(
65+
ArrayPlugin::id(self),
66+
array.array(),
67+
metadata,
68+
)))
69+
}
70+
71+
fn deserialize(
72+
&self,
73+
parts: ArrayDeserialization<'_>,
74+
session: &VortexSession,
75+
) -> VortexResult<ArrayRef> {
76+
vortex_ensure_eq!(
77+
parts.serialized_id,
78+
ArrayPlugin::id(self),
79+
"ALP plugin does not recognize the serialized ID",
80+
);
81+
let ArrayDeserialization {
82+
dtype,
83+
len,
84+
metadata,
85+
children,
86+
..
87+
} = parts;
88+
let metadata = ALPMetadata::decode(metadata)?;
89+
let encoded_ptype = match &dtype {
90+
DType::Primitive(PType::F32, n) => DType::Primitive(PType::I32, *n),
91+
DType::Primitive(PType::F64, n) => DType::Primitive(PType::I64, *n),
92+
d => vortex_bail!("expected type: f32 or f64 but instead got {}", d),
93+
};
94+
let encoded = children.get(0, &encoded_ptype, len)?;
95+
96+
let patches = metadata
97+
.patches
98+
.map(|p| {
99+
let indices = children.get(1, &p.indices_dtype()?, p.len()?)?;
100+
let values = children.get(2, dtype, p.len()?)?;
101+
let chunk_offsets = p
102+
.chunk_offsets_dtype()?
103+
.map(|dtype| children.get(3, &dtype, usize::try_from(p.chunk_offsets_len())?))
104+
.transpose()?;
105+
106+
Patches::from_metadata(
107+
len,
108+
&p,
109+
indices,
110+
values,
111+
chunk_offsets,
112+
&mut session.create_execution_ctx(),
113+
)
114+
})
115+
.transpose()?;
116+
117+
let slots = ALPData::make_slots(&encoded, patches.as_ref());
118+
let data = ALPData::new(
119+
Exponents {
120+
e: u8::try_from(metadata.exp_e)?,
121+
f: u8::try_from(metadata.exp_f)?,
122+
},
123+
patches,
124+
);
125+
Ok(Array::try_from_parts(ArrayParts::new(
126+
self.clone(),
127+
dtype.clone(),
128+
len,
129+
data,
130+
slots,
131+
))?
132+
.into_array())
133+
}
134+
}
27135

28136
/// Custom deserialization plugin that converts an ALP array with interior
29137
/// patches into a PatchedArray holding an ALP array.
@@ -35,15 +143,7 @@ impl ArrayPlugin for ALPPatchedPlugin {
35143
// We reuse the existing `ALP` ID so that we can take over its
36144
// deserialization pathway.
37145
// TODO(joe): dedup method name
38-
ArrayVTable::id(&ALP)
39-
}
40-
41-
fn read_ids(&self) -> Vec<ArrayId> {
42-
vec![self.id()]
43-
}
44-
45-
fn write_ids(&self) -> Vec<ArrayId> {
46-
vec![self.id()]
146+
VTable::id(&ALP)
47147
}
48148

49149
fn serialize(
@@ -65,16 +165,9 @@ impl ArrayPlugin for ALPPatchedPlugin {
65165
self.id(),
66166
"ALP plugin does not recognize serialized ID"
67167
);
68-
let alp_array = Array::<ALP>::try_from_parts(ArrayVTable::deserialize(
69-
&ALP,
70-
parts.dtype,
71-
parts.len,
72-
parts.metadata,
73-
parts.buffers,
74-
parts.children,
75-
session,
76-
)?)
77-
.map_err(|_| vortex_err!("ALP plugin should only deserialize vortex.alp"))?;
168+
let alp_array = ArrayPlugin::deserialize(&ALP, parts, session)?
169+
.try_downcast::<ALP>()
170+
.map_err(|_| vortex_err!("ALP plugin returned an array that is not vortex.alp"))?;
78171

79172
// Check if there are interior patches to externalize.
80173
let Some(patches) = alp_array.patches() else {
@@ -97,7 +190,7 @@ impl ArrayPlugin for ALPPatchedPlugin {
97190

98191
fn is_supported_encoding(&self, id: &ArrayId) -> bool {
99192
// TODO(joe): dedup method name
100-
id == ArrayVTable::id(&Patched) || id == ArrayVTable::id(&ALP)
193+
id == VTable::id(&Patched) || id == VTable::id(&ALP)
101194
}
102195
}
103196

‎encodings/alp/src/alp/mod.rs‎

Lines changed: 15 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -15,10 +15,9 @@ mod compress;
1515
pub(crate) mod compute;
1616
mod decompress;
1717
mod ops;
18-
mod plugin;
1918
mod rules;
2019

21-
pub(crate) use plugin::ALPPatchedPlugin;
20+
pub(crate) use array::ALPPatchedPlugin;
2221

2322
pub(crate) fn initialize(session: &VortexSession) {
2423
rules::initialize(session);
@@ -30,7 +29,8 @@ mod tests {
3029

3130
use prost::Message;
3231
use rstest::rstest;
33-
use vortex_array::Array;
32+
use vortex_array::ArrayDeserialization;
33+
use vortex_array::ArrayPlugin;
3434
use vortex_array::IntoArray;
3535
use vortex_array::VortexSessionExecute;
3636
use vortex_array::arrays::PrimitiveArray;
@@ -39,7 +39,6 @@ mod tests {
3939
use vortex_array::dtype::PType;
4040
use vortex_array::patches::PatchesMetadata;
4141
use vortex_array::test_harness::check_metadata;
42-
use vortex_array::vtable::VTable;
4342
use vortex_error::VortexResult;
4443
use vortex_session::VortexSession;
4544

@@ -81,16 +80,20 @@ mod tests {
8180
let encoded = alp_encode(values.as_view(), Some(Exponents { e: 0, f: 0 }), &mut ctx)?;
8281
let sliced = <ALP as SliceKernel>::slice(encoded.as_view(), start..4096, &mut ctx)?
8382
.expect("ALP slices");
84-
let metadata = ALP::serialize(sliced.as_::<ALP>(), &session)?.expect("ALP serializes");
85-
let parts = ALP.deserialize(
86-
sliced.dtype(),
87-
sliced.len(),
88-
&metadata,
89-
&[],
90-
&sliced.children(),
83+
let serialization =
84+
ArrayPlugin::serialize(&ALP, &sliced, &session)?.expect("ALP serializes");
85+
let read = ArrayPlugin::deserialize(
86+
&ALP,
87+
ArrayDeserialization::new(
88+
serialization.serialized_id,
89+
sliced.dtype(),
90+
sliced.len(),
91+
&serialization.metadata,
92+
&[],
93+
&serialization.children,
94+
),
9195
&session,
9296
)?;
93-
let read = Array::<ALP>::try_from_parts(parts)?.into_array();
9497
let expected = values.into_array().slice(start..4096)?;
9598

9699
let patch_index = ((start / 1024 + 1) * 1024).next_multiple_of(7) - start;

0 commit comments

Comments
 (0)