Repository navigation
Commit 47daf16
authored
Move array serde from VTable to explicit ArrayPlugin impls (#11285)
## Summary
Removes the blanket `impl<V: VTable> ArrayPlugin for V` and the
`serialize` / `deserialize` methods on `VTable`. Every serializable
encoding now implements `ArrayPlugin` explicitly on its vtable type, in
a `plugin.rs` child module next to its `impl VTable`. Redo of #10111 on
current develop.
## Changes
- `VTable` covers only in-memory behaviour; serde is `impl ArrayPlugin
for X`.
- Serde bodies are ported verbatim, so wire formats are unchanged.
- `read_ids` / `write_ids` default to `vec![self.id()]`, what the
blanket impl declared. Plugins with one wire ID no longer spell them
out, including `ALPPatchedPlugin`, `ScalarFnArrayPlugin` and the Python
stub.
- Inside an impl the wire ID is `ArrayPlugin::id(self)`; only the `id`
method itself delegates to `VTable::id`.
- `session.arrays().register(X)` is unchanged for every encoding that
was serializable.
- In-memory-only vtables lose the two stub methods and get no plugin:
Filter, Slice, Interleave, ScalarFn, ScalarFnValidity, ForeignArray,
PiecewiseSequence, Widen, Shared, test-harness vtables.
- `DecimalPlugin` (#10995) becomes `impl ArrayPlugin for Decimal`: one
wire ID, same legacy-node reads, no separate type. Its module,
`StoredDecimal` and tests stay where they are.
- Multi-ID plugins (`BitPackedPlugin`, `FoRPlugin`,
`DecimalBytePartsPlugin`) and `WidenPlugin` are untouched;
`ALPPatchedPlugin` now delegates to `ArrayPlugin::deserialize(&ALP,
..)`.
- `PythonVTable` keeps a stub plugin so the Python `register` API still
compiles.
- Vtable files that gained a `plugin.rs` child are directory modules
(`array.rs` becomes `array/mod.rs`, 14 pure renames). ALP's own impl and
`ALPPatchedPlugin` share `alp/array/plugin.rs`.
- Tests that called the vtable serde directly (ALP, ALPRD, Dict, List,
RunEnd, VarBin, RLE, serde.rs, unary, zstd buffers) go through
`ArrayPlugin` instead.
| | count |
|---|---|
| new `impl ArrayPlugin` (vortex-array) | 20 |
| new `impl ArrayPlugin` (encodings + python) | 17 |
| vtables that only lose the stub methods | 14 |
## API changes
- `VTable` loses `serialize` and `deserialize`. An external vtable that
was registered through the blanket impl must now `impl ArrayPlugin` to
be registrable.
- `ArrayPlugin::read_ids` / `write_ids` gain defaults of
`vec![self.id()]`.
- `vortex_array::arrays::DecimalPlugin` is removed; register `Decimal`
instead.
## Behaviour changes
- `PiecewiseSequence` is no longer registered by
`ArraySession::default()`; it never serialized, and its error is now "is
not registered for serialization".
- The writer seeds the footer's encoding list from the session registry,
so dropping `vortex.piecewise-sequence` makes every default-session file
48 bytes smaller. `test_stock_ticker_file_size` expectations move
accordingly (70060 → 70012, 69996 → 69948, 55144 → 55096).
- `DecimalByteParts`, `BitPacked` and `FoR` can no longer be registered
bare, so `bare_vtable_refuses_serde` and `vtable_serde_requires_plugin`
are removed.
## Testing
- `cargo clippy --all-targets --all-features -- -D warnings` and `cargo
+nightly-2026-09-10 fmt --check` on every touched crate (vortex-array,
vortex-file, vortex-layout, vortex-python and the 13 encodings crates).
- `cargo nextest run` on all touched crates plus vortex-ipc and
vortex-scan: 8981 passed, 0 failed; `cargo test --doc` on vortex-array,
fastlanes, decimal-byte-parts and alp.
- `cargo nextest run -p vortex-file -E
'test(test_stock_ticker_file_size)'` with and without `--features zstd`,
to derive the new sizes.
## Follow-up
- Pre-existing, unchanged here: the Patched, RLE, OnPair, ALPRD, FSST
and Zstd deserializers index children or buffers before checking their
counts, so a malformed node panics in `SerializedArray::child` instead
of returning an error. Worth hoisting the count checks in a separate PR.
- Fold `FoRPlugin` / `BitPackedPlugin` / `DecimalBytePartsPlugin` into
`impl ArrayPlugin for FoR` etc. if wanted.
- Overlaps with #11298, which makes the same change with inline impls
and trait defaults for `read_ids` / `write_ids`.
---------
Signed-off-by: Matt Katz <[email protected]>1 parent 7854a19 commit 47daf16
107 files changed
Lines changed: 4372 additions & 3095 deletions
File tree
- docs/developer-guide/internals
- encodings
- alp/src
- alp_rd/array
- alp
- array
- bytebool/src/array
- datetime-parts/src/array
- decimal-byte-parts/src/decimal_byte_parts
- plugin
- elias-fano/src/unary
- vtable
- fastlanes/src
- bitpacking
- plugin
- vtable
- delta/vtable
- for_
- plugin
- vtable
- rle/vtable
- fsst/src/array
- onpair/src/array
- parquet-variant/src/vtable
- pco/src/array
- sequence/src/array
- sparse/src
- zigzag/src/array
- zstd/src
- array
- zstd_buffers
- vortex-array/src
- arrays
- bool/vtable
- chunked/vtable
- constant/vtable
- decimal
- vtable
- plugin
- tests
- dict
- vtable
- extension/vtable
- filter
- fixed_size_list/vtable
- interleave
- listview/vtable
- list/vtable
- map/vtable
- masked/vtable
- null
- patched/vtable
- piecewise_sequence
- primitive/vtable
- runend/array
- scalar_fn
- vtable
- slice
- struct_/vtable
- union/vtable
- varbinview/vtable
- varbin/vtable
- variant/vtable
- widen/vtable
- array
- vtable
- session
- test_harness/trace
- vortex-file/src
- vortex-python/src/arrays/py/vtable
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
121 | 121 | | |
122 | 122 | | |
123 | 123 | | |
124 | | - | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
125 | 127 | | |
126 | 128 | | |
127 | 129 | | |
| |||
176 | 178 | | |
177 | 179 | | |
178 | 180 | | |
179 | | - | |
180 | | - | |
181 | | - | |
182 | | - | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
183 | 186 | | |
184 | 187 | | |
185 | 188 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | | - | |
11 | 10 | | |
12 | 11 | | |
13 | 12 | | |
| |||
21 | 20 | | |
22 | 21 | | |
23 | 22 | | |
24 | | - | |
25 | 23 | | |
26 | 24 | | |
27 | 25 | | |
| |||
34 | 32 | | |
35 | 33 | | |
36 | 34 | | |
37 | | - | |
38 | 35 | | |
39 | 36 | | |
40 | 37 | | |
| |||
45 | 42 | | |
46 | 43 | | |
47 | 44 | | |
48 | | - | |
49 | 45 | | |
50 | 46 | | |
51 | 47 | | |
52 | 48 | | |
53 | 49 | | |
54 | 50 | | |
55 | 51 | | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
56 | 55 | | |
57 | 56 | | |
58 | 57 | | |
| |||
113 | 112 | | |
114 | 113 | | |
115 | 114 | | |
116 | | - | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
121 | | - | |
122 | | - | |
123 | | - | |
124 | | - | |
125 | | - | |
126 | | - | |
127 | | - | |
128 | | - | |
129 | | - | |
130 | | - | |
131 | | - | |
132 | | - | |
133 | | - | |
134 | | - | |
135 | | - | |
136 | | - | |
137 | | - | |
138 | | - | |
139 | | - | |
140 | | - | |
141 | | - | |
142 | | - | |
143 | | - | |
144 | | - | |
145 | | - | |
146 | | - | |
147 | | - | |
148 | | - | |
149 | | - | |
150 | | - | |
151 | | - | |
152 | | - | |
153 | | - | |
154 | | - | |
155 | | - | |
156 | | - | |
157 | | - | |
158 | | - | |
159 | | - | |
160 | | - | |
161 | | - | |
162 | | - | |
163 | | - | |
164 | | - | |
165 | | - | |
166 | | - | |
167 | | - | |
168 | | - | |
169 | | - | |
170 | | - | |
171 | | - | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | | - | |
176 | | - | |
177 | | - | |
178 | | - | |
179 | | - | |
180 | | - | |
181 | | - | |
182 | | - | |
183 | | - | |
184 | | - | |
185 | | - | |
186 | | - | |
187 | | - | |
188 | | - | |
189 | 115 | | |
190 | 116 | | |
191 | 117 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
5 | | - | |
| 4 | + | |
6 | 5 | | |
7 | | - | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
8 | 9 | | |
| 10 | + | |
9 | 11 | | |
10 | 12 | | |
11 | 13 | | |
| 14 | + | |
12 | 15 | | |
13 | 16 | | |
14 | 17 | | |
15 | | - | |
16 | 18 | | |
| 19 | + | |
17 | 20 | | |
18 | 21 | | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
19 | 25 | | |
| 26 | + | |
20 | 27 | | |
21 | 28 | | |
22 | 29 | | |
23 | 30 | | |
24 | | - | |
25 | | - | |
26 | | - | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
27 | 135 | | |
28 | 136 | | |
29 | 137 | | |
| |||
35 | 143 | | |
36 | 144 | | |
37 | 145 | | |
38 | | - | |
39 | | - | |
40 | | - | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
| 146 | + | |
47 | 147 | | |
48 | 148 | | |
49 | 149 | | |
| |||
65 | 165 | | |
66 | 166 | | |
67 | 167 | | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
78 | 171 | | |
79 | 172 | | |
80 | 173 | | |
| |||
97 | 190 | | |
98 | 191 | | |
99 | 192 | | |
100 | | - | |
| 193 | + | |
101 | 194 | | |
102 | 195 | | |
103 | 196 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
18 | | - | |
19 | 18 | | |
20 | 19 | | |
21 | | - | |
| 20 | + | |
22 | 21 | | |
23 | 22 | | |
24 | 23 | | |
| |||
30 | 29 | | |
31 | 30 | | |
32 | 31 | | |
33 | | - | |
| 32 | + | |
| 33 | + | |
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| |||
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
42 | | - | |
43 | 42 | | |
44 | 43 | | |
45 | 44 | | |
| |||
81 | 80 | | |
82 | 81 | | |
83 | 82 | | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
91 | 95 | | |
92 | 96 | | |
93 | | - | |
94 | 97 | | |
95 | 98 | | |
96 | 99 | | |
| |||
0 commit comments