Skip to content

Commit 1f81692

Browse files
committed
array plugin semantics
Signed-off-by: Connor Tsui <[email protected]>
1 parent 0ac688c commit 1f81692

1 file changed

Lines changed: 44 additions & 16 deletions

File tree

‎docs/specs/versioning/design.md‎

Lines changed: 44 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -55,18 +55,50 @@ and nulls. An array can contain buffers and other arrays, called _children_. A d
5555
example, has a child for its values and another for the codes that refer to those values. Each child
5656
can use its own encoding.
5757

58-
The array's in-memory ID selects its serializer, which returns a wire ID, metadata, buffers, and
59-
children. Those children are then serialized through their own plugins. Conversely, the reader uses
60-
each stored wire ID to select a deserializer and the contract it must validate. Since a plugin can
61-
adapt metadata, buffers, or children in either direction, the in-memory and serialized structures
62-
need not be identical. However, both conversions must preserve values, data types, and nulls.
58+
### Array plugin conversions
59+
60+
An array plugin provides the code to serialize an in-memory array and read serialized data back into
61+
an array. A plugin can support several wire formats. Support grows by adding wire IDs, while the
62+
rules for frozen IDs stay fixed and readers retain the code needed to read them.
63+
64+
When writing an array, Vortex finds its serializer using the array's in-memory encoding ID. The
65+
serializer chooses a wire format and returns that format's wire ID, metadata, buffers, and child
66+
arrays. It can construct different metadata, buffers, or children from those in the input array to
67+
meet the chosen format's requirements. The library can therefore change its in-memory array data
68+
structures while continuing to serialize data according to the same wire format. Each returned child
69+
is then serialized through its own plugin.
70+
71+
When reading, however, Vortex finds the deserializer using the wire ID stored in the file. As it
72+
constructs an in-memory array, the deserializer must check that the metadata, buffers, data types,
73+
and children satisfy the rules for that ID. It can arrange the buffers and children differently in
74+
memory, or return an array whose encoding ID differs from the plugin's own in-memory ID. **A plugin
75+
can therefore read several wire formats into the same in-memory array type.**
76+
77+
These operations do not upgrade or downgrade the file. A reader can use an array implementation
78+
added after the file was written, but the file's contents and wire IDs remain unchanged. Similarly,
79+
a writer can use its current array implementation to serialize data in a wire format introduced by
80+
an earlier library release. Neither operation requires a separate in-memory array type for each wire
81+
format. Both must preserve values, data types, and nulls, although the array's structure in memory
82+
can change when it is serialized and read back.
83+
84+
An application can opt out of a particular conversion during reading by
85+
[registering another plugin for the same wire ID](../../developer-guide/internals/session.md#registering-plugins).
86+
That plugin must read the same serialized data correctly, but it can construct a different encoding
87+
in memory. This gives the application control over how the data is represented without changing the
88+
file or the rules for interpreting it.
89+
90+
For example, the ALP wire format stores exceptional values, called patches, inside the ALP array.
91+
With experimental `Patched` support enabled, the registered plugin constructs a `Patched` parent
92+
that holds those patches and an ALP child that has none. In contrast, the standard ALP plugin keeps
93+
the patches inside the ALP array in memory. Registering the standard plugin opts out of that change
94+
to the array structure, while still reading the same stored data under the same wire ID.
6395

6496
## Example: decimal children
6597

66-
The decimal-byte-parts encoding stores decimal values in integer child arrays. It can store each
67-
value in one child or split it across several children. One in-memory array type handles both
68-
shapes, but the v1 wire contract permits only one child. Supporting additional children therefore
69-
requires a new wire ID.[^decimal-availability]
98+
The decimal-byte-parts plugin can serialize the same in-memory array type using two wire formats.
99+
Its encoding stores decimal values in integer child arrays, either in one child or split across
100+
several children. The v1 wire contract permits only one child, while v2 adds support for multiple
101+
children under a distinct wire ID.[^decimal-availability]
70102

71103
```{figure} ../../_static/versioning-flow.svg
72104
:alt: One decimal encoding holds either one signed child or a signed child with unsigned lower parts. The serializer chooses v1 for one child and v2 for multiple children. Both wire formats deserialize into the same array type.
@@ -91,7 +123,7 @@ Both wire formats deserialize into the same in-memory array type. However, the r
91123
validate the contract identified by the stored ID: `vortex.decimal_byte_parts` requires exactly one
92124
signed integer child. Support for multiple children under v2 does not make them valid under v1.
93125

94-
### Format selection
126+
## Format selection
95127

96128
A serializer must choose the oldest supported writable format that preserves the array's
97129
representation without recompression. A plugin can adapt metadata, buffers, or children to fit an
@@ -104,12 +136,8 @@ through this serializer, which selects v1.
104136

105137
This policy preserves compatibility with readers of the earlier wire format when the existing
106138
representation allows it. If compatibility requires a different encoding of the same values, the
107-
write path must arrange recompression before serialization or fail.
108-
109-
Reading can also change the array structure. For example, the ALP floating-point wire format stores
110-
exceptional values, called patches, inside the ALP array. With the experimental `Patched` encoding
111-
enabled, the reader moves those patches into a `Patched` parent around an ALP child without patches.
112-
The values stay the same even though the reader's array tree differs from the stored tree.
139+
write path must arrange recompression before serialization or fail. Selecting an edition does not
140+
perform that conversion automatically.
113141

114142
## Write and read checks
115143

0 commit comments

Comments
 (0)