You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1f81692
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/specs/versioning/design.md
+44-16Lines changed: 44 additions & 16 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -55,18 +55,50 @@ and nulls. An array can contain buffers and other arrays, called _children_. A d
55
55
example, has a child for its values and another for the codes that refer to those values. Each child
56
56
can use its own encoding.
57
57
58
-
The array's in-memory ID selects its serializer, which returns a wire ID, metadata, buffers, and
59
-
children. Those children are then serialized through their own plugins. Conversely, the reader uses
60
-
each stored wire ID to select a deserializer and the contract it must validate. Since a plugin can
61
-
adapt metadata, buffers, or children in either direction, the in-memory and serialized structures
62
-
need not be identical. However, both conversions must preserve values, data types, and nulls.
58
+
### Array plugin conversions
59
+
60
+
An array plugin provides the code to serialize an in-memory array and read serialized data back into
61
+
an array. A plugin can support several wire formats. Support grows by adding wire IDs, while the
62
+
rules for frozen IDs stay fixed and readers retain the code needed to read them.
63
+
64
+
When writing an array, Vortex finds its serializer using the array's in-memory encoding ID. The
65
+
serializer chooses a wire format and returns that format's wire ID, metadata, buffers, and child
66
+
arrays. It can construct different metadata, buffers, or children from those in the input array to
67
+
meet the chosen format's requirements. The library can therefore change its in-memory array data
68
+
structures while continuing to serialize data according to the same wire format. Each returned child
69
+
is then serialized through its own plugin.
70
+
71
+
When reading, however, Vortex finds the deserializer using the wire ID stored in the file. As it
72
+
constructs an in-memory array, the deserializer must check that the metadata, buffers, data types,
73
+
and children satisfy the rules for that ID. It can arrange the buffers and children differently in
74
+
memory, or return an array whose encoding ID differs from the plugin's own in-memory ID. **A plugin
75
+
can therefore read several wire formats into the same in-memory array type.**
76
+
77
+
These operations do not upgrade or downgrade the file. A reader can use an array implementation
78
+
added after the file was written, but the file's contents and wire IDs remain unchanged. Similarly,
79
+
a writer can use its current array implementation to serialize data in a wire format introduced by
80
+
an earlier library release. Neither operation requires a separate in-memory array type for each wire
81
+
format. Both must preserve values, data types, and nulls, although the array's structure in memory
82
+
can change when it is serialized and read back.
83
+
84
+
An application can opt out of a particular conversion during reading by
85
+
[registering another plugin for the same wire ID](../../developer-guide/internals/session.md#registering-plugins).
86
+
That plugin must read the same serialized data correctly, but it can construct a different encoding
87
+
in memory. This gives the application control over how the data is represented without changing the
88
+
file or the rules for interpreting it.
89
+
90
+
For example, the ALP wire format stores exceptional values, called patches, inside the ALP array.
91
+
With experimental `Patched` support enabled, the registered plugin constructs a `Patched` parent
92
+
that holds those patches and an ALP child that has none. In contrast, the standard ALP plugin keeps
93
+
the patches inside the ALP array in memory. Registering the standard plugin opts out of that change
94
+
to the array structure, while still reading the same stored data under the same wire ID.
63
95
64
96
## Example: decimal children
65
97
66
-
The decimal-byte-parts encoding stores decimal values in integer child arrays. It can store each
67
-
value in one child or split it across several children. One in-memory array type handles both
68
-
shapes, but the v1 wire contract permits only one child. Supporting additional children therefore
69
-
requires a new wire ID.[^decimal-availability]
98
+
The decimal-byte-parts plugin can serialize the same in-memory array type using two wire formats.
99
+
Its encoding stores decimal values in integer child arrays, either in one child or split across
100
+
several children. The v1 wire contract permits only one child, while v2 adds support for multiple
101
+
children under a distinct wire ID.[^decimal-availability]
70
102
71
103
```{figure} ../../_static/versioning-flow.svg
72
104
:alt: One decimal encoding holds either one signed child or a signed child with unsigned lower parts. The serializer chooses v1 for one child and v2 for multiple children. Both wire formats deserialize into the same array type.
@@ -91,7 +123,7 @@ Both wire formats deserialize into the same in-memory array type. However, the r
91
123
validate the contract identified by the stored ID: `vortex.decimal_byte_parts` requires exactly one
92
124
signed integer child. Support for multiple children under v2 does not make them valid under v1.
93
125
94
-
###Format selection
126
+
## Format selection
95
127
96
128
A serializer must choose the oldest supported writable format that preserves the array's
97
129
representation without recompression. A plugin can adapt metadata, buffers, or children to fit an
@@ -104,12 +136,8 @@ through this serializer, which selects v1.
104
136
105
137
This policy preserves compatibility with readers of the earlier wire format when the existing
106
138
representation allows it. If compatibility requires a different encoding of the same values, the
107
-
write path must arrange recompression before serialization or fail.
108
-
109
-
Reading can also change the array structure. For example, the ALP floating-point wire format stores
110
-
exceptional values, called patches, inside the ALP array. With the experimental `Patched` encoding
111
-
enabled, the reader moves those patches into a `Patched` parent around an ALP child without patches.
112
-
The values stay the same even though the reader's array tree differs from the stored tree.
139
+
write path must arrange recompression before serialization or fail. Selecting an edition does not
0 commit comments