Skip to content

Commit f476db4

Browse files
authored
Add an unstable edition that includes all unreleased and upcoming encodings (#10558)
UNSTABLE exists to aggregate all core encodings that are in the works and should be tested and benchmarked but not written by default writer
1 parent cbe2fb5 commit f476db4

56 files changed

Lines changed: 522 additions & 138 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/specs/versioning.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -74,8 +74,12 @@ forbids, or a custom strategy can choose an unsupported layout. The
7474
[write fails](versioning/using-editions.md#write-errors) when its serialized output violates the
7575
selection.
7676

77-
[Draft editions](versioning/editions.md#draft-editions) have no frozen compatibility guarantee.
78-
Custom wire formats written with edition checks disabled also fall outside the edition guarantee.
77+
[Draft editions](versioning/editions.md#draft-editions) have no frozen compatibility guarantee,
78+
and their members can change or be removed. The
79+
[unstable edition](versioning/editions.md#unstable), `UNSTABLE`, is a draft that never
80+
freezes, so the wire contracts of its members can also change under existing IDs. It accepts
81+
plugin-defined members. Readers need matching implementations for those formats. Custom wire formats
82+
written with edition checks disabled also fall outside the guarantee.
7983

8084
## Upgrades
8185

‎docs/specs/versioning/design.md‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -250,9 +250,9 @@ historical writing implementation.
250250
## New formats
251251

252252
A new format starts in a draft edition so it can be tested before its origin commits to reading it
253-
indefinitely. Drafts have no recorded minimum version or frozen guarantee. A format intended for
254-
`core` can progress from its own family to `preview` for broader testing, then to `core` for default
255-
use. Promotion preserves its wire ID and interpretation. The
253+
indefinitely. Drafts have no recorded minimum version or frozen guarantee, and their members can
254+
change or be removed. A format intended for `core` can progress to `preview` for broader testing,
255+
then to `core` for default use. Promotion preserves its wire ID and interpretation. The
256256
[registry instructions](editions.md#format-testing-and-promotion) cover promotion, freezing, and
257257
recording the minimum version.
258258

@@ -271,9 +271,10 @@ variant, which splits values too wide for one 64-bit child into several children
271271
`vortex.decimal_byte_parts.v2` is allowed. Otherwise its v1 variant constructs only single-child
272272
arrays, and values too wide for one child remain in the standard uncompressed decimal
273273
representation. The v2 variant also writes the v1 ID for arrays that fit the v1 format, so allowing
274-
only the v2 ID excludes the scheme. No declared edition permits `vortex.decimal_byte_parts.v2` yet,
275-
so a writer that enforces editions writes only single-child decimal arrays.
274+
only the v2 ID excludes the scheme. No frozen edition permits `vortex.decimal_byte_parts.v2`, so a
275+
writer that enforces editions writes multi-child decimal arrays only when a selected draft edition
276+
permits it.
276277

277278
[^decimal-availability]:
278-
No declared edition currently permits the multi-child format. The
279+
No frozen edition currently permits the multi-child format. The
279280
[compression section](#compression) describes what the default compressor produces today.

‎docs/specs/versioning/editions.md‎

Lines changed: 34 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -62,10 +62,16 @@ Minimum Vortex Rust crate version: `0.85.0`.
6262

6363
## Draft editions
6464

65-
Draft editions have no recorded minimum version of their origin project's code, and each new format
66-
or revision requires a new draft edition. Vortex-maintained draft formats are expected to remain
67-
compatible unless a defect blocks promotion into `core`, while independent plugin projects state
68-
their own policy.
65+
Draft editions have no recorded minimum version of their origin project's code. Until a draft
66+
freezes, its members can change or be removed. Except for `UNSTABLE`, Vortex-maintained draft
67+
formats are expected to remain compatible unless a defect blocks promotion into `core`, while
68+
independent plugin projects state their own policy.
69+
70+
### `unstable`
71+
72+
`UNSTABLE` is a draft edition that never freezes. It holds formats whose contracts are still
73+
changing, so its members and their wire contracts can change under the same IDs. Plugins can add
74+
members through ordinary edition inclusions.
6975

7076
### `preview2026.08.0`
7177

@@ -113,22 +119,30 @@ For example, `core2026.08.0` declares `min`, `max`, `bounded_min`, `bounded_max`
113119
statistics, however, do store sums, in a fixed legacy field governed by the enclosing format's
114120
contract.
115121

122+
Aggregate serialization support alone does not permit an aggregate in persisted zone maps. Its ID
123+
must belong to a selected edition, including `UNSTABLE` for experimental formats. The `all_*`
124+
aggregates are in-memory pruning predicates and do not need entries. Legacy file statistics also
125+
do not need entries.
126+
116127
## Format testing and promotion
117128

118-
A format intended for `core` starts in a dedicated edition family. Its first edition is a draft,
119-
with no recorded `min_library_version` and no frozen compatibility guarantee. Even at this draft
120-
stage, the format is expected to be complete. If testing reveals a defect whose correction changes
121-
what readers must understand, the correction needs a new wire ID and a later edition.
129+
A format with a changing contract belongs in `UNSTABLE`. Its inclusion can be added, changed, or
130+
removed without a new edition or wire ID. Applications writing these formats need readers with
131+
matching implementations.
132+
133+
Once the contract is settled, move its inclusion from `UNSTABLE` to another edition. A format
134+
intended for `core` enters `preview` for broader opt-in use, then a later `core` edition for default
135+
use. Optional modules use their own families. The format can keep its ID, but earlier experimental
136+
files acquire no compatibility guarantee.
122137

123-
After initial testing, the format can enter a new `preview` edition for broader opt-in use, then a
124-
later `core` edition for default use. Promotion changes which editions permit the format while
125-
preserving its contract and wire ID. The current `preview` edition is empty, so its first component
126-
must go into a new edition.
138+
A frozen contract cannot change. A reader-visible correction to a frozen format requires a new wire
139+
ID and a later edition.
127140

128141
## Freezing an edition
129142

130-
A stable edition can freeze when its origin publishes the code that first supports all its members.
131-
For `core`, this is a Vortex Rust crate release. Independent plugins use their own versions.
143+
A draft edition other than `UNSTABLE` can freeze when its origin publishes the code that first
144+
supports all its members. For `core`, this is a Vortex Rust crate release. Independent plugins use
145+
their own versions.
132146

133147
Until the release version is known, the declaration uses `min_library_version: None`. Once it is
134148
known, the field records the first release that supports all members, usually while the next release
@@ -144,10 +158,10 @@ Default declarations live in `vortex-edition/src/declarations/`. Optional module
144158
with their implementation code. The exported TOML records are under `vortex/editions/`, grouped by
145159
family.
146160

147-
1. For a new component, declare its own family and draft edition. For a revision, add a later
148-
edition to the family that owns the earlier ID.
149-
2. To promote a tested format, add it to new `preview` and `core` editions without changing its ID
150-
or contract.
161+
1. Register a changing format in `UNSTABLE`. Third-party plugins use ordinary edition inclusions.
162+
2. Once its contract is settled, move its inclusion from `UNSTABLE` to a draft edition. Formats
163+
intended for `core` pass through `preview`; optional modules use their own families. Revisions
164+
to frozen members require a new ID and a later edition.
151165
3. When an edition freezes, record the first release of its origin that supports every permitted
152166
component, including inherited members.
153167
4. Regenerate the records:
@@ -158,4 +172,5 @@ family.
158172

159173
CI's `check-editions` command rejects changes to frozen records, including renames, unfreezing, and
160174
deletion, so changes to permitted wire formats require a later edition. The command also rejects a
161-
new edition that does not follow its family's chronology.
175+
new edition that does not follow its family's chronology. Draft records, including `UNSTABLE`, can
176+
change or be deleted.

‎docs/specs/versioning/using-editions.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,8 @@ array and a layout can share the same ID string while describing different forma
102102
required optional module.
103103
- For a draft edition, obtain a build that implements the component from the producer. A draft does
104104
not promise support in a published release.
105+
- For an unstable format, obtain the matching implementation from the producer. The same ID in
106+
another build may describe a different contract.
105107
- For a component absent from the registry, obtain its implementation from the producer and register
106108
it with the session.
107109

‎encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -259,7 +259,7 @@ impl VTable for DecimalByteParts {
259259
_array: ArrayView<'_, Self>,
260260
_session: &VortexSession,
261261
) -> VortexResult<Option<Vec<u8>>> {
262-
vortex_bail!("DecimalByteParts serialization requires DecimalBytePartsPlugin")
262+
Ok(None)
263263
}
264264

265265
fn deserialize(

‎encodings/decimal-byte-parts/src/decimal_byte_parts/plugin/tests.rs‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -254,11 +254,9 @@ fn bare_vtable_refuses_serde() -> VortexResult<()> {
254254
session.arrays().register(DecimalByteParts);
255255
let msp = msp();
256256
let array = DecimalByteParts::try_new(msp.clone(), DecimalDType::new(19, 2))?.into_array();
257-
let result = session.array_serialize(&array);
257+
let result = session.array_serialize(&array)?;
258258
assert!(
259-
result.as_ref().is_err_and(|err| err
260-
.to_string()
261-
.contains("DecimalByteParts serialization requires DecimalBytePartsPlugin")),
259+
result.is_none(),
262260
"expected unsupported VTable serialization, got {result:?}"
263261
);
264262

‎encodings/fastlanes/src/bitpacking/plugin/tests.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -247,7 +247,7 @@ fn vtable_serde_requires_plugin() -> VortexResult<()> {
247247
let array = BitPackedData::encode(&values, 2, &mut SESSION.create_execution_ctx())?;
248248
let session = VortexSession::empty();
249249
session.arrays().register(BitPacked);
250-
assert!(session.array_serialize(&array.into_array()).is_err());
250+
assert!(session.array_serialize(&array.into_array())?.is_none());
251251
Ok(())
252252
}
253253

‎encodings/fastlanes/src/bitpacking/vtable/mod.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -170,7 +170,7 @@ impl VTable for BitPacked {
170170
_array: ArrayView<'_, Self>,
171171
_session: &VortexSession,
172172
) -> VortexResult<Option<Vec<u8>>> {
173-
vortex_bail!("BitPacked serialization requires BitPackedPlugin")
173+
Ok(None)
174174
}
175175

176176
fn deserialize(

‎encodings/fastlanes/src/for_/plugin/tests.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -139,6 +139,6 @@ fn vtable_serde_requires_plugin() -> VortexResult<()> {
139139
)?;
140140
let session = VortexSession::empty();
141141
session.arrays().register(FoR);
142-
assert!(session.array_serialize(&array.into_array()).is_err());
142+
assert!(session.array_serialize(&array.into_array())?.is_none());
143143
Ok(())
144144
}

‎encodings/fastlanes/src/for_/vtable/mod.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -132,7 +132,7 @@ impl VTable for FoR {
132132
_array: ArrayView<'_, Self>,
133133
_session: &VortexSession,
134134
) -> VortexResult<Option<Vec<u8>>> {
135-
vortex_bail!("FoR serialization requires FoRPlugin")
135+
Ok(None)
136136
}
137137

138138
fn deserialize(

0 commit comments

Comments
 (0)