Skip to content

Commit 1a493b6

Browse files
Type edition memberships with a component kind (#9340)
Add support for edition object other than array encodings --------- Signed-off-by: Joe Isaacs <[email protected]>
1 parent c17e058 commit 1a493b6

29 files changed

Lines changed: 910 additions & 194 deletions

File tree

‎Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/specs/editions.md‎

Lines changed: 41 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,39 @@ released through June 2026).
1616
Editions can be used to constrain your minimum required vortex reader, since latest version over vortex across all
1717
editions is the earliest version of vortex required to read that file.
1818

19+
## What an edition contains
20+
21+
Every member of an edition is recorded with a **component kind**: `array`, `layout`, or `aggregate`. The kind is
22+
recorded because ids are unique only within a kind — a layout named `vortex.flat` and an array encoding named
23+
`vortex.flat` are different members — and because membership is resolved one kind at a time. The writer holds a
24+
separate id set per kind and enforces each where that kind is written:
25+
26+
| Kind | Written | Enforced at |
27+
| --- | --- | --- |
28+
| `array` | every serialized array | array serialization context |
29+
| `layout` | the footer's layout tree | layout serialization context |
30+
| `aggregate` | zone maps in zoned layouts | the layout writer context |
31+
32+
**Writing a component outside the enabled editions fails the write, for every kind.** A zone map is only an
33+
optimization, so a forbidden aggregate could in principle be dropped instead — but a file that silently prunes
34+
worse than the writer was configured for is a bug you find in a benchmark six months later, not an error you can
35+
act on. Violations surface at write time or not at all.
36+
37+
Aggregates are checked against the set the write would actually record: an aggregate that a column's dtype cannot
38+
hold is not written, so it is not a violation either.
39+
40+
**A kind with no declared members is unrestricted.** An edition that declares no layouts makes no promise about
41+
layouts, so the writer leaves them alone rather than forbidding all of them; declaring the first member of a kind is
42+
what arms its filter. `core2026.08.0` declares the aggregates the default writer records in zone maps — `min`,
43+
`max`, `bounded_min`, `bounded_max`, `nan_count`, `null_count` — so that filter is armed by default. `sum` is not
44+
among them: a zone sum prunes nothing, so the writer records none. File-level statistics still carry a sum, which
45+
this filter does not govern. A
46+
session that registers components outside `core`, such as the spatial extension types, enables its own edition
47+
family alongside `core`, and the writer may emit the union.
48+
1949
## Resolving an unknown-encoding error
2050

21-
If a read failed with an unknown encoding ID and pointed you here, the reader met an encoding
51+
If a read failed with an unknown encoding ID and pointed you here, the reader met an array encoding
2252
it does not support. Find the encoding ID in the [registry](#edition-registry) below:
2353

2454
1. **The ID is listed under an edition.** The file is newer than your Vortex build. Upgrade to
@@ -39,15 +69,15 @@ someone else's read error later.
3969

4070
The enabled editions are stored on the writer's Vortex session. Registering an edition makes
4171
its declaration available to the session; enabling it separately allows the writer to emit its
42-
encodings. Enabling another edition from the same family replaces the earlier selection.
72+
array encodings. Enabling another edition from the same family replaces the earlier selection.
4373

4474
Two knobs exist when the default is not what you want:
4575

4676
- **Pin an older edition** when files must stay readable by deployments running older Vortex.
4777
- **Opt in to additional edition families.** Editions come in independently versioned,
4878
additive families — `core` today, with families for more specialised encoding groups (for
4979
example spatial encodings) possible later. A writer targets at most one edition per family
50-
and may emit any encoding in their union; each encoding belongs to exactly one family.
80+
and may emit any encoding in their union; each member belongs to exactly one family.
5181

5282
Lower-level sessions without an enabled-editions store opt out of editions entirely and can write
5383
custom or experimental encodings. A raw `with_allow_encodings` writer policy is another explicit
@@ -56,12 +86,15 @@ encodings can read the files.
5686

5787
## How editions change
5888

59-
A published edition is frozen — its encoding list never grows or shrinks. New encodings are
89+
A published edition is frozen — its member list never grows or shrinks. New members are
6090
staged in a **draft** edition and become guaranteed only when that draft is frozen as the next
61-
edition; each encoding's registry entry records the edition it joined in. In the future an
62-
encoding may be *deprecated*, meaning writers stop emitting it — but readers keep decoding it
63-
indefinitely, so deprecation never invalidates existing files.
91+
edition; each member's registry entry records its component kind and the edition it joined in.
92+
Declaring the first member of a kind arms that kind's write-time filter, so a kind gains
93+
enforcement at the edition that first declares one.
94+
In the future an encoding may be *deprecated*, meaning writers stop emitting it — but readers
95+
keep decoding it indefinitely, so deprecation never invalidates existing files.
6496

6597
## Edition registry
6698

67-
Coming soon..
99+
Coming soon.. It will list each edition's members with their component kind, the edition they
100+
joined in, and the Vortex release required to read them.

‎encodings/parquet-variant/src/vtable.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -416,7 +416,7 @@ mod tests {
416416
.read(|map| map.keys().copied().collect::<Vec<_>>());
417417
for id in ids {
418418
editions
419-
.declare_inclusion(EditionInclusion::new(&id, TEST_EDITION))
419+
.declare_inclusion(EditionInclusion::array(&id, TEST_EDITION))
420420
.map_err(|error| vortex_err!("{error}"))?;
421421
}
422422
session

‎vortex-edition/src/lib.rs‎

Lines changed: 128 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,24 @@
11
// SPDX-License-Identifier: Apache-2.0
22
// SPDX-FileCopyrightText: Copyright the Vortex contributors
33

4-
//! Definitions of Vortex *editions*: named, frozen sets of encodings that a writer may put in
5-
//! a file, carrying a forever read-compatibility guarantee.
4+
//! Definitions of Vortex *editions*: named, frozen sets of components that a writer may put
5+
//! in a file, carrying a forever read-compatibility guarantee.
66
//!
77
//! Editions live on the session, like encodings do: [`EditionSession`] holds the registered
88
//! editions and [`EnabledEditions`] selects which of them a writer may emit. Declarations
99
//! are plain constants — an [`EditionId`] plus an [`Edition`] record, and one
10-
//! [`EditionInclusion`] per encoding stating that it is a member of an edition *and every
10+
//! [`EditionInclusion`] per member stating that it is a member of an edition *and every
1111
//! later edition of the same family*. Any crate can register declarations into a session,
12-
//! so inclusions can live next to the encoding they describe.
12+
//! so inclusions can live next to the component they describe.
13+
//!
14+
//! Every membership is typed by a [`ComponentKind`], and members are resolved one kind at a
15+
//! time with [`EditionSessionExt::enabled_component_ids`]: the file writer restricts the
16+
//! arrays, layouts, and aggregates it writes from three separate id sets, never one untyped
17+
//! set.
1318
//!
1419
//! An edition is a **draft** until its [`Edition::min_vortex_version`] is recorded —
15-
//! recording it is the act of freezing. The per-edition encoding sets are computed from the
16-
//! registered declarations by [`EditionSession::encodings_in`], and correctness is enforced
20+
//! recording it is the act of freezing. The per-edition member sets are computed from the
21+
//! registered declarations by [`EditionSession::components_in`], and correctness is enforced
1722
//! by unit tests: [`EditionSession::validate`] checks a whole registry, and
1823
//! [`test_harness::validate_edition`] validates one edition's constraints — call it once in
1924
//! the `#[cfg(test)]` module of each edition definition.
@@ -109,15 +114,42 @@ impl Display for EditionId {
109114
}
110115
}
111116

112-
/// An edition: a named set of encodings with a read-compatibility guarantee, registered with
117+
/// The kind of component an edition membership covers.
118+
///
119+
/// Ids are unique per kind, not globally: a layout named `vortex.flat` and an array named
120+
/// `vortex.flat` are different members. Every membership records its kind, and the writer
121+
/// resolves one kind at a time, so the set restricting written arrays never restricts
122+
/// written layouts. Further kinds (scalar functions, say) can be added the same way.
123+
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
124+
pub enum ComponentKind {
125+
/// An array encoding, e.g. `vortex.alp`, registered in the session's array registry.
126+
Array,
127+
/// A layout encoding, e.g. `vortex.flat`, registered in the session's layout registry.
128+
Layout,
129+
/// An aggregate function, e.g. `vortex.min`, written into zone maps and registered in
130+
/// the session's aggregate function registry.
131+
Aggregate,
132+
}
133+
134+
impl Display for ComponentKind {
135+
fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
136+
f.write_str(match self {
137+
Self::Array => "array",
138+
Self::Layout => "layout",
139+
Self::Aggregate => "aggregate",
140+
})
141+
}
142+
}
143+
144+
/// An edition: a named set of components with a read-compatibility guarantee, registered with
113145
/// [`EditionSession::declare_edition`]. The set itself is computed from the registered
114-
/// [`EditionInclusion`]s by [`EditionSession::encodings_in`].
146+
/// [`EditionInclusion`]s by [`EditionSession::components_in`].
115147
#[derive(Clone, Copy, Debug)]
116148
pub struct Edition {
117149
/// The edition identifier. Also carries the freeze date: `core2026.07.0` freezes in
118150
/// 2026-07.
119151
pub id: EditionId,
120-
/// The minimum Vortex version whose reader supports every encoding in this edition.
152+
/// The minimum Vortex version whose reader supports every member of this edition.
121153
///
122154
/// Recording this is the act of freezing: an edition with `None` is a **draft** — being
123155
/// assembled, carrying no guarantee, free to change, never the default write target.
@@ -133,83 +165,133 @@ impl Edition {
133165
}
134166
}
135167

136-
/// Declares that an encoding is a member of an edition — and of every later edition of the
168+
/// Declares that a component is a member of an edition — and of every later edition of the
137169
/// same family. Registered with [`EditionSession::declare_inclusion`].
138170
#[derive(Clone, Copy, Debug)]
139171
pub struct EditionInclusion {
140-
/// The interned encoding id, e.g. `vortex.alp`. Globally unique across everything an
141-
/// edition can cover: when layout encodings join editions, their ids must be distinct
142-
/// from array encoding ids.
143-
pub encoding_id: Id,
144-
/// The first edition this encoding is a member of.
172+
/// What the membership covers. Ids are unique per kind, so this is part of the
173+
/// member's identity, not a label.
174+
pub kind: ComponentKind,
175+
/// The interned component id, e.g. `vortex.alp`.
176+
pub component_id: Id,
177+
/// The first edition this component is a member of.
145178
pub since: EditionId,
146-
/// The earliest Vortex release able to read and execute this encoding, recorded from
179+
/// The earliest Vortex release able to read and execute this component, recorded from
147180
/// evidence (e.g. compat-fixture history). `None` until recorded.
148181
pub required_vortex_release: Option<&'static str>,
149182
}
150183

151-
/// A source of an encoding id for edition declarations.
184+
/// A source of a component id for edition declarations.
152185
///
153186
/// Implemented for raw id strings (`"vortex.alp"`) and interned [`Id`]s here; encoding
154187
/// vtables implement it where they are defined, so a declaration can name the vtable
155-
/// (`&Primitive`) instead of spelling its id.
156-
pub trait AsEncodingId: Debug + Send + Sync {
157-
/// The interned encoding id.
158-
fn encoding_id(&self) -> Id;
188+
/// (`&Primitive`) instead of spelling its id. The id alone does not say what kind of
189+
/// component it names — [`EditionMember`] pairs it with a [`ComponentKind`].
190+
pub trait AsComponentId: Debug + Send + Sync {
191+
/// The interned component id.
192+
fn component_id(&self) -> Id;
159193
}
160194

161-
impl AsEncodingId for str {
195+
impl AsComponentId for str {
162196
#[expect(
163197
clippy::disallowed_methods,
164-
reason = "interning a dynamic encoding id at declaration time"
198+
reason = "interning a dynamic component id at declaration time"
165199
)]
166-
fn encoding_id(&self) -> Id {
200+
fn component_id(&self) -> Id {
167201
Id::new(self)
168202
}
169203
}
170204

171-
impl AsEncodingId for Id {
172-
fn encoding_id(&self) -> Id {
205+
impl AsComponentId for Id {
206+
fn component_id(&self) -> Id {
173207
*self
174208
}
175209
}
176210

177-
// `str` is unsized and cannot be a trait object, so declaration blocks (slices of
178-
// `&dyn AsEncodingId`) name encodings as `&"vortex.alp"` through this impl.
179-
impl AsEncodingId for &'static str {
180-
fn encoding_id(&self) -> Id {
181-
(**self).encoding_id()
211+
// `str` is unsized and cannot be a trait object, so declaration blocks name components as
212+
// `&"vortex.alp"` through this impl.
213+
impl AsComponentId for &'static str {
214+
fn component_id(&self) -> Id {
215+
(**self).component_id()
182216
}
183217
}
184218

185-
/// Declares an edition together with the encodings that join the family at it, in one
186-
/// block. Registered with [`EditionSession::declare`], which derives each encoding's
219+
/// A component that joins an edition, named by id string or vtable and tagged with the kind
220+
/// of registry it belongs to. Built with the per-kind constructors, so a declaration reads
221+
/// as `EditionMember::array(&"vortex.alp")`.
222+
#[derive(Clone, Copy, Debug)]
223+
pub struct EditionMember {
224+
/// What kind of component this is.
225+
pub kind: ComponentKind,
226+
/// The component, named by id string or by vtable.
227+
pub component: &'static dyn AsComponentId,
228+
}
229+
230+
impl EditionMember {
231+
/// An array encoding member, e.g. `vortex.alp`.
232+
pub const fn array(component: &'static dyn AsComponentId) -> Self {
233+
Self {
234+
kind: ComponentKind::Array,
235+
component,
236+
}
237+
}
238+
239+
/// A layout member, e.g. `vortex.flat`.
240+
pub const fn layout(component: &'static dyn AsComponentId) -> Self {
241+
Self {
242+
kind: ComponentKind::Layout,
243+
component,
244+
}
245+
}
246+
247+
/// An aggregate function member, e.g. `vortex.min`.
248+
pub const fn aggregate(component: &'static dyn AsComponentId) -> Self {
249+
Self {
250+
kind: ComponentKind::Aggregate,
251+
component,
252+
}
253+
}
254+
}
255+
256+
/// Declares an edition together with the components that join the family at it, in one
257+
/// block. Registered with [`EditionSession::declare`], which derives each member's
187258
/// membership (`since` = the declared edition) from the block structure.
188259
#[derive(Clone, Copy, Debug)]
189260
pub struct EditionDeclaration {
190261
/// The edition being declared.
191262
pub edition: Edition,
192-
/// The encodings that join the family at this edition, named by id string or by
193-
/// vtable. Members of earlier editions are inherited and never restated.
194-
pub added: &'static [&'static dyn AsEncodingId],
263+
/// The components that join the family at this edition, each tagged with its
264+
/// [`ComponentKind`]. Members of earlier editions are inherited and never restated.
265+
pub added: &'static [EditionMember],
195266
}
196267

197268
impl EditionInclusion {
198-
/// Declare that an encoding is a member of `since` and every later edition of the same
199-
/// family. The encoding can be named by id string or by vtable.
200-
pub fn new<E: AsEncodingId + ?Sized>(encoding: &E, since: EditionId) -> Self {
269+
/// Declare that a component of `kind` is a member of `since` and every later edition of
270+
/// the same family. The component can be named by id string or by vtable.
271+
pub fn new<C: AsComponentId + ?Sized>(
272+
kind: ComponentKind,
273+
component: &C,
274+
since: EditionId,
275+
) -> Self {
201276
Self {
202-
encoding_id: encoding.encoding_id(),
277+
kind,
278+
component_id: component.component_id(),
203279
since,
204280
required_vortex_release: None,
205281
}
206282
}
207283

208-
/// Validate the declaration's form: a lowercase `namespace.name` encoding id and, if
284+
/// Declare that an array encoding is a member of `since` and every later edition of the
285+
/// same family.
286+
pub fn array<C: AsComponentId + ?Sized>(encoding: &C, since: EditionId) -> Self {
287+
Self::new(ComponentKind::Array, encoding, since)
288+
}
289+
290+
/// Validate the declaration's form: a lowercase `namespace.name` component id and, if
209291
/// recorded, a well-formed `major.minor.patch` release. Checked for every declared
210292
/// inclusion by [`EditionSession::validate`].
211293
pub fn validate(&self) -> Result<(), EditionError> {
212-
let id = self.encoding_id.as_str();
294+
let id = self.component_id.as_str();
213295
let well_formed = !id.starts_with('.')
214296
&& !id.ends_with('.')
215297
&& id.contains('.')
@@ -218,15 +300,16 @@ impl EditionInclusion {
218300
.all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || "._-".contains(c));
219301
if !well_formed {
220302
return Err(EditionError::new(format!(
221-
"invalid encoding id {id:?}: expected lowercase `namespace.name`, e.g. \
222-
`vortex.alp`"
303+
"invalid {} id {id:?}: expected lowercase `namespace.name`, e.g. `vortex.alp`",
304+
self.kind
223305
)));
224306
}
225307
if let Some(release) = self.required_vortex_release
226308
&& parse_release(release).is_none()
227309
{
228310
return Err(EditionError::new(format!(
229-
"encoding {id} declares malformed required_vortex_release {release:?}"
311+
"{} {id} declares malformed required_vortex_release {release:?}",
312+
self.kind
230313
)));
231314
}
232315
Ok(())

0 commit comments

Comments
 (0)