Skip to content

Commit 122ca21

Browse files
authored
Filter compressor schemes by the session's allowed serialized IDs on build (#10067)
Stacked on #10066. ## Summary `BtrBlocksCompressorBuilder` now takes the serialized array IDs its schemes may produce directly from the session, and `build()` keeps only the schemes whose `produced_encodings()` are all allowed. This is the same rule the file writer uses in `new_array_context`: registered plugins ∩ enabled editions, or every registered plugin when editions are disabled. ## Allowed IDs A new crate-private `AllowedIds` holds three independent restrictions: | Field | Meaning | `None` / empty | |---|---|---| | `registered: Option<HashSet<ArrayId>>` | IDs of the session's registered array plugins | no restriction | | `editions: Option<HashSet<ArrayId>>` | IDs the session's enabled editions permit | no restriction | | `excluded: HashSet<ArrayId>` | IDs a builder preset excludes (e.g. future CUDA scheme versions) | nothing excluded | An ID is allowed when it passes all three. Each builder method touches its own field, so lifting one restriction never undoes another. ## Builder API | Method | Effect | |---|---| | `from_session` | Session's schemes; `registered` and `editions` from the session | | `empty()` | No schemes, no restrictions | | `disable_editions()` (new) | `editions = None`, still requires registered plugins. The writer's default strategy calls it when `disable_editions` is set | | `unrestricted()` (new) | `registered = None`, `editions = None`; `excluded` is kept | | `exclude_schemes` | Unchanged: removes schemes by `SchemeId` | Filtering happens in `build()`, so it also covers schemes added with `with_new_scheme` / `with_compact` / `only_cuda_compatible`. ## Removed - `BtrBlocksCompressorBuilder::retain_allowed_encodings` and `vortex-bench`'s `retain_edition_encodings`. Their callers (writer, cuda, python `io`, tui, vortex-bench, compress-bench, golden tests) now just use `from_session`. ## Behaviour change `from_session` compressors are now restricted on every session, in-memory compression included. A session that doesn't register an encoding, or doesn't enable an edition containing it, won't produce it. A session with no enabled editions gets a compressor with no schemes. Tests that compress on such a session, like the dict layout tests in `vortex-layout` and `editions::tests::explicit_btrblocks_strategy_is_not_reconfigured` in `vortex`, now use `unrestricted()`. ## Tests - btrblocks unit tests, integration tests and benches that compress with a bare `array_session()` use `unrestricted()`. - The `vortex-layout` dict layout tests and the `vortex` editions test for explicit strategies use `unrestricted()`, so they keep compressing and the editions test still reaches the serialization context check. - Golden tests keep their edition filtering by registering the compressor's encodings in `edition_session`. - The CUDA Zstd test chooses the surviving Zstd scheme by enabling `core` or `zstd`. - New builder tests cover no enabled editions, unregistered encodings, `disable_editions`, `unrestricted`, and excluded IDs surviving both. ## Other - `vortex-edition` moves from a dev-dependency to a dependency of `vortex-btrblocks`. ## Checks - `cargo clippy -p vortex-btrblocks --all-targets --all-features -- -D warnings`: passed - `cargo nextest run -p vortex-btrblocks --all-features`: 71 passed, 1 skipped - `cargo nextest run -p vortex-layout layouts::dict::reader::tests`: 9 passed - `cargo nextest run -p vortex editions::tests`: 18 passed, 5 skipped - `cargo clippy -p vortex-layout -p vortex --all-targets --all-features -- -D warnings`: passed - Not run: doctests, full fmt, and other downstream crates (`vortex-file`, `vortex-cuda`, `vortex-bench`, …); CI covers them. --------- Signed-off-by: Matt Katz <[email protected]>
1 parent c78569e commit 122ca21

28 files changed

Lines changed: 422 additions & 203 deletions

File tree

‎benchmarks/compress-bench/src/gpu/vortex.rs‎

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,6 @@ use vortex_bench::compress::CompressedData;
3737
use vortex_bench::compress::Compressor;
3838
use vortex_bench::compress::Uncompressed;
3939
use vortex_bench::conversions::parquet_to_vortex_chunks_with_batch_size;
40-
use vortex_bench::retain_edition_encodings;
4140
use vortex_cuda::CanonicalCudaExt;
4241
use vortex_cuda::CudaExecutionCtx;
4342
use vortex_cuda::CudaOpenOptionsExt;
@@ -100,11 +99,9 @@ impl Compressor for GpuVortexCompressor {
10099
// partition rather than whatever the default strategy would regroup them into.
101100
let strategy = Arc::new(ChunkedLayoutStrategy::new(CompressingStrategy::new(
102101
CudaFlatLayoutStrategy::default(),
103-
retain_edition_encodings(
104-
&SESSION,
105-
BtrBlocksCompressorBuilder::from_session(&SESSION).only_cuda_compatible(),
106-
)
107-
.build(),
102+
BtrBlocksCompressorBuilder::from_session(&SESSION)
103+
.only_cuda_compatible()
104+
.build(),
108105
)));
109106
let start = Instant::now();
110107
SESSION

‎vortex-bench/src/conversions.rs‎

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,6 @@ use wkb::writer::write_geometry;
6666
use crate::CompactionStrategy;
6767
use crate::Format;
6868
use crate::SESSION;
69-
use crate::retain_edition_encodings;
7069
use crate::utils::file::idempotent_async;
7170

7271
/// Memory budget per concurrent conversion stream in GB. This is somewhat arbitary.
@@ -248,10 +247,9 @@ fn write_options_for(
248247

249248
let mut builder = WriteStrategyBuilder::from_session(&SESSION);
250249
if matches!(compaction, CompactionStrategy::Compact) {
251-
builder = builder.with_btrblocks_builder(retain_edition_encodings(
252-
&SESSION,
250+
builder = builder.with_btrblocks_builder(
253251
BtrBlocksCompressorBuilder::from_session(&SESSION).with_compact(),
254-
));
252+
);
255253
}
256254
for name in binary_fields {
257255
builder = builder.with_field_writer(FieldPath::from_name(name), no_dict_layout());
@@ -263,8 +261,7 @@ fn write_options_for(
263261
fn no_dict_layout() -> Arc<dyn LayoutStrategy> {
264262
Arc::new(CompressingStrategy::new(
265263
ChunkedLayoutStrategy::new(FlatLayoutStrategy::default()),
266-
retain_edition_encodings(&SESSION, BtrBlocksCompressorBuilder::from_session(&SESSION))
267-
.build(),
264+
BtrBlocksCompressorBuilder::from_session(&SESSION).build(),
268265
))
269266
}
270267

‎vortex-bench/src/lib.rs‎

Lines changed: 2 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -70,8 +70,6 @@ pub use datasets::BenchmarkDataset;
7070
pub use output::BenchmarkOutput;
7171
pub use output::create_output_writer;
7272
use vortex::VortexSessionDefault;
73-
use vortex::editions::ComponentKind;
74-
use vortex::editions::EditionSessionExt;
7573
pub use vortex::error::vortex_panic;
7674
use vortex::io::session::RuntimeSessionExt;
7775
use vortex::session::VortexSession;
@@ -255,32 +253,16 @@ impl CompactionStrategy {
255253
match self {
256254
CompactionStrategy::Compact => options.with_strategy(
257255
WriteStrategyBuilder::from_session(&SESSION)
258-
.with_btrblocks_builder(retain_edition_encodings(
259-
&SESSION,
256+
.with_btrblocks_builder(
260257
BtrBlocksCompressorBuilder::from_session(&SESSION).with_compact(),
261-
))
258+
)
262259
.build(),
263260
),
264261
CompactionStrategy::Default => options,
265262
}
266263
}
267264
}
268265

269-
/// Restrict `builder` to the encodings permitted by the session's enabled editions.
270-
///
271-
/// The default writer applies this filter itself. An explicit strategy bypasses it, so a
272-
/// benchmark that builds its own compressor applies it here to stay within editions.
273-
pub fn retain_edition_encodings(
274-
session: &VortexSession,
275-
builder: BtrBlocksCompressorBuilder,
276-
) -> BtrBlocksCompressorBuilder {
277-
let allowed = session
278-
.enabled_component_ids(ComponentKind::Array)
279-
.into_iter()
280-
.collect();
281-
builder.retain_allowed_encodings(&allowed)
282-
}
283-
284266
/// Verify that local data has already been prepared for the requested benchmark formats.
285267
///
286268
/// Engine-specific benchmark binaries call this before running queries. Data generation itself

‎vortex-btrblocks/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ vortex-buffer = { workspace = true }
2626
vortex-compressor = { workspace = true }
2727
vortex-datetime-parts = { workspace = true }
2828
vortex-decimal-byte-parts = { workspace = true }
29+
vortex-edition = { workspace = true }
2930
vortex-error = { workspace = true }
3031
vortex-fastlanes = { workspace = true }
3132
vortex-fsst = { workspace = true }
@@ -50,7 +51,6 @@ tpchgen = { workspace = true }
5051
tpchgen-arrow = { workspace = true }
5152
vortex-array = { workspace = true, features = ["_test-harness"] }
5253
vortex-arrow = { workspace = true }
53-
vortex-edition = { workspace = true }
5454
vortex-mask = { workspace = true }
5555

5656
[features]

‎vortex-btrblocks/benches/compress.rs‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ mod benchmarks {
1919
use vortex_array::IntoArray;
2020
use vortex_array::VortexSessionExecute;
2121
use vortex_array::arrays::PrimitiveArray;
22-
use vortex_btrblocks::BtrBlocksCompressor;
22+
use vortex_btrblocks::BtrBlocksCompressorBuilder;
2323
use vortex_buffer::buffer_mut;
2424
use vortex_session::VortexSession;
2525
use vortex_utils::aliases::hash_set::HashSet;
@@ -51,7 +51,9 @@ mod benchmarks {
5151
let array = make_clickbench_window_name()
5252
.execute::<PrimitiveArray>(&mut ctx)
5353
.unwrap();
54-
let compressor = BtrBlocksCompressor::from_session(&SESSION);
54+
let compressor = BtrBlocksCompressorBuilder::from_session(&SESSION)
55+
.unrestricted()
56+
.build();
5557
bencher
5658
.with_inputs(|| (&array, SESSION.create_execution_ctx()))
5759
.input_counter(|(array, _)| ItemsCount::new(array.len()))

‎vortex-btrblocks/benches/compress_listview.rs‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ mod benchmarks {
2424
use vortex_array::arrays::VarBinViewArray;
2525
use vortex_array::dtype::FieldNames;
2626
use vortex_array::validity::Validity;
27-
use vortex_btrblocks::BtrBlocksCompressor;
27+
use vortex_btrblocks::BtrBlocksCompressorBuilder;
2828
use vortex_buffer::buffer_mut;
2929
use vortex_session::VortexSession;
3030

@@ -183,7 +183,9 @@ mod benchmarks {
183183
fn compress_listview(bencher: Bencher, layout: OffsetLayout) {
184184
let array = build_nested_listview(NUM_ROWS, layout);
185185
let nbytes = array.nbytes();
186-
let compressor = BtrBlocksCompressor::from_session(&SESSION);
186+
let compressor = BtrBlocksCompressorBuilder::from_session(&SESSION)
187+
.unrestricted()
188+
.build();
187189
bencher
188190
.with_inputs(|| (&array, SESSION.create_execution_ctx()))
189191
.input_counter(|_| ItemsCount::new(NUM_ROWS))
Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
// SPDX-License-Identifier: Apache-2.0
2+
// SPDX-FileCopyrightText: Copyright the Vortex contributors
3+
4+
//! The serialized array IDs a compressor may produce.
5+
6+
use vortex_array::ArrayId;
7+
use vortex_array::session::ArraySessionExt;
8+
use vortex_edition::ComponentKind;
9+
use vortex_edition::EditionSessionExt;
10+
use vortex_session::VortexSession;
11+
use vortex_utils::aliases::hash_set::HashSet;
12+
13+
/// The serialized array IDs a compressor's schemes may produce.
14+
///
15+
/// An ID is allowed when it passes every restriction: it is in `registered` and `editions` (when
16+
/// set) and not in `excluded`. A scheme is kept only if every ID it declares in
17+
/// [`produced_encodings`](crate::Scheme::produced_encodings) is allowed.
18+
///
19+
/// The restrictions are independent, so lifting one never undoes another.
20+
#[derive(Debug, Clone, Default, PartialEq, Eq)]
21+
pub(crate) struct AllowedIds {
22+
/// The IDs of the session's registered array plugins, or `None` for no restriction.
23+
pub(crate) registered: Option<HashSet<ArrayId>>,
24+
/// The IDs the session's enabled editions permit, or `None` for no restriction.
25+
pub(crate) editions: Option<HashSet<ArrayId>>,
26+
/// IDs excluded by a builder preset, such as the CUDA one. Always applied.
27+
pub(crate) excluded: HashSet<ArrayId>,
28+
}
29+
30+
impl AllowedIds {
31+
/// Allows every serialized ID.
32+
pub(crate) fn all() -> Self {
33+
Self::default()
34+
}
35+
36+
/// Allows the serialized array IDs that are registered in the session and permitted by its
37+
/// enabled editions, matching what the file writer can write.
38+
///
39+
/// A session with no enabled editions permits no IDs.
40+
pub(crate) fn from_session(session: &VortexSession) -> Self {
41+
let registered = session
42+
.arrays()
43+
.registry()
44+
.read(|registry| registry.keys().copied().collect());
45+
let editions = session
46+
.enabled_component_ids(ComponentKind::Array)
47+
.into_iter()
48+
.collect();
49+
Self {
50+
registered: Some(registered),
51+
editions: Some(editions),
52+
excluded: HashSet::default(),
53+
}
54+
}
55+
56+
/// Returns whether `id` is allowed.
57+
pub(crate) fn is_allowed(&self, id: &ArrayId) -> bool {
58+
!self.excluded.contains(id)
59+
&& self.registered.as_ref().is_none_or(|ids| ids.contains(id))
60+
&& self.editions.as_ref().is_none_or(|ids| ids.contains(id))
61+
}
62+
}

0 commit comments

Comments
 (0)