Repository navigation
Commit 122ca21
authored
Filter compressor schemes by the session's allowed serialized IDs on build (#10067)
Stacked on #10066.
## Summary
`BtrBlocksCompressorBuilder` now takes the serialized array IDs its
schemes may produce directly from the session, and `build()` keeps only
the schemes whose `produced_encodings()` are all allowed. This is the
same rule the file writer uses in `new_array_context`: registered
plugins ∩ enabled editions, or every registered plugin when editions are
disabled.
## Allowed IDs
A new crate-private `AllowedIds` holds three independent restrictions:
| Field | Meaning | `None` / empty |
|---|---|---|
| `registered: Option<HashSet<ArrayId>>` | IDs of the session's
registered array plugins | no restriction |
| `editions: Option<HashSet<ArrayId>>` | IDs the session's enabled
editions permit | no restriction |
| `excluded: HashSet<ArrayId>` | IDs a builder preset excludes (e.g.
future CUDA scheme versions) | nothing excluded |
An ID is allowed when it passes all three. Each builder method touches
its own field, so lifting one restriction never undoes another.
## Builder API
| Method | Effect |
|---|---|
| `from_session` | Session's schemes; `registered` and `editions` from
the session |
| `empty()` | No schemes, no restrictions |
| `disable_editions()` (new) | `editions = None`, still requires
registered plugins. The writer's default strategy calls it when
`disable_editions` is set |
| `unrestricted()` (new) | `registered = None`, `editions = None`;
`excluded` is kept |
| `exclude_schemes` | Unchanged: removes schemes by `SchemeId` |
Filtering happens in `build()`, so it also covers schemes added with
`with_new_scheme` / `with_compact` / `only_cuda_compatible`.
## Removed
- `BtrBlocksCompressorBuilder::retain_allowed_encodings` and
`vortex-bench`'s `retain_edition_encodings`. Their callers (writer,
cuda, python `io`, tui, vortex-bench, compress-bench, golden tests) now
just use `from_session`.
## Behaviour change
`from_session` compressors are now restricted on every session,
in-memory compression included. A session that doesn't register an
encoding, or doesn't enable an edition containing it, won't produce it.
A session with no enabled editions gets a compressor with no schemes.
Tests that compress on such a session, like the dict layout tests in
`vortex-layout` and
`editions::tests::explicit_btrblocks_strategy_is_not_reconfigured` in
`vortex`, now use `unrestricted()`.
## Tests
- btrblocks unit tests, integration tests and benches that compress with
a bare `array_session()` use `unrestricted()`.
- The `vortex-layout` dict layout tests and the `vortex` editions test
for explicit strategies use `unrestricted()`, so they keep compressing
and the editions test still reaches the serialization context check.
- Golden tests keep their edition filtering by registering the
compressor's encodings in `edition_session`.
- The CUDA Zstd test chooses the surviving Zstd scheme by enabling
`core` or `zstd`.
- New builder tests cover no enabled editions, unregistered encodings,
`disable_editions`, `unrestricted`, and excluded IDs surviving both.
## Other
- `vortex-edition` moves from a dev-dependency to a dependency of
`vortex-btrblocks`.
## Checks
- `cargo clippy -p vortex-btrblocks --all-targets --all-features -- -D
warnings`: passed
- `cargo nextest run -p vortex-btrblocks --all-features`: 71 passed, 1
skipped
- `cargo nextest run -p vortex-layout layouts::dict::reader::tests`: 9
passed
- `cargo nextest run -p vortex editions::tests`: 18 passed, 5 skipped
- `cargo clippy -p vortex-layout -p vortex --all-targets --all-features
-- -D warnings`: passed
- Not run: doctests, full fmt, and other downstream crates
(`vortex-file`, `vortex-cuda`, `vortex-bench`, …); CI covers them.
---------
Signed-off-by: Matt Katz <[email protected]>1 parent c78569e commit 122ca21
28 files changed
Lines changed: 422 additions & 203 deletions
File tree
- benchmarks/compress-bench/src/gpu
- vortex-bench/src
- vortex-btrblocks
- benches
- src
- schemes
- float
- integer
- string
- tests
- vortex-cuda/src
- vortex-file
- benches
- src
- vortex-layout/src/layouts/dict
- vortex-python/src
- vortex-test/compat-gen/src/fixtures
- vortex-tui/src
- vortex/src/editions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
40 | | - | |
41 | 40 | | |
42 | 41 | | |
43 | 42 | | |
| |||
100 | 99 | | |
101 | 100 | | |
102 | 101 | | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
107 | | - | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
108 | 105 | | |
109 | 106 | | |
110 | 107 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
66 | 66 | | |
67 | 67 | | |
68 | 68 | | |
69 | | - | |
70 | 69 | | |
71 | 70 | | |
72 | 71 | | |
| |||
248 | 247 | | |
249 | 248 | | |
250 | 249 | | |
251 | | - | |
252 | | - | |
| 250 | + | |
253 | 251 | | |
254 | | - | |
| 252 | + | |
255 | 253 | | |
256 | 254 | | |
257 | 255 | | |
| |||
263 | 261 | | |
264 | 262 | | |
265 | 263 | | |
266 | | - | |
267 | | - | |
| 264 | + | |
268 | 265 | | |
269 | 266 | | |
270 | 267 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
70 | 70 | | |
71 | 71 | | |
72 | 72 | | |
73 | | - | |
74 | | - | |
75 | 73 | | |
76 | 74 | | |
77 | 75 | | |
| |||
255 | 253 | | |
256 | 254 | | |
257 | 255 | | |
258 | | - | |
259 | | - | |
| 256 | + | |
260 | 257 | | |
261 | | - | |
| 258 | + | |
262 | 259 | | |
263 | 260 | | |
264 | 261 | | |
265 | 262 | | |
266 | 263 | | |
267 | 264 | | |
268 | 265 | | |
269 | | - | |
270 | | - | |
271 | | - | |
272 | | - | |
273 | | - | |
274 | | - | |
275 | | - | |
276 | | - | |
277 | | - | |
278 | | - | |
279 | | - | |
280 | | - | |
281 | | - | |
282 | | - | |
283 | | - | |
284 | 266 | | |
285 | 267 | | |
286 | 268 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| |||
50 | 51 | | |
51 | 52 | | |
52 | 53 | | |
53 | | - | |
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| |||
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
54 | | - | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
55 | 57 | | |
56 | 58 | | |
57 | 59 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| |||
183 | 183 | | |
184 | 184 | | |
185 | 185 | | |
186 | | - | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
187 | 189 | | |
188 | 190 | | |
189 | 191 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
0 commit comments