Repository navigation
Filter compressor schemes by the session's allowed serialized IDs on build - #10067
Conversation
09e28d4 to
b7a53d4
Compare
Merging this PR will improve performance by 20.48%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | mul_u64_nonnull_neon |
39.9 µs | 29.1 µs | +37.38% |
| ⚡ | WallTime | multiply_shapes_neon[(32768, PerRowPerRow)] |
38.6 µs | 32.8 µs | +17.69% |
| ⚡ | WallTime | mul_i64_nonnull_neon |
38.5 µs | 32.8 µs | +17.24% |
| ⚡ | Simulation | sum_v2_i64 |
224.6 µs | 193.9 µs | +15.87% |
| ⚡ | Simulation | sum_i64 |
223.7 µs | 193.6 µs | +15.56% |
| Simulation | fixed_16_advancing_ptr_safe[100] |
< 1 ns | < 1 ns | N/A | |
| Simulation | preverify_advancing_ptr_unchecked[1000] |
< 1 ns | < 1 ns | N/A | |
| Simulation | preverify_advancing_ptr_unchecked[10000] |
< 1 ns | < 1 ns | N/A |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing mk/compressor-allowed-ids (689cf25) with develop (23a4249)3
Footnotes
-
437 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
1 benchmark was run, but is now archived. If it was deleted in another branch, consider rebasing to remove it from the report. Instead if it was added back, click here to restore it. ↩
-
No successful run was found on
develop(d8e45e0) during the generation of this report, so 23a4249 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
8cea229 to
f31e31f
Compare
Signed-off-by: Matt Katz <[email protected]>
…ncodings with disable_editions Signed-off-by: Matt Katz <[email protected]>
Signed-off-by: Matt Katz <[email protected]>
f31e31f to
11abba0
Compare
…compressor docs Signed-off-by: Matt Katz <[email protected]>
Signed-off-by: Matt Katz <[email protected]>
Signed-off-by: Matt Katz <[email protected]>
Signed-off-by: Matt Katz <[email protected]>
… by builder mode (#10069) Stacked on #10067. ## Summary `CompressionSession::default()` now registers every built-in scheme, including Delta and the feature-gated Zstd and Pco schemes. The builder picks which of them to use with a crate-private preset mode, applied in `build()`. ## Registered schemes The session now also registers: - Delta, from a private static in `session.rs` - string and binary `ZstdScheme`, and binary `ZstdBuffersScheme` (with `zstd`) - integer and float `PcoScheme` (with `pco`) Each goes last in its type's group, so tie-break order within a type is unchanged. String-level `ZstdBuffersScheme` is still not registered, since no preset used it. ## Presets The builder holds one crate-private `CompressionMode`. `with_compact()` and `only_cuda_compatible()` now just set it, and the last one set wins: | Mode | Set by | Excluded in `build()` | |---|---|---| | `All` | `empty()` | nothing | | `Default` | `from_session` | all Zstd, Pco | | `Compact` | `with_compact()` | binary `ZstdBuffers` | | `Cuda` | `only_cuda_compatible()` | the existing CUDA exclusion list, Delta, Pco, string Zstd | `build()` drops the mode's exclusions, then applies the `AllowedIds` serialized ID filter from #10067. `with_compact()` is no longer gated behind the `zstd` feature; only the schemes are. `with_new_scheme` and `exclude_schemes` are unchanged from `develop`. ## Behaviour changes - The order of preset calls no longer matters. `with_compact().only_cuda_compatible()` used to panic on the duplicate binary Zstd scheme; now it gives the CUDA preset. - `with_compact()` and `only_cuda_compatible()` on an `empty()` builder no longer add any schemes, because they only filter. - Delta is registered and only `Cuda` excludes it, resolving the `TODO(robert)` on `DELTA_SCHEME`. The `DELTA_SCHEME` public static is removed: the session owns the only instance, and `integer::DeltaScheme` stays public for callers that want a custom ratio. No edition includes `fastlanes.delta`, so the enabled editions keep it out of edition-enforced writes. With editions disabled or `unrestricted()`, the compressor may now choose Delta: compat fixtures published with editions disabled, and benchmarks such as `split_collection` and the btrblocks benches, can start using it. - Adding Delta with `with_new_scheme` on a `from_session` builder now panics as a duplicate, since Delta is registered. The btrblocks tests that did this drop the call. ## Other - `string-bench`'s scheme inventory test filters out string Zstd, which is registered but excluded by the default preset. ## Checks - `cargo clippy -p vortex-btrblocks --all-targets --all-features -- -D warnings` and `--no-default-features`: passed - `cargo nextest run -p vortex-btrblocks --all-features`: 73 passed, 1 skipped - `cargo test --doc -p vortex-btrblocks`: 3 passed - `RUSTDOCFLAGS="-D warnings" cargo doc -p vortex-btrblocks --no-deps --all-features`: passed - `cargo clippy -p vortex-file --all-targets` with `--all-features`, default, and `--no-default-features`: passed - `cargo nextest run -p vortex -p vortex-file -p vortex-layout -p string-bench -p vortex-compat`: 493 passed - `rustfmt +nightly-2026-09-10 --check` on touched files: passed - Not run: workspace-wide clippy and tests, benchmarks, CUDA e2e; CI covers them. --------- Signed-off-by: Matt Katz <[email protected]>
## Summary Compat fixtures are written with `disable_editions()`, so a fixture can record an encoding with no read-compatibility guarantee. Once published, the additive-only manifest rule makes that encoding permanent. This PR writes fixtures under the default session's enabled editions (`core2026.08.3`), so a fixture can only emit encodings an edition already guarantees. ## Changes - **Gated writes.** `adapter::write_compressed` and `adapter::write_compressed_to_bytes_with_session` no longer call `disable_editions()`. `check` regenerates its comparison files through the same path, so it is gated too. - **Dataset compressor.** TPC-H and ClickBench fixtures build their compressor and write strategy from `VortexSession::default()`, the session the file is written with, instead of from `array_session()` with `disable_editions()`. Since #10067, `BtrBlocksCompressorBuilder::from_session` keeps only schemes whose encodings the session registers and its editions allow. The compressor therefore can't choose an encoding the gated write would reject, such as `vortex.patched` under `VORTEX_EXPERIMENTAL_PATCHES`. The regular and compact variants now share one code path. - **`decimal_byte_parts_v2.vortex` disabled**, like `delta` and `patched`. It writes `vortex.decimal_byte_parts.v2`, which no edition includes yet. It isn't in any published manifest (checked through 0.86.1), so removing it doesn't break the additive-only rule. It comes back once DBP v2 joins an edition. - **`DESIGN.md`** notes that new fixtures must use encodings from an enabled edition. Every fixture in the 0.86.1 manifest uses `core` encodings. That comes from reading the code, not from a run, so the published fixture set should be unchanged. ## Checks - [x] `cargo clippy -p vortex-compat --all-targets --all-features -- -D warnings` - [x] `cargo +nightly-2026-09-10 fmt -p vortex-compat --check` - [ ] `cargo nextest run -p vortex-compat` - [ ] `cargo run -p vortex-compat --release -- generate --output /tmp/compat --exclude clickbench`: confirms every remaining fixture writes under `core` --------- Signed-off-by: Matt Katz <[email protected]>
Since #10067, BtrBlocksCompressor::from_session only keeps schemes whose serialized IDs the session allows, and the bare test session enables no edition, so the probe never chose Dict and write_dict_layout fell back to a flat layout. Build the probe unrestricted, as the dict reader tests do. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_012f6YvQVEjuUwc3VEet3Hst Signed-off-by: Joe Isaacs <[email protected]>
Stacked on #10066.
Summary
BtrBlocksCompressorBuildernow takes the serialized array IDs its schemes may produce directly from the session, andbuild()keeps only the schemes whoseproduced_encodings()are all allowed. This is the same rule the file writer uses innew_array_context: registered plugins ∩ enabled editions, or every registered plugin when editions are disabled.Allowed IDs
A new crate-private
AllowedIdsholds three independent restrictions:None/ emptyregistered: Option<HashSet<ArrayId>>editions: Option<HashSet<ArrayId>>excluded: HashSet<ArrayId>An ID is allowed when it passes all three. Each builder method touches its own field, so lifting one restriction never undoes another.
Builder API
from_sessionregisteredandeditionsfrom the sessionempty()disable_editions()(new)editions = None, still requires registered plugins. The writer's default strategy calls it whendisable_editionsis setunrestricted()(new)registered = None,editions = None;excludedis keptexclude_schemesSchemeIdFiltering happens in
build(), so it also covers schemes added withwith_new_scheme/with_compact/only_cuda_compatible.Removed
BtrBlocksCompressorBuilder::retain_allowed_encodingsandvortex-bench'sretain_edition_encodings. Their callers (writer, cuda, pythonio, tui, vortex-bench, compress-bench, golden tests) now just usefrom_session.Behaviour change
from_sessioncompressors are now restricted on every session, in-memory compression included. A session that doesn't register an encoding, or doesn't enable an edition containing it, won't produce it. A session with no enabled editions gets a compressor with no schemes. Tests that compress on such a session, like the dict layout tests invortex-layoutandeditions::tests::explicit_btrblocks_strategy_is_not_reconfiguredinvortex, now useunrestricted().Tests
array_session()useunrestricted().vortex-layoutdict layout tests and thevortexeditions test for explicit strategies useunrestricted(), so they keep compressing and the editions test still reaches the serialization context check.edition_session.coreorzstd.disable_editions,unrestricted, and excluded IDs surviving both.Other
vortex-editionmoves from a dev-dependency to a dependency ofvortex-btrblocks.Checks
cargo clippy -p vortex-btrblocks --all-targets --all-features -- -D warnings: passedcargo nextest run -p vortex-btrblocks --all-features: 71 passed, 1 skippedcargo nextest run -p vortex-layout layouts::dict::reader::tests: 9 passedcargo nextest run -p vortex editions::tests: 18 passed, 5 skippedcargo clippy -p vortex-layout -p vortex --all-targets --all-features -- -D warnings: passedvortex-file,vortex-cuda,vortex-bench, …); CI covers them.