Repository navigation
Move compression schemes to CompressionSession and construct BtrBlocksCompressorBuilder from Session - #10066
Conversation
Signed-off-by: Matt Katz <[email protected]>
…om_session Signed-off-by: Matt Katz <[email protected]>
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | filter_powerlaw_by_mostly_true[250000] |
102.3 µs | 149 µs | -31.35% |
| ⚡ | Simulation | allocate_drop_vortex[64] |
66.7 µs | 52.1 µs | +28.02% |
| ⚡ | Simulation | allocate_drop_vortex[256] |
67.5 µs | 53.1 µs | +27.1% |
| ⚡ | Simulation | allocate_drop_vortex[1024] |
110.1 µs | 91.4 µs | +20.45% |
| ⚡ | Simulation | take[nullable/sorted/primitive/nullable_all_valid/chunks=4/indices=1000] |
269.4 µs | 231.4 µs | +16.42% |
| ⚡ | Simulation | take[duplicates/duplicate90/primitive/nonnull/chunks=16/indices=1000] |
298.2 µs | 260.1 µs | +14.64% |
| ⚡ | Simulation | take_fsl_chunked_elements_sorted[16, 64] |
252.5 µs | 225.5 µs | +12% |
| Simulation | fixed_16_advancing_ptr_safe[100] |
< 1 ns | < 1 ns | N/A | |
| Simulation | preverify_advancing_ptr_unchecked[1000] |
< 1 ns | < 1 ns | N/A | |
| Simulation | preverify_advancing_ptr_unchecked[10000] |
< 1 ns | < 1 ns | N/A |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing mk/compression-session (299f964) with develop (21b3dea)3
Footnotes
-
437 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
1 benchmark was run, but is now archived. If it was deleted in another branch, consider rebasing to remove it from the report. Instead if it was added back, click here to restore it. ↩
-
No successful run was found on
develop(0fea4a3) during the generation of this report, so 21b3dea was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
Signed-off-by: Matt Katz <[email protected]>
BtrBlocksCompressorBuilder from session
BtrBlocksCompressorBuilder from session CompressionSession and construct BtrBlocksCompressorBuilder from Session
Signed-off-by: Matt Katz <[email protected]>
Signed-off-by: Matt Katz <[email protected]>
Signed-off-by: Matt Katz <[email protected]>
…build (#10067) Stacked on #10066. ## Summary `BtrBlocksCompressorBuilder` now takes the serialized array IDs its schemes may produce directly from the session, and `build()` keeps only the schemes whose `produced_encodings()` are all allowed. This is the same rule the file writer uses in `new_array_context`: registered plugins ∩ enabled editions, or every registered plugin when editions are disabled. ## Allowed IDs A new crate-private `AllowedIds` holds three independent restrictions: | Field | Meaning | `None` / empty | |---|---|---| | `registered: Option<HashSet<ArrayId>>` | IDs of the session's registered array plugins | no restriction | | `editions: Option<HashSet<ArrayId>>` | IDs the session's enabled editions permit | no restriction | | `excluded: HashSet<ArrayId>` | IDs a builder preset excludes (e.g. future CUDA scheme versions) | nothing excluded | An ID is allowed when it passes all three. Each builder method touches its own field, so lifting one restriction never undoes another. ## Builder API | Method | Effect | |---|---| | `from_session` | Session's schemes; `registered` and `editions` from the session | | `empty()` | No schemes, no restrictions | | `disable_editions()` (new) | `editions = None`, still requires registered plugins. The writer's default strategy calls it when `disable_editions` is set | | `unrestricted()` (new) | `registered = None`, `editions = None`; `excluded` is kept | | `exclude_schemes` | Unchanged: removes schemes by `SchemeId` | Filtering happens in `build()`, so it also covers schemes added with `with_new_scheme` / `with_compact` / `only_cuda_compatible`. ## Removed - `BtrBlocksCompressorBuilder::retain_allowed_encodings` and `vortex-bench`'s `retain_edition_encodings`. Their callers (writer, cuda, python `io`, tui, vortex-bench, compress-bench, golden tests) now just use `from_session`. ## Behaviour change `from_session` compressors are now restricted on every session, in-memory compression included. A session that doesn't register an encoding, or doesn't enable an edition containing it, won't produce it. A session with no enabled editions gets a compressor with no schemes. Tests that compress on such a session, like the dict layout tests in `vortex-layout` and `editions::tests::explicit_btrblocks_strategy_is_not_reconfigured` in `vortex`, now use `unrestricted()`. ## Tests - btrblocks unit tests, integration tests and benches that compress with a bare `array_session()` use `unrestricted()`. - The `vortex-layout` dict layout tests and the `vortex` editions test for explicit strategies use `unrestricted()`, so they keep compressing and the editions test still reaches the serialization context check. - Golden tests keep their edition filtering by registering the compressor's encodings in `edition_session`. - The CUDA Zstd test chooses the surviving Zstd scheme by enabling `core` or `zstd`. - New builder tests cover no enabled editions, unregistered encodings, `disable_editions`, `unrestricted`, and excluded IDs surviving both. ## Other - `vortex-edition` moves from a dev-dependency to a dependency of `vortex-btrblocks`. ## Checks - `cargo clippy -p vortex-btrblocks --all-targets --all-features -- -D warnings`: passed - `cargo nextest run -p vortex-btrblocks --all-features`: 71 passed, 1 skipped - `cargo nextest run -p vortex-layout layouts::dict::reader::tests`: 9 passed - `cargo nextest run -p vortex editions::tests`: 18 passed, 5 skipped - `cargo clippy -p vortex-layout -p vortex --all-targets --all-features -- -D warnings`: passed - Not run: doctests, full fmt, and other downstream crates (`vortex-file`, `vortex-cuda`, `vortex-bench`, …); CI covers them. --------- Signed-off-by: Matt Katz <[email protected]>
Moves the btrblocks scheme list out of the builder and onto the session, and makes every compressor and write strategy take its schemes from a session.
vortex_btrblocks::CompressionSessionsession var holds the registered schemes in registration order (the tie-break order).Defaultregisters what wasALL_SCHEMES;CompressionSessionExt::{compression, register_scheme}read and extend it. Registering a duplicateSchemeIdpanics, matchingwith_new_scheme.Defaultis removed fromBtrBlocksCompressorBuilder,BtrBlocksCompressorandWriteStrategyBuilder. Each is constructed withfrom_session(&session)(the builder also keepsempty()). The other builder methods are unchanged.ALL_SCHEMESis removed.DELTA_SCHEMEstays.SESSION,session,ctx.session()). Two signatures change:string-bench'sserialized_write_strategynow takes the session, and the parquet-variantwrite_strategyfixture is replaced by building the strategy from the test's session.vortex::compressorre-exportsCompressionSessionandCompressionSessionExt.