Repository navigation
Register Delta, Zstd and Pco in the compression session, exclude them by builder mode - #10069
Merged
Merged
Conversation
Merging this PR will not alter performance
|
robert3005
force-pushed
the
mk/compressor-scheme-modes
branch
from
September 25, 2026 17:09
963ed9b to
f97c93b
Compare
mhk197
force-pushed
the
mk/compressor-scheme-modes
branch
from
September 25, 2026 19:23
f97c93b to
e47cc71
Compare
mhk197
marked this pull request as ready for review
September 25, 2026 19:41
mhk197
requested review from
connortsui20,
joseph-isaacs and
robert3005
and removed request for
connortsui20 and
joseph-isaacs
September 25, 2026 20:15
… by builder mode Signed-off-by: Matt Katz <[email protected]>
Signed-off-by: Matt Katz <[email protected]>
Signed-off-by: Matt Katz <[email protected]>
Signed-off-by: Matt Katz <[email protected]>
Signed-off-by: Matt Katz <[email protected]>
…mode crate-private Signed-off-by: Matt Katz <[email protected]>
connortsui20
force-pushed
the
mk/compressor-scheme-modes
branch
from
September 25, 2026 20:52
5fa660d to
8e2ebad
Compare
robert3005
reviewed
Sep 25, 2026
Signed-off-by: Matt Katz <[email protected]>
robert3005
approved these changes
Sep 25, 2026
mhk197
added a commit
that referenced
this pull request
Sep 28, 2026
…zed IDs permit it (#10076) Stacked on #10069. ## Summary Wide `i128` and `i256` decimals compress into multi-part `DecimalByteParts` arrays when the compressor's allowed serialized IDs include `vortex.decimal_byte_parts.v2`. The session registers the v1 decimal scheme, and `build()` refines it to v2 wherever both decimal IDs are allowed. This PR sets the stage for versioned schemes. ## `Scheme::refine` New defaulted hook in `vortex-compressor`: ```rust fn refine(&self, _allowed: &dyn Fn(&ArrayId) -> bool) -> Option<&'static dyn Scheme> { None } ``` - It returns the variant of the scheme to use for the allowed IDs; `None` keeps the scheme. - A variant must keep the scheme's `SchemeId`, and every ID it declares in `produced_encodings` must still be allowed for it to be used. - The allowed IDs are a predicate rather than a set, so builders that allow every ID (`empty()`, `unrestricted()`) are represented exactly. ## Builder `build()` now: 1. drops the mode's excluded schemes, 2. refines each remaining scheme against the allowed IDs, 3. keeps the schemes whose produced IDs are all allowed. The allowed IDs are the session's `AllowedIds` minus the mode's excluded serialized IDs. The `Cuda` mode excludes `vortex.decimal_byte_parts.v2`, since multi-part arrays have no CUDA decode kernel, so the CUDA preset keeps v1. ## `DecimalScheme` - `DecimalScheme` gains v1 and v2 modes with `v1()` and `v2()` constructors; `Default` is v1. The session registers v1. - v1 leaves values wider than `i64` canonical after narrowing. v2 splits them with `split_decimal` and compresses the most significant part and each lower part as children. Single-part arrays serialize as v1 in either mode. - `refine` returns v2 when both decimal IDs are allowed and v1 otherwise, from either mode. Unlike #10035's `try_upgrade`, a v2 scheme is refined down to v1 rather than dropped. ## Behaviour | Builder | Wide decimals | |---|---| | `from_session` with the core edition | canonical (v1), unchanged | | `from_session(..).disable_editions()`, including the writer with `disable_editions()` | split (v2) | | `unrestricted()` / `empty()` with the v2 scheme | split (v2) | | `only_cuda_compatible()` | canonical (v1) | Default writes are unchanged: the core edition allows v1 only. Strategies built from `WriteStrategyBuilder::from_session` keep the enabled editions' restrictions even when a write disables editions. Compared with #10035, `vortex-file`'s writer and `vortex-cuda` need no changes: the writer's `disable_editions` path already lifts the edition restriction, and the CUDA restriction lives in the `Cuda` mode. ## Tests - `vortex-btrblocks/src/schemes/decimal.rs`: `refine` for every combination of allowed decimal IDs, from both modes. - `vortex-btrblocks/tests/decimal_config.rs`: decimal output for no editions, the core edition, editions disabled, unrestricted, and the CUDA preset; excluding the decimal scheme is not undone by refinement; signed/nullable `i128`/`i256` round trips with and without child compression. - `vortex-file`: the default writer splits wide decimals only with editions disabled; strategies built from the session keep the enabled editions' restrictions. ## Checks - `cargo clippy -D warnings` on `vortex-btrblocks`, `vortex-compressor`, `vortex-file` with `--all-targets --all-features`, `--all-targets`, and `--no-default-features`: passed - `cargo nextest run --all-features -p vortex-btrblocks -p vortex-compressor -p vortex-file -p vortex -p vortex-layout -p string-bench -p vortex-compat -p vortex-decimal-byte-parts`: 988 passed, 1 skipped - `cargo test --doc -p vortex-btrblocks -p vortex-compressor`: passed - `RUSTDOCFLAGS="-D warnings" cargo doc -p vortex-btrblocks -p vortex-compressor --no-deps --all-features`: passed - `rustfmt +nightly-2026-09-10 --check` on touched files: passed - Not run: `vortex-cuda` (needs `nvcc`), benchmarks, workspace-wide clippy and tests. --------- Signed-off-by: Matt Katz <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #10067.
Summary
CompressionSession::default()now registers every built-in scheme, including Delta and the feature-gated Zstd and Pco schemes. The builder picks which of them to use with a crate-private preset mode, applied inbuild().Registered schemes
The session now also registers:
session.rsZstdScheme, and binaryZstdBuffersScheme(withzstd)PcoScheme(withpco)Each goes last in its type's group, so tie-break order within a type is unchanged. String-level
ZstdBuffersSchemeis still not registered, since no preset used it.Presets
The builder holds one crate-private
CompressionMode.with_compact()andonly_cuda_compatible()now just set it, and the last one set wins:build()Allempty()Defaultfrom_sessionCompactwith_compact()ZstdBuffersCudaonly_cuda_compatible()build()drops the mode's exclusions, then applies theAllowedIdsserialized ID filter from #10067.with_compact()is no longer gated behind thezstdfeature; only the schemes are.with_new_schemeandexclude_schemesare unchanged fromdevelop.Behaviour changes
with_compact().only_cuda_compatible()used to panic on the duplicate binary Zstd scheme; now it gives the CUDA preset.with_compact()andonly_cuda_compatible()on anempty()builder no longer add any schemes, because they only filter.Cudaexcludes it, resolving theTODO(robert)onDELTA_SCHEME. TheDELTA_SCHEMEpublic static is removed: the session owns the only instance, andinteger::DeltaSchemestays public for callers that want a custom ratio. No edition includesfastlanes.delta, so the enabled editions keep it out of edition-enforced writes. With editions disabled orunrestricted(), the compressor may now choose Delta: compat fixtures published with editions disabled, and benchmarks such assplit_collectionand the btrblocks benches, can start using it.with_new_schemeon afrom_sessionbuilder now panics as a duplicate, since Delta is registered. The btrblocks tests that did this drop the call.Other
string-bench's scheme inventory test filters out string Zstd, which is registered but excluded by the default preset.Checks
cargo clippy -p vortex-btrblocks --all-targets --all-features -- -D warningsand--no-default-features: passedcargo nextest run -p vortex-btrblocks --all-features: 73 passed, 1 skippedcargo test --doc -p vortex-btrblocks: 3 passedRUSTDOCFLAGS="-D warnings" cargo doc -p vortex-btrblocks --no-deps --all-features: passedcargo clippy -p vortex-file --all-targetswith--all-features, default, and--no-default-features: passedcargo nextest run -p vortex -p vortex-file -p vortex-layout -p string-bench -p vortex-compat: 493 passedrustfmt +nightly-2026-09-10 --checkon touched files: passed