Skip to content

Register Delta, Zstd and Pco in the compression session, exclude them by builder mode - #10069

Merged
mhk197 merged 7 commits into
developfrom
mk/compressor-scheme-modes
Sep 25, 2026
Merged

mhk197 merged 7 commits into
developfrom
mk/compressor-scheme-modes

Conversation

@mhk197

@mhk197 mhk197 commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #10067.

Summary

CompressionSession::default() now registers every built-in scheme, including Delta and the feature-gated Zstd and Pco schemes. The builder picks which of them to use with a crate-private preset mode, applied in build().

Registered schemes

The session now also registers:

  • Delta, from a private static in session.rs
  • string and binary ZstdScheme, and binary ZstdBuffersScheme (with zstd)
  • integer and float PcoScheme (with pco)

Each goes last in its type's group, so tie-break order within a type is unchanged. String-level ZstdBuffersScheme is still not registered, since no preset used it.

Presets

The builder holds one crate-private CompressionMode. with_compact() and only_cuda_compatible() now just set it, and the last one set wins:

Mode Set by Excluded in build()
All empty() nothing
Default from_session all Zstd, Pco
Compact with_compact() binary ZstdBuffers
Cuda only_cuda_compatible() the existing CUDA exclusion list, Delta, Pco, string Zstd

build() drops the mode's exclusions, then applies the AllowedIds serialized ID filter from #10067. with_compact() is no longer gated behind the zstd feature; only the schemes are.

with_new_scheme and exclude_schemes are unchanged from develop.

Behaviour changes

  • The order of preset calls no longer matters. with_compact().only_cuda_compatible() used to panic on the duplicate binary Zstd scheme; now it gives the CUDA preset.
  • with_compact() and only_cuda_compatible() on an empty() builder no longer add any schemes, because they only filter.
  • Delta is registered and only Cuda excludes it, resolving the TODO(robert) on DELTA_SCHEME. The DELTA_SCHEME public static is removed: the session owns the only instance, and integer::DeltaScheme stays public for callers that want a custom ratio. No edition includes fastlanes.delta, so the enabled editions keep it out of edition-enforced writes. With editions disabled or unrestricted(), the compressor may now choose Delta: compat fixtures published with editions disabled, and benchmarks such as split_collection and the btrblocks benches, can start using it.
  • Adding Delta with with_new_scheme on a from_session builder now panics as a duplicate, since Delta is registered. The btrblocks tests that did this drop the call.

Other

  • string-bench's scheme inventory test filters out string Zstd, which is registered but excluded by the default preset.

Checks

  • cargo clippy -p vortex-btrblocks --all-targets --all-features -- -D warnings and --no-default-features: passed
  • cargo nextest run -p vortex-btrblocks --all-features: 73 passed, 1 skipped
  • cargo test --doc -p vortex-btrblocks: 3 passed
  • RUSTDOCFLAGS="-D warnings" cargo doc -p vortex-btrblocks --no-deps --all-features: passed
  • cargo clippy -p vortex-file --all-targets with --all-features, default, and --no-default-features: passed
  • cargo nextest run -p vortex -p vortex-file -p vortex-layout -p string-bench -p vortex-compat: 493 passed
  • rustfmt +nightly-2026-09-10 --check on touched files: passed
  • Not run: workspace-wide clippy and tests, benchmarks, CUDA e2e; CI covers them.

@codspeed

codspeed Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 2062 untouched benchmarks
⏩ 503 skipped benchmarks1


Comparing mk/compressor-scheme-modes (940ee84) with develop (122ca21)

Open in CodSpeed

Footnotes

  1. 503 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@mhk197
mhk197 added this pull request to stack #10070 September 25, 2026 17:05
@robert3005
robert3005 force-pushed the mk/compressor-scheme-modes branch from 963ed9b to f97c93b Compare September 25, 2026 17:09
@mhk197
mhk197 force-pushed the mk/compressor-scheme-modes branch from f97c93b to e47cc71 Compare September 25, 2026 19:23
@mhk197
mhk197 marked this pull request as ready for review September 25, 2026 19:41
@mhk197 mhk197 added the changelog/break A breaking API change label Sep 25, 2026
@mhk197
mhk197 requested review from connortsui20, joseph-isaacs and robert3005 and removed request for connortsui20 and joseph-isaacs September 25, 2026 20:15
Base automatically changed from mk/compressor-allowed-ids to develop September 25, 2026 20:52
@connortsui20
connortsui20 force-pushed the mk/compressor-scheme-modes branch from 5fa660d to 8e2ebad Compare September 25, 2026 20:52
Comment thread vortex-btrblocks/src/builder.rs Resolved
@mhk197
mhk197 merged commit eb91521 into develop Sep 25, 2026
93 checks passed
@mhk197
mhk197 deleted the mk/compressor-scheme-modes branch September 25, 2026 21:20
mhk197 added a commit that referenced this pull request Sep 28, 2026
…zed IDs permit it (#10076)

Stacked on #10069.

## Summary

Wide `i128` and `i256` decimals compress into multi-part
`DecimalByteParts` arrays when the compressor's allowed serialized IDs
include `vortex.decimal_byte_parts.v2`. The session registers the v1
decimal scheme, and `build()` refines it to v2 wherever both decimal IDs
are allowed.

This PR sets the stage for versioned schemes.

## `Scheme::refine`

New defaulted hook in `vortex-compressor`:

```rust
fn refine(&self, _allowed: &dyn Fn(&ArrayId) -> bool) -> Option<&'static dyn Scheme> {
    None
}
```

- It returns the variant of the scheme to use for the allowed IDs;
`None` keeps the scheme.
- A variant must keep the scheme's `SchemeId`, and every ID it declares
in `produced_encodings` must still be allowed for it to be used.
- The allowed IDs are a predicate rather than a set, so builders that
allow every ID (`empty()`, `unrestricted()`) are represented exactly.

## Builder

`build()` now:

1. drops the mode's excluded schemes,
2. refines each remaining scheme against the allowed IDs,
3. keeps the schemes whose produced IDs are all allowed.

The allowed IDs are the session's `AllowedIds` minus the mode's excluded
serialized IDs. The `Cuda` mode excludes `vortex.decimal_byte_parts.v2`,
since multi-part arrays have no CUDA decode kernel, so the CUDA preset
keeps v1.

## `DecimalScheme`

- `DecimalScheme` gains v1 and v2 modes with `v1()` and `v2()`
constructors; `Default` is v1. The session registers v1.
- v1 leaves values wider than `i64` canonical after narrowing. v2 splits
them with `split_decimal` and compresses the most significant part and
each lower part as children. Single-part arrays serialize as v1 in
either mode.
- `refine` returns v2 when both decimal IDs are allowed and v1
otherwise, from either mode. Unlike #10035's `try_upgrade`, a v2 scheme
is refined down to v1 rather than dropped.

## Behaviour

| Builder | Wide decimals |
|---|---|
| `from_session` with the core edition | canonical (v1), unchanged |
| `from_session(..).disable_editions()`, including the writer with
`disable_editions()` | split (v2) |
| `unrestricted()` / `empty()` with the v2 scheme | split (v2) |
| `only_cuda_compatible()` | canonical (v1) |

Default writes are unchanged: the core edition allows v1 only.
Strategies built from `WriteStrategyBuilder::from_session` keep the
enabled editions' restrictions even when a write disables editions.

Compared with #10035, `vortex-file`'s writer and `vortex-cuda` need no
changes: the writer's `disable_editions` path already lifts the edition
restriction, and the CUDA restriction lives in the `Cuda` mode.

## Tests

- `vortex-btrblocks/src/schemes/decimal.rs`: `refine` for every
combination of allowed decimal IDs, from both modes.
- `vortex-btrblocks/tests/decimal_config.rs`: decimal output for no
editions, the core edition, editions disabled, unrestricted, and the
CUDA preset; excluding the decimal scheme is not undone by refinement;
signed/nullable `i128`/`i256` round trips with and without child
compression.
- `vortex-file`: the default writer splits wide decimals only with
editions disabled; strategies built from the session keep the enabled
editions' restrictions.

## Checks

- `cargo clippy -D warnings` on `vortex-btrblocks`, `vortex-compressor`,
`vortex-file` with `--all-targets --all-features`, `--all-targets`, and
`--no-default-features`: passed
- `cargo nextest run --all-features -p vortex-btrblocks -p
vortex-compressor -p vortex-file -p vortex -p vortex-layout -p
string-bench -p vortex-compat -p vortex-decimal-byte-parts`: 988 passed,
1 skipped
- `cargo test --doc -p vortex-btrblocks -p vortex-compressor`: passed
- `RUSTDOCFLAGS="-D warnings" cargo doc -p vortex-btrblocks -p
vortex-compressor --no-deps --all-features`: passed
- `rustfmt +nightly-2026-09-10 --check` on touched files: passed
- Not run: `vortex-cuda` (needs `nvcc`), benchmarks, workspace-wide
clippy and tests.

---------

Signed-off-by: Matt Katz <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants