Skip to content

Filter compressor schemes by the session's allowed serialized IDs on build - #10067

Merged
connortsui20 merged 7 commits into
developfrom
mk/compressor-allowed-ids
Sep 25, 2026
Merged

connortsui20 merged 7 commits into
developfrom
mk/compressor-allowed-ids

Conversation

@mhk197

@mhk197 mhk197 commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #10066.

Summary

BtrBlocksCompressorBuilder now takes the serialized array IDs its schemes may produce directly from the session, and build() keeps only the schemes whose produced_encodings() are all allowed. This is the same rule the file writer uses in new_array_context: registered plugins ∩ enabled editions, or every registered plugin when editions are disabled.

Allowed IDs

A new crate-private AllowedIds holds three independent restrictions:

Field Meaning None / empty
registered: Option<HashSet<ArrayId>> IDs of the session's registered array plugins no restriction
editions: Option<HashSet<ArrayId>> IDs the session's enabled editions permit no restriction
excluded: HashSet<ArrayId> IDs a builder preset excludes (e.g. future CUDA scheme versions) nothing excluded

An ID is allowed when it passes all three. Each builder method touches its own field, so lifting one restriction never undoes another.

Builder API

Method Effect
from_session Session's schemes; registered and editions from the session
empty() No schemes, no restrictions
disable_editions() (new) editions = None, still requires registered plugins. The writer's default strategy calls it when disable_editions is set
unrestricted() (new) registered = None, editions = None; excluded is kept
exclude_schemes Unchanged: removes schemes by SchemeId

Filtering happens in build(), so it also covers schemes added with with_new_scheme / with_compact / only_cuda_compatible.

Removed

  • BtrBlocksCompressorBuilder::retain_allowed_encodings and vortex-bench's retain_edition_encodings. Their callers (writer, cuda, python io, tui, vortex-bench, compress-bench, golden tests) now just use from_session.

Behaviour change

from_session compressors are now restricted on every session, in-memory compression included. A session that doesn't register an encoding, or doesn't enable an edition containing it, won't produce it. A session with no enabled editions gets a compressor with no schemes. Tests that compress on such a session, like the dict layout tests in vortex-layout and editions::tests::explicit_btrblocks_strategy_is_not_reconfigured in vortex, now use unrestricted().

Tests

  • btrblocks unit tests, integration tests and benches that compress with a bare array_session() use unrestricted().
  • The vortex-layout dict layout tests and the vortex editions test for explicit strategies use unrestricted(), so they keep compressing and the editions test still reaches the serialization context check.
  • Golden tests keep their edition filtering by registering the compressor's encodings in edition_session.
  • The CUDA Zstd test chooses the surviving Zstd scheme by enabling core or zstd.
  • New builder tests cover no enabled editions, unregistered encodings, disable_editions, unrestricted, and excluded IDs surviving both.

Other

  • vortex-edition moves from a dev-dependency to a dependency of vortex-btrblocks.

Checks

  • cargo clippy -p vortex-btrblocks --all-targets --all-features -- -D warnings: passed
  • cargo nextest run -p vortex-btrblocks --all-features: 71 passed, 1 skipped
  • cargo nextest run -p vortex-layout layouts::dict::reader::tests: 9 passed
  • cargo nextest run -p vortex editions::tests: 18 passed, 5 skipped
  • cargo clippy -p vortex-layout -p vortex --all-targets --all-features -- -D warnings: passed
  • Not run: doctests, full fmt, and other downstream crates (vortex-file, vortex-cuda, vortex-bench, …); CI covers them.

@mhk197
mhk197 force-pushed the mk/compressor-allowed-ids branch from 09e28d4 to b7a53d4 Compare September 25, 2026 15:34
@codspeed

codspeed Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 20.48%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 3 benchmarks measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. These results are not comparable, so they count as unchanged.

Preventing compiler optimizations

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 5 improved benchmarks
✅ 2123 untouched benchmarks
⏩ 437 skipped benchmarks1
🗄️ 1 archived benchmark run2

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ WallTime mul_u64_nonnull_neon 39.9 µs 29.1 µs +37.38%
⚡ WallTime multiply_shapes_neon[(32768, PerRowPerRow)] 38.6 µs 32.8 µs +17.69%
⚡ WallTime mul_i64_nonnull_neon 38.5 µs 32.8 µs +17.24%
⚡ Simulation sum_v2_i64 224.6 µs 193.9 µs +15.87%
⚡ Simulation sum_i64 223.7 µs 193.6 µs +15.56%
⚠️ Simulation fixed_16_advancing_ptr_safe[100] < 1 ns < 1 ns N/A
⚠️ Simulation preverify_advancing_ptr_unchecked[1000] < 1 ns < 1 ns N/A
⚠️ Simulation preverify_advancing_ptr_unchecked[10000] < 1 ns < 1 ns N/A

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing mk/compressor-allowed-ids (689cf25) with develop (23a4249)3

Open in CodSpeed

Footnotes

  1. 437 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. 1 benchmark was run, but is now archived. If it was deleted in another branch, consider rebasing to remove it from the report. Instead if it was added back, click here to restore it. ↩

  3. No successful run was found on develop (d8e45e0) during the generation of this report, so 23a4249 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@mhk197
mhk197 added this pull request to stack #10068 September 25, 2026 15:42
Comment thread vortex-btrblocks/src/builder.rs Resolved
@mhk197
mhk197 marked this pull request as ready for review September 25, 2026 16:23
@mhk197
mhk197 force-pushed the mk/compressor-allowed-ids branch from 8cea229 to f31e31f Compare September 25, 2026 16:24
@mhk197 mhk197 added the changelog/break A breaking API change label Sep 25, 2026
@mhk197
mhk197 removed this pull request from stack #10068 September 25, 2026 17:05
@mhk197
mhk197 added this pull request to stack #10070 September 25, 2026 17:05
Base automatically changed from mk/compression-session to develop September 25, 2026 17:09
@robert3005
robert3005 force-pushed the mk/compressor-allowed-ids branch from f31e31f to 11abba0 Compare September 25, 2026 17:09
Comment thread vortex-btrblocks/Cargo.toml Resolved
@connortsui20
connortsui20 merged commit 122ca21 into develop Sep 25, 2026
97 checks passed
@connortsui20
connortsui20 deleted the mk/compressor-allowed-ids branch September 25, 2026 20:52
@mhk197 mhk197 mentioned this pull request Sep 25, 2026
2 of 4 tasks
mhk197 added a commit that referenced this pull request Sep 25, 2026
… by builder mode (#10069)

Stacked on #10067.

## Summary

`CompressionSession::default()` now registers every built-in scheme,
including Delta and the feature-gated Zstd and Pco schemes. The builder
picks which of them to use with a crate-private preset mode, applied in
`build()`.

## Registered schemes

The session now also registers:

- Delta, from a private static in `session.rs`
- string and binary `ZstdScheme`, and binary `ZstdBuffersScheme` (with
`zstd`)
- integer and float `PcoScheme` (with `pco`)

Each goes last in its type's group, so tie-break order within a type is
unchanged. String-level `ZstdBuffersScheme` is still not registered,
since no preset used it.

## Presets

The builder holds one crate-private `CompressionMode`. `with_compact()`
and `only_cuda_compatible()` now just set it, and the last one set wins:

| Mode | Set by | Excluded in `build()` |
|---|---|---|
| `All` | `empty()` | nothing |
| `Default` | `from_session` | all Zstd, Pco |
| `Compact` | `with_compact()` | binary `ZstdBuffers` |
| `Cuda` | `only_cuda_compatible()` | the existing CUDA exclusion list,
Delta, Pco, string Zstd |

`build()` drops the mode's exclusions, then applies the `AllowedIds`
serialized ID filter from #10067. `with_compact()` is no longer gated
behind the `zstd` feature; only the schemes are.

`with_new_scheme` and `exclude_schemes` are unchanged from `develop`.

## Behaviour changes

- The order of preset calls no longer matters.
`with_compact().only_cuda_compatible()` used to panic on the duplicate
binary Zstd scheme; now it gives the CUDA preset.
- `with_compact()` and `only_cuda_compatible()` on an `empty()` builder
no longer add any schemes, because they only filter.
- Delta is registered and only `Cuda` excludes it, resolving the
`TODO(robert)` on `DELTA_SCHEME`. The `DELTA_SCHEME` public static is
removed: the session owns the only instance, and `integer::DeltaScheme`
stays public for callers that want a custom ratio. No edition includes
`fastlanes.delta`, so the enabled editions keep it out of
edition-enforced writes. With editions disabled or `unrestricted()`, the
compressor may now choose Delta: compat fixtures published with editions
disabled, and benchmarks such as `split_collection` and the btrblocks
benches, can start using it.
- Adding Delta with `with_new_scheme` on a `from_session` builder now
panics as a duplicate, since Delta is registered. The btrblocks tests
that did this drop the call.

## Other

- `string-bench`'s scheme inventory test filters out string Zstd, which
is registered but excluded by the default preset.

## Checks

- `cargo clippy -p vortex-btrblocks --all-targets --all-features -- -D
warnings` and `--no-default-features`: passed
- `cargo nextest run -p vortex-btrblocks --all-features`: 73 passed, 1
skipped
- `cargo test --doc -p vortex-btrblocks`: 3 passed
- `RUSTDOCFLAGS="-D warnings" cargo doc -p vortex-btrblocks --no-deps
--all-features`: passed
- `cargo clippy -p vortex-file --all-targets` with `--all-features`,
default, and `--no-default-features`: passed
- `cargo nextest run -p vortex -p vortex-file -p vortex-layout -p
string-bench -p vortex-compat`: 493 passed
- `rustfmt +nightly-2026-09-10 --check` on touched files: passed
- Not run: workspace-wide clippy and tests, benchmarks, CUDA e2e; CI
covers them.

---------

Signed-off-by: Matt Katz <[email protected]>
mhk197 added a commit that referenced this pull request Sep 25, 2026
## Summary

Compat fixtures are written with `disable_editions()`, so a fixture can
record an encoding with no read-compatibility guarantee. Once published,
the additive-only manifest rule makes that encoding permanent. This PR
writes fixtures under the default session's enabled editions
(`core2026.08.3`), so a fixture can only emit encodings an edition
already guarantees.

## Changes

- **Gated writes.** `adapter::write_compressed` and
`adapter::write_compressed_to_bytes_with_session` no longer call
`disable_editions()`. `check` regenerates its comparison files through
the same path, so it is gated too.
- **Dataset compressor.** TPC-H and ClickBench fixtures build their
compressor and write strategy from `VortexSession::default()`, the
session the file is written with, instead of from `array_session()` with
`disable_editions()`. Since #10067,
`BtrBlocksCompressorBuilder::from_session` keeps only schemes whose
encodings the session registers and its editions allow. The compressor
therefore can't choose an encoding the gated write would reject, such as
`vortex.patched` under `VORTEX_EXPERIMENTAL_PATCHES`. The regular and
compact variants now share one code path.
- **`decimal_byte_parts_v2.vortex` disabled**, like `delta` and
`patched`. It writes `vortex.decimal_byte_parts.v2`, which no edition
includes yet. It isn't in any published manifest (checked through
0.86.1), so removing it doesn't break the additive-only rule. It comes
back once DBP v2 joins an edition.
- **`DESIGN.md`** notes that new fixtures must use encodings from an
enabled edition.

Every fixture in the 0.86.1 manifest uses `core` encodings. That comes
from reading the code, not from a run, so the published fixture set
should be unchanged.

## Checks

- [x] `cargo clippy -p vortex-compat --all-targets --all-features -- -D
warnings`
- [x] `cargo +nightly-2026-09-10 fmt -p vortex-compat --check`
- [ ] `cargo nextest run -p vortex-compat`
- [ ] `cargo run -p vortex-compat --release -- generate --output
/tmp/compat --exclude clickbench`: confirms every remaining fixture
writes under `core`

---------

Signed-off-by: Matt Katz <[email protected]>
joseph-isaacs added a commit that referenced this pull request Oct 8, 2026
Since #10067, BtrBlocksCompressor::from_session only keeps schemes whose
serialized IDs the session allows, and the bare test session enables no
edition, so the probe never chose Dict and write_dict_layout fell back
to a flat layout. Build the probe unrestricted, as the dict reader
tests do.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012f6YvQVEjuUwc3VEet3Hst
Signed-off-by: Joe Isaacs <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants