Skip to content

Move compression schemes to CompressionSession and construct BtrBlocksCompressorBuilder from Session - #10066

Merged
robert3005 merged 6 commits into
developfrom
mk/compression-session
Sep 25, 2026
Merged

robert3005 merged 6 commits into
developfrom
mk/compression-session

Conversation

@mhk197

@mhk197 mhk197 commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Moves the btrblocks scheme list out of the builder and onto the session, and makes every compressor and write strategy take its schemes from a session.

  • New vortex_btrblocks::CompressionSession session var holds the registered schemes in registration order (the tie-break order). Default registers what was ALL_SCHEMES; CompressionSessionExt::{compression, register_scheme} read and extend it. Registering a duplicate SchemeId panics, matching with_new_scheme.
  • Default is removed from BtrBlocksCompressorBuilder, BtrBlocksCompressor and WriteStrategyBuilder. Each is constructed with from_session(&session) (the builder also keeps empty()). The other builder methods are unchanged.
  • ALL_SCHEMES is removed. DELTA_SCHEME stays.
  • The writer's default strategy builds from the writer's session.
  • Every caller now passes the session it already had in scope (SESSION, session, ctx.session()). Two signatures change: string-bench's serialized_write_strategy now takes the session, and the parquet-variant write_strategy fixture is replaced by building the strategy from the test's session.
  • vortex::compressor re-exports CompressionSession and CompressionSessionExt.

@mhk197 mhk197 changed the title Register compression schemes on a CompressionSession Register compression schemes on a CompressionSession; build compressors from_session Sep 25, 2026
@codspeed

codspeed Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Merging this PR will regress 1 benchmark

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 3 benchmarks measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. These results are not comparable, so they count as unchanged.

Preventing compiler optimizations

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 6 improved benchmarks
❌ 1 regressed benchmark
✅ 2121 untouched benchmarks
⏩ 437 skipped benchmarks1
🗄️ 1 archived benchmark run2

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Simulation filter_powerlaw_by_mostly_true[250000] 102.3 µs 149 µs -31.35%
⚡ Simulation allocate_drop_vortex[64] 66.7 µs 52.1 µs +28.02%
⚡ Simulation allocate_drop_vortex[256] 67.5 µs 53.1 µs +27.1%
⚡ Simulation allocate_drop_vortex[1024] 110.1 µs 91.4 µs +20.45%
⚡ Simulation take[nullable/sorted/primitive/nullable_all_valid/chunks=4/indices=1000] 269.4 µs 231.4 µs +16.42%
⚡ Simulation take[duplicates/duplicate90/primitive/nonnull/chunks=16/indices=1000] 298.2 µs 260.1 µs +14.64%
⚡ Simulation take_fsl_chunked_elements_sorted[16, 64] 252.5 µs 225.5 µs +12%
⚠️ Simulation fixed_16_advancing_ptr_safe[100] < 1 ns < 1 ns N/A
⚠️ Simulation preverify_advancing_ptr_unchecked[1000] < 1 ns < 1 ns N/A
⚠️ Simulation preverify_advancing_ptr_unchecked[10000] < 1 ns < 1 ns N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing mk/compression-session (299f964) with develop (21b3dea)3

Open in CodSpeed

Footnotes

  1. 437 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. 1 benchmark was run, but is now archived. If it was deleted in another branch, consider rebasing to remove it from the report. Instead if it was added back, click here to restore it. ↩

  3. No successful run was found on develop (0fea4a3) during the generation of this report, so 21b3dea was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@mhk197 mhk197 changed the title Register compression schemes on a CompressionSession; build compressors from_session Construct BtrBlocksCompressorBuilder from session Sep 25, 2026
@mhk197 mhk197 changed the title Construct BtrBlocksCompressorBuilder from session Move compression schemes to CompressionSession and construct BtrBlocksCompressorBuilder from Session Sep 25, 2026
Signed-off-by: Matt Katz <[email protected]>
@mhk197
mhk197 marked this pull request as ready for review September 25, 2026 15:38
@mhk197 mhk197 added the changelog/break A breaking API change label Sep 25, 2026
@mhk197
mhk197 added this pull request to stack #10068 September 25, 2026 15:42

@connortsui20 connortsui20 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ship it

@mhk197
mhk197 removed this pull request from stack #10068 September 25, 2026 17:05
@mhk197
mhk197 added this pull request to stack #10070 September 25, 2026 17:05
@robert3005
robert3005 merged commit bc9a579 into develop Sep 25, 2026
126 of 130 checks passed
@robert3005
robert3005 deleted the mk/compression-session branch September 25, 2026 17:09
connortsui20 pushed a commit that referenced this pull request Sep 25, 2026
…build (#10067)

Stacked on #10066.

## Summary

`BtrBlocksCompressorBuilder` now takes the serialized array IDs its
schemes may produce directly from the session, and `build()` keeps only
the schemes whose `produced_encodings()` are all allowed. This is the
same rule the file writer uses in `new_array_context`: registered
plugins ∩ enabled editions, or every registered plugin when editions are
disabled.

## Allowed IDs

A new crate-private `AllowedIds` holds three independent restrictions:

| Field | Meaning | `None` / empty |
|---|---|---|
| `registered: Option<HashSet<ArrayId>>` | IDs of the session's
registered array plugins | no restriction |
| `editions: Option<HashSet<ArrayId>>` | IDs the session's enabled
editions permit | no restriction |
| `excluded: HashSet<ArrayId>` | IDs a builder preset excludes (e.g.
future CUDA scheme versions) | nothing excluded |

An ID is allowed when it passes all three. Each builder method touches
its own field, so lifting one restriction never undoes another.

## Builder API

| Method | Effect |
|---|---|
| `from_session` | Session's schemes; `registered` and `editions` from
the session |
| `empty()` | No schemes, no restrictions |
| `disable_editions()` (new) | `editions = None`, still requires
registered plugins. The writer's default strategy calls it when
`disable_editions` is set |
| `unrestricted()` (new) | `registered = None`, `editions = None`;
`excluded` is kept |
| `exclude_schemes` | Unchanged: removes schemes by `SchemeId` |

Filtering happens in `build()`, so it also covers schemes added with
`with_new_scheme` / `with_compact` / `only_cuda_compatible`.

## Removed

- `BtrBlocksCompressorBuilder::retain_allowed_encodings` and
`vortex-bench`'s `retain_edition_encodings`. Their callers (writer,
cuda, python `io`, tui, vortex-bench, compress-bench, golden tests) now
just use `from_session`.

## Behaviour change

`from_session` compressors are now restricted on every session,
in-memory compression included. A session that doesn't register an
encoding, or doesn't enable an edition containing it, won't produce it.
A session with no enabled editions gets a compressor with no schemes.
Tests that compress on such a session, like the dict layout tests in
`vortex-layout` and
`editions::tests::explicit_btrblocks_strategy_is_not_reconfigured` in
`vortex`, now use `unrestricted()`.

## Tests

- btrblocks unit tests, integration tests and benches that compress with
a bare `array_session()` use `unrestricted()`.
- The `vortex-layout` dict layout tests and the `vortex` editions test
for explicit strategies use `unrestricted()`, so they keep compressing
and the editions test still reaches the serialization context check.
- Golden tests keep their edition filtering by registering the
compressor's encodings in `edition_session`.
- The CUDA Zstd test chooses the surviving Zstd scheme by enabling
`core` or `zstd`.
- New builder tests cover no enabled editions, unregistered encodings,
`disable_editions`, `unrestricted`, and excluded IDs surviving both.

## Other

- `vortex-edition` moves from a dev-dependency to a dependency of
`vortex-btrblocks`.

## Checks

- `cargo clippy -p vortex-btrblocks --all-targets --all-features -- -D
warnings`: passed
- `cargo nextest run -p vortex-btrblocks --all-features`: 71 passed, 1
skipped
- `cargo nextest run -p vortex-layout layouts::dict::reader::tests`: 9
passed
- `cargo nextest run -p vortex editions::tests`: 18 passed, 5 skipped
- `cargo clippy -p vortex-layout -p vortex --all-targets --all-features
-- -D warnings`: passed
- Not run: doctests, full fmt, and other downstream crates
(`vortex-file`, `vortex-cuda`, `vortex-bench`, …); CI covers them.

---------

Signed-off-by: Matt Katz <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants