Skip to content

Upgrade DecimalScheme to the v2 wire format when the writer permits it - #10035

Closed
mhk197 wants to merge 2 commits into
mk/scheme-registryfrom
mk/decimal-modes
Closed

mhk197 wants to merge 2 commits into
mk/scheme-registryfrom
mk/decimal-modes

Conversation

@mhk197

@mhk197 mhk197 commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #10034.

Summary

Wide i128 and i256 decimals compress into multi-part DecimalByteParts arrays when the writer permits the vortex.decimal_byte_parts.v2 wire format. The registry holds the v1 scheme, which only ever writes the frozen v1 format; permission filtering swaps in the v2 variant wherever the permitted serialized IDs allow it.

Changes

  • vortex-compressor: Scheme::try_upgrade(&self, &HashSet<ArrayId>) -> Option<&'static dyn Scheme> is a new defaulted hook returning a newer variant the permitted IDs support; it must keep the SchemeId and never downgrade. permit_schemes(schemes, &allowed) (the function behind CompressionSessionExt::permit) calls it for each scheme before filtering by declared IDs, and is exported for callers with an explicit permission set.
  • vortex-btrblocks: DEFAULT_SCHEMES registers DecimalScheme::v1(). DecimalScheme gains v1 and v2 modes with v1() and v2() constructors; Default is v1. v1 leaves values wider than i64 canonical after narrowing. v2 splits them with split_decimal and compresses the most significant part and each lower part as children. Single-part arrays serialize as v1 in either mode. try_upgrade returns a static v2 instance when both decimal IDs are permitted; an explicitly listed v2 is dropped, not downgraded, when either ID is missing.
  • vortex-file: with editions disabled the writer now permits every registered serialized ID through permit_schemes, so the upgrade applies there too.
  • vortex-cuda: cuda_compatible_schemes pins the decimal scheme to v1 after permissions are applied, since multi-part arrays have no CUDA decode kernel.

Behavior

  • Default sessions enable the core edition, which permits v1 only, so default writes are unchanged.
  • Sessions that permit v2, or writes with editions disabled, split wide decimals.
  • Strategies built from WriteStrategyBuilder::from_session keep the enabled editions' permissions even when a write disables editions.

Tests

  • vortex-btrblocks/tests/decimal_config.rs: mode selection against the core edition and explicit ID sets, v2 on top of the core edition, explicit modes only upgrade, exclusion is not undone by an upgrade, and signed/nullable i128/i256 round trips with and without child compression.
  • vortex-file: the default writer splits wide decimals only with editions disabled; strategies built from the session keep the enabled editions' permissions.

Validation

Run locally: cargo +nightly-2026-09-10 fmt on the touched crates; cargo clippy --all-targets --all-features -- -D warnings on vortex-compressor, vortex-btrblocks, vortex-file, vortex (clean); cargo nextest run --all-features on vortex-compressor, vortex-btrblocks, vortex-file (296 passed, 1 skipped); doctests on the same three crates (passed). Not run: vortex-cuda, which needs nvcc.

@codspeed

codspeed Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 8.3%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 4 benchmarks measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. These results are not comparable, so they count as unchanged.

Preventing compiler optimizations

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
❌ 4 regressed benchmarks
✅ 2170 untouched benchmarks
⏩ 389 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Simulation take_fsl_f16_force_per_index[256, 10] 38 µs 55.5 µs -31.54%
❌ WallTime filtered_sink_i64_avx2[OneNullInEight] 22.1 µs 26.1 µs -15.21%
❌ WallTime decode_avx512[8192, (Inline, OneNullInEight)] 80.5 µs 93.5 µs -13.87%
❌ Simulation set_indices_vortex_buffer[128] 1.8 µs 2.1 µs -10.73%
⚡ WallTime filtered_sink_i64_avx512[OneNullInEight] 26.3 µs 22.3 µs +17.94%
⚡ WallTime scalar_subtract_neon 13.4 µs 11.9 µs +12.98%
⚠️ Simulation fixed_16_advancing_ptr_safe[100] < 1 ns < 1 ns N/A
⚠️ Simulation preverify_advancing_ptr_unchecked[1000] < 1 ns < 1 ns N/A
⚠️ Simulation preverify_advancing_ptr_unchecked[10000] < 1 ns < 1 ns N/A
⚠️ Simulation bench_compare_sliced_dict_primitive[(3333, 10000)] < 1 ns < 1 ns N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing mk/decimal-modes (9e757a2) with mk/scheme-registry (80af2c5)2

Open in CodSpeed

Footnotes

  1. 389 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on mk/scheme-registry (70b0bfe) during the generation of this report, so 62c3359 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@mhk197 mhk197 closed this Sep 25, 2026
mhk197 added a commit that referenced this pull request Sep 28, 2026
…zed IDs permit it (#10076)

Stacked on #10069.

## Summary

Wide `i128` and `i256` decimals compress into multi-part
`DecimalByteParts` arrays when the compressor's allowed serialized IDs
include `vortex.decimal_byte_parts.v2`. The session registers the v1
decimal scheme, and `build()` refines it to v2 wherever both decimal IDs
are allowed.

This PR sets the stage for versioned schemes.

## `Scheme::refine`

New defaulted hook in `vortex-compressor`:

```rust
fn refine(&self, _allowed: &dyn Fn(&ArrayId) -> bool) -> Option<&'static dyn Scheme> {
    None
}
```

- It returns the variant of the scheme to use for the allowed IDs;
`None` keeps the scheme.
- A variant must keep the scheme's `SchemeId`, and every ID it declares
in `produced_encodings` must still be allowed for it to be used.
- The allowed IDs are a predicate rather than a set, so builders that
allow every ID (`empty()`, `unrestricted()`) are represented exactly.

## Builder

`build()` now:

1. drops the mode's excluded schemes,
2. refines each remaining scheme against the allowed IDs,
3. keeps the schemes whose produced IDs are all allowed.

The allowed IDs are the session's `AllowedIds` minus the mode's excluded
serialized IDs. The `Cuda` mode excludes `vortex.decimal_byte_parts.v2`,
since multi-part arrays have no CUDA decode kernel, so the CUDA preset
keeps v1.

## `DecimalScheme`

- `DecimalScheme` gains v1 and v2 modes with `v1()` and `v2()`
constructors; `Default` is v1. The session registers v1.
- v1 leaves values wider than `i64` canonical after narrowing. v2 splits
them with `split_decimal` and compresses the most significant part and
each lower part as children. Single-part arrays serialize as v1 in
either mode.
- `refine` returns v2 when both decimal IDs are allowed and v1
otherwise, from either mode. Unlike #10035's `try_upgrade`, a v2 scheme
is refined down to v1 rather than dropped.

## Behaviour

| Builder | Wide decimals |
|---|---|
| `from_session` with the core edition | canonical (v1), unchanged |
| `from_session(..).disable_editions()`, including the writer with
`disable_editions()` | split (v2) |
| `unrestricted()` / `empty()` with the v2 scheme | split (v2) |
| `only_cuda_compatible()` | canonical (v1) |

Default writes are unchanged: the core edition allows v1 only.
Strategies built from `WriteStrategyBuilder::from_session` keep the
enabled editions' restrictions even when a write disables editions.

Compared with #10035, `vortex-file`'s writer and `vortex-cuda` need no
changes: the writer's `disable_editions` path already lifts the edition
restriction, and the CUDA restriction lives in the `Cuda` mode.

## Tests

- `vortex-btrblocks/src/schemes/decimal.rs`: `refine` for every
combination of allowed decimal IDs, from both modes.
- `vortex-btrblocks/tests/decimal_config.rs`: decimal output for no
editions, the core edition, editions disabled, unrestricted, and the
CUDA preset; excluding the decimal scheme is not undone by refinement;
signed/nullable `i128`/`i256` round trips with and without child
compression.
- `vortex-file`: the default writer splits wide decimals only with
editions disabled; strategies built from the session keep the enabled
editions' restrictions.

## Checks

- `cargo clippy -D warnings` on `vortex-btrblocks`, `vortex-compressor`,
`vortex-file` with `--all-targets --all-features`, `--all-targets`, and
`--no-default-features`: passed
- `cargo nextest run --all-features -p vortex-btrblocks -p
vortex-compressor -p vortex-file -p vortex -p vortex-layout -p
string-bench -p vortex-compat -p vortex-decimal-byte-parts`: 988 passed,
1 skipped
- `cargo test --doc -p vortex-btrblocks -p vortex-compressor`: passed
- `RUSTDOCFLAGS="-D warnings" cargo doc -p vortex-btrblocks -p
vortex-compressor --no-deps --all-features`: passed
- `rustfmt +nightly-2026-09-10 --check` on touched files: passed
- Not run: `vortex-cuda` (needs `nvcc`), benchmarks, workspace-wide
clippy and tests.

---------

Signed-off-by: Matt Katz <[email protected]>
@robert3005
robert3005 deleted the mk/decimal-modes branch November 10, 2026 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant