Skip to content

Commit 42bea5d

Browse files
connortsui20claude
andauthored
Add golden-corpus determinism tests for the default compressor (#8697)
Tracking Issue: #8701 Closes: #6171 ## Rationale for this change Groundwork for making the compressor's selection policy pluggable (#7697): before any refactoring of scheme selection, pin the compressor's current decisions with reviewable golden snapshots, so each subsequent PR can demonstrate unchanged selection behavior by showing zero snapshot churn. Test-only, and lands first so later PRs diff against a committed baseline. This also closes #6171 by adding dedicated regression coverage for whether the compressor produces the expected array tree, across a representative corpus rather than one-off encoding assertions. ## What changes are included in this PR? A golden test suite in `vortex-btrblocks`: 20 seed-generated corpus entries spanning the registered schemes' habitats, each compressed twice per run (direct determinism check) and snapshotted as its full encoding tree with exact array-level byte counts. Three feature variants cover the default scheme pool, `unstable_encodings`, and `with_compact` (+`zstd`+`pco`). A dedicated `btrblocks-golden` CI job runs all three feature combinations explicitly. The variants are mutually exclusive at compile time (`unstable_encodings` changes `ALL_SCHEMES`), so the existing `--all-features` workspace test jobs compile the `default` variant out, and it would otherwise only run incidentally in the default-features musl job. OnPair is excluded from the golden compressors: its dictionary training is nondeterministic run-to-run (randomly-seeded hash maps in the upstream crate), so it cannot serve as a baseline. That's a pre-existing bug worth its own issue. ## What APIs are changed? Are there any user-facing changes? None (tests and CI only). --------- Signed-off-by: Connor Tsui <[email protected]> Co-authored-by: Claude <[email protected]>
1 parent a530b3b commit 42bea5d

65 files changed

Lines changed: 1170 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -452,6 +452,38 @@ jobs:
452452
alert-title: "Rust tests (linux-arm64) failed on develop"
453453
deduplication-key: ci-rust-test-linux-arm64-failure
454454

455+
btrblocks-golden:
456+
name: "Rust (btrblocks golden corpus)"
457+
timeout-minutes: 10
458+
runs-on: >-
459+
${{ github.repository == 'vortex-data/vortex'
460+
&& format('runs-on={0}/runner=amd64-medium/image=ubuntu24-full-x64-pre-v2/extras=s3-cache/tag=btrblocks-golden', github.run_id)
461+
|| 'ubuntu-latest' }}
462+
steps:
463+
- uses: runs-on/action@v2
464+
if: github.repository == 'vortex-data/vortex'
465+
with:
466+
sccache: s3
467+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
468+
- uses: ./.github/actions/setup-prebuild
469+
with:
470+
enable-sccache: "true"
471+
# The golden suite pins the compressor's decisions per feature variant, and the
472+
# variants are mutually exclusive at compile time: the `default` variant only exists
473+
# without `unstable_encodings` (which changes ALL_SCHEMES), so the `--all-features`
474+
# workspace test jobs cannot run it. Run each feature combination explicitly.
475+
- name: Golden corpus (default features)
476+
run: |
477+
cargo nextest run --cargo-profile ci --locked --no-fail-fast -p vortex-btrblocks --test golden
478+
- name: Golden corpus (unstable_encodings)
479+
run: |
480+
cargo nextest run --cargo-profile ci --locked --no-fail-fast -p vortex-btrblocks --test golden \
481+
--features unstable_encodings
482+
- name: Golden corpus (compact, unstable_encodings + zstd + pco)
483+
run: |
484+
cargo nextest run --cargo-profile ci --locked --no-fail-fast -p vortex-btrblocks --test golden \
485+
--features unstable_encodings,zstd,pco
486+
455487
build-java:
456488
name: "Java"
457489
runs-on: >-

‎Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎vortex-btrblocks/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ vortex-zstd = { workspace = true, optional = true }
4141

4242
[dev-dependencies]
4343
divan = { workspace = true }
44+
insta = { workspace = true }
4445
rstest = { workspace = true }
4546
test-with = { workspace = true }
4647
vortex-array = { workspace = true, features = ["_test-harness"] }

‎vortex-btrblocks/REUSE.toml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
version = 1
2+
3+
# `insta` snapshot files do not allow leading comment lines.
4+
[[annotations]]
5+
path = "tests/snapshots/**.snap"
6+
SPDX-FileCopyrightText = "Copyright the Vortex contributors"
7+
SPDX-License-Identifier = "Apache-2.0"

0 commit comments

Comments
 (0)