Repository navigation
build(deps): pin vortex to a patched fork for the #468 and #472 determinism fixes - #481
Merged
Merged
Conversation
…hid on vortex 0.75.0 writes different container bytes for an integer column whose encoding produces patches, depending on whether `debug_assertions` is enabled. `[i64::MIN, 0, MAX]` is the minimal case: 2796 bytes with them on, 2772 with them off, a constant 24-byte delta. The compression is identical in both — same chosen schemes (`zigzag -> bitpacking`), same `compressed_nbytes`, same estimated and achieved ratios — so the difference is in the file metadata written around the values. The cause is an **assertion with a side effect**. `vortex-array/src/patches.rs` guards patch construction with a `#[cfg(debug_assertions)]` `is_sorted(&indices, …)` check; evaluating that aggregate executes and annotates the indices array, and the written container reflects it. Measured, not reasoned: toggling `debug-assertions` alone flips the bytes in both directions, while `overflow-checks` and the optimisation level change nothing — so this is neither UB nor codegen. Only patch-producing inputs diverge: `[i64::MIN, i64::MAX]`, which is byte-identical across profiles, produces no patch arrays at all. Three additions, and the middle one is the point: - `known_limitation_468_full_span_int_container_bytes` reproduces it through **vortex APIs only**, with the stock `WriteStrategyBuilder::default()` — no tessera types, no `exclude_schemes`, no Pco — so it cannot be mistaken for something our compressor configuration causes. It asserts the bug is STILL PRESENT and therefore **fails when upstream fixes it**, which is the signal to restore the `i64` extremes that #460's fixture currently gives up. A `#[ignore]` would let the fix land unnoticed. - `seal-profile-determinism` regenerates the conformance corpus under BOTH profiles and requires them to agree with each other and with the committed file. Worth stating plainly: the committed `corpus/corpus.json` was produced by a bare `cargo run` (dev profile) while `workspace-test` runs the same code release-built, so the two halves of our own CI were checking different builds and nothing compared them. - ADR-0057 §5 records the lesson, which is sharper than "an exclusion list is not a proof": **a determinism argument about the codec does not cover the container.** The ALP/ALPRD exclusion's reasoning is sound and is about the encoding of *values*; this defect has identical codec decisions and differs in the metadata around them, and the seal covers the whole container. Blast radius, stated honestly: the axis is `debug_assertions`, so every release build agrees with every other release build on any architecture, and shipped binaries are mutually consistent. What diverges is a debug-built tool against a release-built one. It is still an S15 violation, because "the bytes are a function of the data" has to hold for *a* build of a version, not only for the release ones. `seal-profile-determinism` is verified to evaluate (`nix eval …drvPath`) but NOT built locally: `/` is at 97%, and a two-profile derivation in one sandbox is what exhausted it before. CI has its own disk. Refs: #468
The third determinism lesson from the same sitting, and the one most likely to recur, so it
belongs next to the other two in ADR-0057 §5 rather than only in the commit that fixed it.
The `.tsra` STORED invariant — what makes a block's bytes fetchable by range, and therefore
what makes a cloud reader possible at all — was documented and written but never *checked* on
read. It did not need to be: the workspace pinned `zip = { default-features = false }`, so no
decompressor existed and a compressed member simply failed to decode. Adding `deflate-flate2`
for the `.npz` reader removes that accident workspace-wide, because cargo unifies features, and
a deflated `.tsra` would then have **verified perfectly** while silently ceasing to be
range-readable.
Stated as a rule rather than a war story: an invariant that holds because some capability is
*missing* stops holding the moment anyone adds that capability for an unrelated reason, and
nothing in the diff will mention it.
Also records what the two determinism defects have in common, since it explains why the
existing exclusion list could not have caught either: #468 and #472 both diverge in the metadata
*around* the values — a container record and a persisted `Stat::Sum` — while the encoded values
are identical. `deterministic_table_compressor` reasons about the codec; the seal covers the
whole container.
Refs: #468
Keeping this PR current. `dev` brought #445 (devkit 1.17.0), #473 (the #463 fix — stacked PRs now get the PR checks), #470, #465 (the encode-trace callsite race), #464, #469 and #471. Clean merge; only `flake.nix` needed auto-merging, alongside the `seal-profile-determinism` check this branch adds. #465 touches `tessera-io`, which is where this branch's reproducer lives, so the thing worth checking was whether the #468 divergence still behaves: `known_limitation_468_full_span_int_container_bytes` still passes under **both** profiles on the merge, so the known byte counts are unchanged and the test still fails-when-fixed as intended. Refs: #468
…claim to this defect Review found two statements that read as broader than the evidence, both of which would have merged onto `dev` as claims this PR cannot support. THE RELEASE-BUILD CLAIM CONTRADICTED ITS OWN PARAGRAPH. §5 said "every release build agrees with every other release build, on any architecture" — three paragraphs above a note that #472 is a **same-profile** divergence between x86-64 and aarch64. Both cannot be true. The claim is really about #468's axis only, so it now says so explicitly, and says outright that it is not a general statement, naming #472 as the counter-example. Two release binaries really can disagree; just not because of this defect. THE STORED CHECK IS NOT IN THIS PR. The "guarantee that depends on an absence" paragraph said the invariant is "enforced now", but the check lands with the `.npz` lane in #461 — this branch is based on `dev` and adds no `zip` feature, so on `dev` at this PR's merge point nothing is enforced yet. Reworded to attribute it to #461, which is where the decompressor and the replacement arrive together. The reproducer's own message and doc comment had the same problem from the other direction: they pointed at `tessera_ingest::corpus::write_scalars`, a function that exists only on the ingest branches, so a reader on `dev` would be sent to something that is not there. Both now describe the fixture neutrally and stay true wherever they are read. TWO SCOPE NOTES, so the gate is not read as doing more than it does: - `tessera-io`'s conformance corpus contains no integer column that produces bitpacking **patches**, so `seal-profile-determinism` does not currently reproduce #468. It guards the class going forward; the specific shape is pinned by the reproducer. §5 now says this, along with what would change it (a patch-producing fixture, once upstream is fixed and the expected bytes stop depending on the build). - The check's cost is documented: `cargoArtifacts` pre-builds the **release** dependency graph, so its dev half compiles vortex from source every run, which is the bulk of the leg. A second dev-profile `buildDepsOnly` is not cleanly expressible — a derivation inherits one `cargoArtifacts` — so it is left as a stated cost with the cheaper fallback named (release-only on PRs, both profiles on `main`) rather than a speculative nix refactor. Refs: #468
…minism fixes vortex 0.75.0 has two defects that both leak a WRITE-TIME STATISTIC into the serialised bytes, so a sealed `content_hash` stops being a function of the data alone: #468 the `debug_assertions`-gated `is_sorted` assert on patch indices calls `cache_is_sorted`, which SETS `Stat::IsSorted` on them. Cached stats are serialised, so a debug-built writer emitted 24 bytes more than a release-built one. #472 `sum_float_all` skips NaN, so a column with both `+inf` and `-inf` evaluates `inf + -inf`. IEEE leaves that NaN's bits unspecified and the targets disagree (x86_64 `0xfff8…`, aarch64 `0x7ff8…`); `Stat::Sum` is persisted, so the bytes depended on the writing host. Both are fixed on a fork of the 0.75.0 tag and pinned here by FULL COMMIT SHA until a release carries them. #480 tracks removing the pin. Two non-obvious requirements, both of which fail SILENTLY: - all 27 vortex crates must be patched together. Patching `vortex-array` alone leaves its siblings on crates.io, giving two copies of `vortex-proto`/`vortex-buffer` and ~1000 type errors. - the 0.75.0 tag ships the placeholder version 0.1.0 (the real one is injected at publish), so without a version-setting commit on the fork branch cargo reports `patch ... was not used in the crate graph` and keeps using crates.io. An ignored patch looks exactly like a patch that changed nothing, so the pin was verified POSITIVELY: on x86_64 the sealed float bytes no longer contain fff8000000000000, now contain 7ff8000000000000, and became identical to what unpatched aarch64 produced. #474's `known_limitation_468_*` asserted the bug was still present, so it fails once fixed — by design. It is flipped here into the positive guard: `full_span_int_container_bytes_are_build_config_independent`, with ONE expected length per shape and no `cfg!(debug_assertions)` branch. That absence IS the assertion; flake.nix runs it under both profiles, so a re-divergence fails in exactly one of the two lines. Both profiles agree on the former release numbers (bare 2772, sealed 2916) — the debug build was the one carrying the extra 24 bytes. Gate B moves, and only in one way: 458 insertions against 458 deletions across 8 snapshots, every changed line identical apart from gaining the `(https://github.com/gerchowl/vortex?rev=…)` source tag. No feature changed anywhere — verified by re-comparing with that tag stripped. blake3, zarrs and zstd are untouched. cargo-deny `[sources]` is tightened rather than left permissive: `unknown-git` goes from `allow` to `deny` with `allow-git` naming exactly the pinned repository, so a future stray git dependency is a review event instead of silently fine. Verified: crane vendors the git dependency hermetically with no `outputHashes` needed; `gen_corpus` output is byte-identical to the committed corpus AND agrees between dev and release; all 3 conformance tests pass; every `DECODE_PATH_CRATES` version is unchanged, so the `ingest_decoder` digest cannot move. Refs: #468, #472, #474, #480
Contributor
Author
Scope correction on "no golden moved", plus the ingest-lane verificationClarifying my own claim before it misleads anyone. The table above says "no golden moved", and that tsra-386 regenerated Two things make that the right movement rather than a surprise:
The regenerated ingest corpus is not carried here — this branch has no ingest corpus to update. It
|
| defect | issue | PR |
|---|---|---|
| #468 | vortex-data/vortex#10119 | vortex-data/vortex#10121 |
| #472 | vortex-data/vortex#10120 | vortex-data/vortex#10122 |
Both were rebased onto vortex's develop rather than the 0.75.0 tag — 23 commits had touched those
files since, including an AggregateVTable rework and a new sum_v2 that has its own float finalisation
site, so #10122 fixes two sites there. Both defects were re-verified as still present on develop
before filing. The tessera pin itself stays on the 0.75.0-based fork branch, which is unaffected.
This was referenced Sep 29, 2026
…via the ingest golden #472's fix had no direct guard here: the only thing that would have caught an early pin removal was #460's ingest corpus, on another branch. Dropping the pin before the upstream fix ships would have gone unnoticed in this crate. Seals an f64 column holding both `+inf` and `-inf` — the shape that makes vortex's `Sum` statistic evaluate `inf + -inf` — and asserts the sealed bytes carry the canonical quiet NaN and NOT x86_64's `0xfff8_0000_0000_0000`. Added to the same flake.nix both-profiles gate as the #468 guard. Proven to bite: it FAILS with the `[patch.crates-io]` block removed and passes with it, so it is a real guard and not a tautology. Two honest limits, both stated on the test: - it cannot fire on aarch64, which always produced the canonical value, so the sealed bytes there are identical with and without the fix. The guard bites on x86_64 — the host whose bytes were wrong. A green aarch64 leg is not evidence against a dropped pin. - it asserts byte PATTERNS rather than a digest, following the #468 guard's reasoning: a vortex bump legitimately moves bytes, so the test should fire for a determinism regression and not for every dependency update. Refs: #472, #480
gerchowl
changed the base branch from
fix/468-int-full-span-container-determinism
to
dev
September 29, 2026 01:02
#474 landed as a squash (38c9efa), so this branch's copies of its original commits conflicted with it. Resolved by taking this branch's versions of flake.nix and tessera/crates/tessera-io/src/table.rs: this branch is a strict superset of #474's content there — verified by diffing both files against origin/dev, whose only differences were exactly the two intended changes (the #468 test flipped to `full_span_int_container_bytes_are_build_config_independent` and the new #472 canonical-NaN guard, plus the matching flake.nix invocations). No dev-only content exists in either conflicted region, so nothing from the squash is dropped. Refs: #481
gerchowl
added a commit
that referenced
this pull request
Sep 29, 2026
… are fixed `dev` now pins vortex to a patched fork (#481), which fixes both determinism defects this branch was waiting on. What that does to the corpus is the substance of this commit, so it is worth stating as a result rather than a claim. **One fixture moves, and to a value predicted in advance.** `ingest_parquet_floats`: `2ee5bf4d1b33…` → `33b065510f5e…`. That second value is not new — it is what **aarch64 was already producing** before the pin, taken from this PR's own failing aarch64 log. #472 was a persisted `Stat::Sum` NaN: `sum_float_all` skips NaN, so a column holding `+inf` and `-inf` summed to a NaN whose sign bit differed by architecture (x86-64 `0xfff8…`, aarch64 `0x7ff8…`), one bit. Canonicalising to `0x7ff8_0000_0000_0000` therefore has to leave aarch64 untouched and move x86-64 onto it — so predicting the value beforehand and hitting it distinguishes "the fix changed the bytes" from "the fix made x86 agree with ARM", and only the second is consistent with that diagnosis. Everything else is byte-identical and no `ingest_decoder` triple moved. **And the corpus is now profile-independent**, which is #468's half: the regeneration was written only after confirming the dev-profile and release-profile outputs are byte-identical, the property ADR-0057 §5's `seal-profile-determinism` check exists to enforce and the one #468 made false. `ingest_parquet_scalars` does **not** move, because it was narrowed to ±(2^53+1) earlier on this branch — profile-stable by construction, so the pin has nothing to change there. GATE B — the three conflicted snapshots were **regenerated, not hand-merged.** Both sides legitimately changed them (the pin adds the fork's source tags; this branch adds the `parquet` node's edges), and a generated file that has been hand-merged looks authoritative without being generated. Verified afterwards rather than assumed: no **pre-existing** third-party crate gains a feature — every gain belongs to the new `parquet` node — nothing is lost, and both watched canaries are unchanged (`vortex-btrblocks/alp` still absent, which is the one that would silently re-encode every float column; `arrow-array/chrono-tz` unchanged). ADR-0057 §5 is rewritten for the post-pin state: both defects fixed via the fork, with upstream `vortex-data/vortex` #10119 / PR #10121 and #10120 / PR #10122, and #480 tracking removal of the pin; the agreed byte counts under the pin (2772 bare, 2916 sealed, in **both** profiles — the debug build was the one carrying the extra 24 bytes of cached statistic); #472's one-bit cause recorded beside #468's, since the pair is what the section argues from; and the note that a patch-producing integer fixture is now **safe** to add to `tessera-io`'s corpus, because the expected bytes no longer depend on the build — worth doing while #480 is open, as it is exactly what would catch a regression when the pin is dropped. The reproducer that guarded the shape has been inverted upstream of here (#481): it asserted the divergence was *still present*, so it failed the moment the fix landed. That failure is what forced this section to be rewritten instead of going quietly stale. One more thing the merge surfaced, which is a gate earning its keep: `dev` brought a **generated** schema reference (#484) that is asserted against the live registry, and this branch adds the `table` and `array` builtin schemas — so the committed copy went stale the instant the merge happened, and `schema_reference_matches_the_committed_copy` failed. Regenerated: +28 lines documenting exactly those two schemas, nothing removed. Neither branch could have caught it alone, which is the argument for asserting derived docs against their source rather than reviewing them. Refs: #386
gerchowl
added a commit
that referenced
this pull request
Sep 29, 2026
`nix flake check` stops at the first failing derivation, so on #481 an unrelated `minio` substitution failure took down all 17 aarch64 checks: the leg went red with every check reporting `(cancelled)` and nothing naming a real failure. The determinism gates, clippy and the wasm checks do not depend on minio at all. Two reviewers independently had to read the job log to establish that the pin under test was not implicated — the cost a signal imposes when a cancelled check and a failed check look identical from the PR list. Three changes, one per problem. `--keep-going`: a failure no longer cancels the other checks, so every check reaches a verdict and the failing one is named. `--fallback`: a derivation that cannot be substituted is built from source rather than failing the run. Measured on these exact runner images for the derivation that actually bit us: minio, from source x86_64 102s aarch64 73s minio, substituted x86_64 13s aarch64 80s On aarch64 building it is FASTER than pulling its 36 MiB from cache.nixos.org — the download is the slow part, not the Go build. Against the leg's existing 90-minute cap, noise. The flag applies to every derivation, so a pathological outage on something large could build until that cap instead of failing in seconds; accepted deliberately, because both outcomes are a red leg, the budget already bounds it, and the common case stops being a spurious failure. A per-check summary table, derived from the STORE rather than by parsing nix's human output: a check whose output path is valid was built and so passed. `--keep-going` is what makes that inference sound — every check is attempted, so a missing output means that check failed rather than that it was skipped. Both directions were verified locally (a built check reads as passed, an unbuilt one as not built) rather than assumed. `nix flake check` stays the command. A per-check `nix build` loop would give exact verdicts and keep-going for free, but this file's header argues that the same command running on a developer's machine and in CI is what prevents "passes locally / fails in CI" drift, and that property is worth more than tidier reporting. Also documents, in flake.nix at the point the decision is made, why the minio-range-read check strips minio's `knownVulnerabilities`: a sandboxed, network-less, throwaway test fixture, reached only over loopback by the test itself, where the alternative is having no test of the cloud range-read path against a real object store. `meta` is not part of the derivation, so the override changes no output hash. Refs: #483
gerchowl
added a commit
that referenced
this pull request
Sep 29, 2026
…C|CSV> (#460) The table lane of ADR-0056: Parquet, Arrow IPC and CSV (explicit schema only, locale-independent parsing) normalised into Tessera table primitives, with nested-nullability composed at every level, the ingest_decoder recipe triple sealed per ADR-0056 §6a (hand-written triples rejected for every backend), the §7 laundering rule, new sealed fields (Column.referencing, Column.sensitivity incl. Unknown, manifest ingest_transform) with the existing corpus byte-identical, a three-producer (pyarrow/polars/DuckDB → one content_hash) nix check, and the ADR-0056 §12a amendment. The ADR-0057 §5 note is updated for the vortex pin (#481). Refs: #386
This was referenced Sep 29, 2026
gerchowl
added a commit
that referenced
this pull request
Oct 5, 2026
…543) Refs #513 — clears Dependabot **#42, #43 and #44**. The fourth (`thrift`, #45) is blocked upstream; evidence at the bottom. ## Why a manual bump was needed `dicom-transfer-syntax-registry` 0.9.1 declares `jxl-oxide = "^0.10.2"`, and the patched 0.12.6 is outside that range, so neither bot could fix it in place. 0.10.0 requires `^0.12.6`, so the family moves together: | crate | before | after | fixed in | advisory | |---|---|---|---|---| | `jxl-grid` | 0.5.3 | **0.6.2** | 0.6.2 | GHSA-5pmv-rx8r-wmv5 (high) | | `jxl-oxide` | 0.10.2 | **0.12.6** | 0.12.6 | GHSA-66m8-c62j-h6v5 | | `jxl-modular` | 0.9.1 | **0.11.3** | 0.11.3 | GHSA-2v8p-fqpx-2q3w | 24 lockfile versions move in total (ten `dicom-*`, twelve `jxl-*`, and `snafu` 0.8.9 → 0.9.2 which dicom 0.10 requires); one package added (`jxl-jbr`), none removed. **No workspace source change was needed** — the 0.9 → 0.10 API break does not reach our call sites. Context worth keeping: we do not enable `jpegxl`, so the jxl crates are **lockfile-only** and were never in the build graph. The advisories were real as lockfile facts and inert as runtime ones — hence `chore`, not `fix`. ## No sealed byte moves — run, not assumed - **The ingest corpus regenerates byte-identical** to the committed goldens: no `content_hash` and no `manifest_hash` moved. Expected, since the ADR-0056 §6a decoder digest covers the *generic* lanes and dicom is a vendor backend outside it — but the DICOM lane carries its own golden hashes (#207), and a pixel-decode change would have moved those. **15 DICOM tests pass.** - **Gate B feature snapshots regenerate unchanged** (`scripts/feature-snapshots.sh`, zero files differ). Nothing dicom pulls sits on the sealed byte path those snapshots watch. - Full workspace tests pass under `--locked`; clippy clean; prek clean; commit signed. The one deliberate snapshot edit is `info.trycmd`'s human `backends:` line, which pins resolved backend versions on purpose: ```diff -backends: … dicom 0.9.1 · dicom-series 0.9.1 · … +backends: … dicom 0.10.0 · dicom-series 0.10.0 · … ``` Hand-edited as the single literal that changed — the JSON block already uses `[..]` for versions. Deliberately **not** `TRYCMD=overwrite`, which records whatever the binary printed and cannot distinguish an intended value from a regression. ## Why `thrift` is not here: it is blocked on Vortex, not on us #513 asks for `parquet`/`arrow-*` ≥ 59.0.0, since parquet 59 drops `thrift` entirely. That is not currently possible: ``` vortex-array 0.75.0 requires arrow-array ^58.3 vortex-buffer 0.75.0 requires arrow-buffer ^58.3 vortex-error 0.75.0 requires arrow-schema ^58.3 vortex-layout 0.75.0 requires arrow-array ^58.3 vortex-runend 0.75.0 requires arrow-array ^58.3 ``` `^58.3` means `< 59.0.0`. So moving our `=58.3.0` pins to 59+ would **not upgrade arrow — it would resolve a second arrow tree**, which is precisely what the pin comment says those pins exist to prevent: > *"58.3.0 is the version Vortex already resolves, so `arrow-array` / `-buffer` / `-schema` are ALREADY in the > default graph (Gate B snapshots all three) and these pins add no second arrow copy."* Two arrow versions either side of the `tessera-ingest` → `tessera-io` boundary is an ADR-0056 hazard surface, not a tidiness question. And `thrift` is a **non-optional** dependency of parquet 58.3.0, so no feature selection drops it either. So #45 needs Vortex on arrow ≥ 59 first — either upstream, or carried in the `gerchowl/vortex` fork we already pin for #468/#472 (#481), which is a materially bigger change than a version bump and wants its own decision. Left open and reported rather than forced.
This was referenced Oct 18, 2026
gerchowl
added a commit
that referenced
this pull request
Oct 19, 2026
## 🤖 New release * `tessera-core`: 0.1.0-alpha.1 -> 0.1.0-alpha.2 * `tessera-io`: 0.1.0-alpha.1 -> 0.1.0-alpha.2 * `tessera-ingest`: 0.1.0-alpha.1 -> 0.1.0-alpha.2 * `tessera-cli`: 0.1.0-alpha.1 -> 0.1.0-alpha.2 * `tessera-py`: 0.1.0-alpha.1 -> 0.1.0-alpha.2 * `tessera-wasm`: 0.1.0-alpha.1 -> 0.1.0-alpha.2 <details><summary><i><b>Changelog</b></i></summary><p> ## `tessera-core` <blockquote> ## [0.1.0-alpha.2](tessera-core-v0.1.0-alpha.1...tessera-core-v0.1.0-alpha.2) - 2026-10-19 ### Added - *(cli,core)* tessera provenance walk and inspect --resolve-config ([#490](#490)) - *(ingest)* generic table ingest — tessera ingest table <PARQUET|IPC|CSV> ([#460](#460)) ### Fixed - *(core,cli)* compute array statistics in exact integer arithmetic ### Other - ingest cookbook, generated product-schema reference, tested Python write example, cli.md stubs ([#484](#484)) </blockquote> ## `tessera-io` <blockquote> ## [0.1.0-alpha.2](tessera-io-v0.1.0-alpha.1...tessera-io-v0.1.0-alpha.2) - 2026-10-19 ### Added - *(cli)* serve array stats from the chunk-index with no decode, and add stats --json - *(ingest)* generic array ingest — NumPy .npy and .npz (ADR-0056 P1) - *(cli,core)* tessera provenance walk and inspect --resolve-config ([#490](#490)) ### Fixed - *(io)* chunk statistics on nullable columns with nulls - *(deps)* upgrade rustls and zip past RUSTSEC-2026-0346..0349/0367..0369 - *(cli,io)* pin inspect --json, guard stage-dir downgrade resume, route verify through the resource cap, 5.5x single-column reads ([#479](#479)) - *(io)* gate seal determinism across build profiles and pin the #468 vortex limitation ([#474](#474)) - *(io)* close the encode-trace callsite race under CPU contention ([#465](#465)) - *(ingest)* seal the ADR-0058 recipe on the streaming path and surface it in inspect ([#453](#453)) ### Other - *(deps)* pin vortex to a patched fork for #468/#472 ([#481](#481)) </blockquote> ## `tessera-ingest` <blockquote> ## [0.1.0-alpha.2](tessera-ingest-v0.1.0-alpha.1...tessera-ingest-v0.1.0-alpha.2) - 2026-10-19 ### Added - *(ingest)* generic array ingest — NumPy .npy and .npz (ADR-0056 P1) - *(ingest)* generic table ingest — tessera ingest table <PARQUET|IPC|CSV> ([#460](#460)) ### Fixed - *(deps)* upgrade rustls and zip past RUSTSEC-2026-0346..0349/0367..0369 - *(ingest)* the arrow lane builds on its own, and the gate proves it - *(ingest)* a decode-path package is (name, version), not a name - *(ingest)* the decoder digest is per lane, carries each crate's source, and covers the whole decode path - *(ingest)* NIfTI spatial units to mm, big-endian NIfTI-1, FSL qform cross-check ([#470](#470)) - *(ingest)* close the five NIfTI decoder bugs and verify the .nii.gz gzip trailer ([#447](#447)) - *(ingest)* seal the ADR-0058 recipe on the streaming path and surface it in inspect ([#453](#453)) ### Other - *(ingest)* align the hdf-compound streaming slab to the dataset chunking ([#464](#464)) </blockquote> ## `tessera-cli` <blockquote> ## [0.1.0-alpha.2](tessera-cli-v0.1.0-alpha.1...tessera-cli-v0.1.0-alpha.2) - 2026-10-19 ### Added - *(cli)* serve array stats from the chunk-index with no decode, and add stats --json - *(cli)* add Parquet to the bench compare head-to-head - *(cli)* bench compare reports BOTH table fixtures - *(ingest)* generic array ingest — NumPy .npy and .npz (ADR-0056 P1) - *(cli,core)* tessera provenance walk and inspect --resolve-config ([#490](#490)) - *(ingest)* generic table ingest — tessera ingest table <PARQUET|IPC|CSV> ([#460](#460)) - *(cli)* tessera bench compare — .tsra vs HDF5 head-to-head, every read verified, losses included ([#487](#487)) - *(cli)* native dtypes for npy grid output and native max reduction ([#486](#486)) - *(cli)* non-CSV grid output for slice/project, terminal-only preview caps, read on extra/ ([#476](#476)) ### Fixed - *(deps)* upgrade rustls and zip past RUSTSEC-2026-0346..0349/0367..0369 - *(core,cli)* compute array statistics in exact integer arithmetic - *(ingest)* the decoder digest is per lane, carries each crate's source, and covers the whole decode path - *(cli,io)* pin inspect --json, guard stage-dir downgrade resume, route verify through the resource cap, 5.5x single-column reads ([#479](#479)) - *(ingest)* seal the ADR-0058 recipe on the streaming path and surface it in inspect ([#453](#453)) - *(ci)* stop the whitespace autofixers rewriting trycmd snapshots - *(security)* generate the OpenSSH test key instead of committing one ### Other - *(deps)* bump dicom-rs to 0.10 to clear the three jxl advisories - *(book)* put Parquet in the comparison table - *(book)* report both table fixtures and say which one tracks reality - *(test)* versioning walkthrough succeeds instead of documenting blessed errors ([#491](#491)) - ingest cookbook, generated product-schema reference, tested Python write example, cli.md stubs ([#484](#484)) - *(ingest)* align the hdf-compound streaming slab to the dataset chunking ([#464](#464)) - clear the lint backlog the new CI gate surfaced </blockquote> ## `tessera-py` <blockquote> ## [0.1.0-alpha.2](tessera-py-v0.1.0-alpha.1...tessera-py-v0.1.0-alpha.2) - 2026-10-19 ### Added - *(py)* expose with_producer / with_generation on Builder - *(py)* pin a parent's seal on Builder.add_source - *(py)* expose the array codec on Builder.add_array ### Other - *(py)* prove the codec knob round-trips and the default is unchanged - ingest cookbook, generated product-schema reference, tested Python write example, cli.md stubs ([#484](#484)) - clear the lint backlog the new CI gate surfaced </blockquote> ## `tessera-wasm` <blockquote> ## [0.1.0-alpha.1](https://github.com/vig-os/tessera/releases/tag/tessera-wasm-v0.1.0-alpha.1) - 2026-09-23 ### Added - *(signing)* bind signed_at + key_format; ssh-ed25519 loader; usage tests-as-docs - *(wasm)* tessera-wasm JS bindings — in-browser verify, build→bindgen→node gated ([#210](#210)) </blockquote> </p></details> --- This PR was generated with [release-plz](https://github.com/release-plz/release-plz/).
gerchowl
added a commit
that referenced
this pull request
Oct 19, 2026
## 🤖 New release * `tessera-core`: 0.1.0-alpha.2 -> 0.1.0-alpha.3 * `tessera-io`: 0.1.0-alpha.2 -> 0.1.0-alpha.3 * `tessera-ingest`: 0.1.0-alpha.2 -> 0.1.0-alpha.3 * `tessera-cli`: 0.1.0-alpha.2 -> 0.1.0-alpha.3 * `tessera-py`: 0.1.0-alpha.2 -> 0.1.0-alpha.3 * `tessera-wasm`: 0.1.0-alpha.2 -> 0.1.0-alpha.3 <details><summary><i><b>Changelog</b></i></summary><p> ## `tessera-core` <blockquote> ## [0.1.0-alpha.2](tessera-core-v0.1.0-alpha.1...tessera-core-v0.1.0-alpha.2) - 2026-10-19 ### Added - *(cli,core)* tessera provenance walk and inspect --resolve-config ([#490](#490)) - *(ingest)* generic table ingest — tessera ingest table <PARQUET|IPC|CSV> ([#460](#460)) ### Fixed - *(core,cli)* compute array statistics in exact integer arithmetic ### Other - ingest cookbook, generated product-schema reference, tested Python write example, cli.md stubs ([#484](#484)) </blockquote> ## `tessera-io` <blockquote> ## [0.1.0-alpha.2](tessera-io-v0.1.0-alpha.1...tessera-io-v0.1.0-alpha.2) - 2026-10-19 ### Added - *(cli)* serve array stats from the chunk-index with no decode, and add stats --json - *(ingest)* generic array ingest — NumPy .npy and .npz (ADR-0056 P1) - *(cli,core)* tessera provenance walk and inspect --resolve-config ([#490](#490)) ### Fixed - *(io)* chunk statistics on nullable columns with nulls - *(deps)* upgrade rustls and zip past RUSTSEC-2026-0346..0349/0367..0369 - *(cli,io)* pin inspect --json, guard stage-dir downgrade resume, route verify through the resource cap, 5.5x single-column reads ([#479](#479)) - *(io)* gate seal determinism across build profiles and pin the #468 vortex limitation ([#474](#474)) - *(io)* close the encode-trace callsite race under CPU contention ([#465](#465)) - *(ingest)* seal the ADR-0058 recipe on the streaming path and surface it in inspect ([#453](#453)) ### Other - *(deps)* pin vortex to a patched fork for #468/#472 ([#481](#481)) </blockquote> ## `tessera-ingest` <blockquote> ## [0.1.0-alpha.2](tessera-ingest-v0.1.0-alpha.1...tessera-ingest-v0.1.0-alpha.2) - 2026-10-19 ### Added - *(ingest)* generic array ingest — NumPy .npy and .npz (ADR-0056 P1) - *(ingest)* generic table ingest — tessera ingest table <PARQUET|IPC|CSV> ([#460](#460)) ### Fixed - *(deps)* upgrade rustls and zip past RUSTSEC-2026-0346..0349/0367..0369 - *(ingest)* the arrow lane builds on its own, and the gate proves it - *(ingest)* a decode-path package is (name, version), not a name - *(ingest)* the decoder digest is per lane, carries each crate's source, and covers the whole decode path - *(ingest)* NIfTI spatial units to mm, big-endian NIfTI-1, FSL qform cross-check ([#470](#470)) - *(ingest)* close the five NIfTI decoder bugs and verify the .nii.gz gzip trailer ([#447](#447)) - *(ingest)* seal the ADR-0058 recipe on the streaming path and surface it in inspect ([#453](#453)) ### Other - *(ingest)* align the hdf-compound streaming slab to the dataset chunking ([#464](#464)) </blockquote> ## `tessera-cli` <blockquote> ## [0.1.0-alpha.3](tessera-cli-v0.1.0-alpha.2...tessera-cli-v0.1.0-alpha.3) - 2026-10-19 ### Fixed - *(cli)* make bench compare --cold Linux-only and refuse it elsewhere </blockquote> ## `tessera-py` <blockquote> ## [0.1.0-alpha.3](tessera-py-v0.1.0-alpha.2...tessera-py-v0.1.0-alpha.3) - 2026-10-19 ### Fixed - *(py)* link the _native extension dylib on macOS </blockquote> ## `tessera-wasm` <blockquote> ## [0.1.0-alpha.1](https://github.com/vig-os/tessera/releases/tag/tessera-wasm-v0.1.0-alpha.1) - 2026-09-23 ### Added - *(signing)* bind signed_at + key_format; ssh-ed25519 loader; usage tests-as-docs - *(wasm)* tessera-wasm JS bindings — in-browser verify, build→bindgen→node gated ([#210](#210)) </blockquote> </p></details> --- This PR was generated with [release-plz](https://github.com/release-plz/release-plz/).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #474 (which adds the test this PR flips). Targeting #474's branch rather than
devso thediff is only this change; GitHub retargets when #474 merges. Stacked PRs get CI now thanks to #473.
Why
vortex 0.75.0 has two defects that are the same underlying bug class: a write-time statistic
computation leaking into the serialised bytes, so a sealed
content_hashstops being a function ofthe data alone.
debug_assertionsis_sortedassert callscache_is_sorted, which setsStat::IsSortedon the patch indicessum_float_allskips NaN, so+infand-infmeet and evaluateinf + -inf;Stat::Sumis persisted0xfff8…vs aarch640x7ff8…Both fixed on a fork of the
0.75.0tag, pinned here by full commit SHA (never a branch name — abranch would let pinned content move under an unchanged-looking lockfile). #480 tracks removal.
Two requirements that fail silently
Worth reading before reviewing, because each one produces a confident, meaningless "nothing changed":
vortex-arrayalone leaves itssiblings resolving from crates.io → two copies of
vortex-proto/vortex-bufferand ~1000 typeerrors.
0.75.0tag ships placeholder version0.1.0(the real version is injected at publish).Without a version-setting commit on the fork branch, cargo prints
patch ... was not used in the crate graphand quietly keeps using crates.io.So the pin was verified positively, not by the absence of a warning: on x86_64 the sealed float
bytes no longer contain
fff8000000000000, now contain7ff8000000000000, and became byte-identicalto what unpatched aarch64 produced.
#474's known-limitation test, flipped
known_limitation_468_*asserted the bug was still present, so it fails once fixed — by design.It becomes
full_span_int_container_bytes_are_build_config_independent, with one expected lengthper shape and no
cfg!(debug_assertions)branch. That absence is the assertion:flake.nixrunsit under both
cargo testandcargo test --release, so a re-divergence fails in exactly one line.Measured both profiles on the pinned fork before rewriting the expectations, as requested:
[i64::MIN, 0, i64::MAX]table::encode[i64::MIN, i64::MAX]Both profiles agree on the former release numbers — the debug build was the one carrying the extra
24 bytes of cached statistic, which is the expected direction.
Gate B: every line explained
458 insertions against 458 deletions across 8 snapshots — perfectly balanced. Every changed line
is identical apart from gaining the
(https://github.com/gerchowl/vortex?rev=…)source tag, verifiedby re-comparing with that tag stripped:
No seal-path crate gains or loses a feature, so no codec resolves differently.
Verification
outputHashesneeded — verified by buildingchecks.x86_64-linux.feature-snapshots, which vendoredcargo-git-https-github.com-gerchowl-vortexand reached the Gate B diffgen_corpusvs committedcorpus/corpus.jsongen_corpusdev vs releasecargo deny check sourcessources okDECODE_PATH_CRATESlocked versionsingest_decoderdigest cannot movecargo-deny tightened rather than left permissive:
unknown-gitgoes fromallowtodeny, withallow-gitnaming exactly the pinned repository. A blanket allow would let a future stray gitdependency in unnoticed, which is precisely the review signal a fork pin should raise.
Upstream — the exit path
Both defects are filed and fixed upstream, so this pin has a visible end:
is_sortedassertSumBoth were rebased onto vortex's
develop, not the0.75.0tag: 23 commits had touched those filessince, including an AggregateVTable rework and a new
sum_v2that finalises floats through its ownpath, so #10122 fixes two sites there. Both defects were re-verified as still present on
developbefore filing. The tessera pin stays on the
0.75.0-based fork branch and is unaffected by thatrebase.
#480 tracks removing the pin, with the checklist.
Notes for reviewers
cp: cannot stat 'README.md'per vortex crate. Cosmetic: the workspaceincludekey lists a per-crate README that does not exist in the git checkout. No compiled input isaffected, and
cargo treeover the vendored deps works.parquetis absent from this branch's lockfile (the ingest lane arrives with feat(ingest): generic table ingest — Parquet, Arrow IPC and CSV (ADR-0056 P1) #460), so its versionis trivially unchanged here. tsra-386 is regenerating the ingest corpus under both profiles on
this branch — expected movement only in the patch-producing int and NaN-sum float fixtures.
gerchowl/vortex:tessera/0.75.0-patched:0.75.0+ the two signed fixes + onenon-upstream version commit. It is branch-protected (locked, no force-push, no deletion,
enforce_admins) so the pinned commit cannot be GC'd out from under a fresh build.Refs #468, #472, #474, #480