You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
determinism: vortex writes different container bytes under debug_assertions (side-effecting is_sorted assert on patch indices) — debug- vs release-built tools seal differently #468
Found by #460's corpus fixtures (#386). It is a pre-existing tessera-io defect, not something #386 introduced.
The same tree and the same inputs seal a different content_hash depending on target arch and build profile. Each result was verified on a clean target dir, which rules out stale build artifacts:
ingest_parquet_scalars
ingest_parquet_floats
x86_64 debug
b4fe24
2ee5bf
x86_64 release
b26b10
2ee5bf
aarch64 release
b4fe24
33b065
Two different fixtures move under two different (arch × opt-level) combinations. Both are the float-bearing fixtures:
floats holds NaN payloads, ±inf, −0.0, MIN_POSITIVE and subnormals.
scalars holds f32 and f64, integer extremes, Date32, Timestamp and Decimal128.
That points to float codegen (FMA and opt-level differences) reaching the encoder's output bytes.
tessera-io::table::deterministic_table_compressor already excludes ALP/ALPRD for exactly this reason (their exponent search is sensitive to codegen). So either:
that exclusion is not effective on these columns, or
another scheme, or btrblocks' own sample-based cost model (float compression-ratio comparisons), has the same property for short or pathological float columns.
Pco has only been proven cross-arch on normal listmode data.
Ruled out: tzdb, LC_ALL, CPU count, SOURCE_DATE_EPOCH, workspaceFeatures/sql, nextest isolation, the nix sandbox, a stale rlib, and float pow in our own seal path.
Plan
Bisect to the exact column, dtype and scheme with a per-column probe run in both profiles.
Make scheme selection a function of the data alone. Pin the float scheme, or exclude the remaining float-cost-model paths, or make the cost comparison integer-exact.
Add a regression gate that seals the pathological float fixtures under debug and release in CI, since Gate A currently runs one profile per arch.
Constraint: prefer a fix that leaves every golden already on dev unmoved. If an existing dev golden must move, that is a format event and needs an explicit owner decision.
#460 cannot go green until this lands, because its corpus would otherwise assert a value that depends on the host.
Correction from bisection: this is NOT float codegen. A single-column probe through tessera_io::table::encode (no arrow, no ingest code) shows exactly one column differs between debug and release:
i64 [i64::MIN, 0, i64::MAX]: debug len=2940, release len=2916. A different encoding is chosen.
All other dtypes are identical in both profiles: i8–u64, f32/f64 (including NaN payload, ±inf, −0.0 and subnormals), bool, text, date, timestamps and decimal. So are the two-element i64 spans and every narrower i64 range.
This explains the x86_64 failure of ingest_parquet_scalars, which carries the i64 extremes. The aarch64 failure of ingest_parquet_floats is not explained by it. It depends on the arch, not the profile, and is tracked separately.
Already ruled out by reading vortex 0.75.0: the sampling RNG uses a fixed seed; the scheme order is a fixed slice; best_bit_width is integer-only; max_minus_min goes through i128; candidate ratios are single exact int→f64 divisions; and no debug_assertions or catch_unwind paths exist. The selector compares f64 ratios with a strict > (estimate.rs:156), but no codegen-sensitive ratio has been found yet. A TRACE probe of the per-candidate verdicts is next.
Root cause narrowed further. It is not scheme selection. A TRACE of vortex_compressor::encode shows compression decisions that are byte-identical across profiles (zigzag → bitpacking, the same ratios and winner, and the same stats). The 24-byte difference is in vortex file container/layout metadata (the first differing byte is at offset 101, a flatbuffer length of 440 vs 416). It correlates with an extra execute_parent rewrite (vortex.cast(u8, len=1) → vortex.primitive) in the debug executor.
Trigger: i64/u64 with at least 3 rows, spanning nearly the full width and containing 0 (e.g. [MIN,0,MAX], [MIN+1,0,MAX], u64 [MAX,0,MAX-1]). Stable: [MIN,1,MAX], [MIN,-1,MAX], [0,1,MAX].
Impact: CI shows aarch64 release equals x86 debug, which differs from x86 release. So the released x86 and aarch64 binaries seal this shape differently. That makes it a cross-arch S15 hole for such data. The dev corpus is profile-stable and nothing on dev is mis-sealed: gen_corpus is identical in both profiles and matches the committed corpus.json.
Correction to my earlier comment: there is no cross-arch component. The discriminator is debug_assertions alone (measured by toggling only that knob, in both directions). The cause is a side-effecting #[cfg(debug_assertions)] assert!(is_sorted(&indices…)) in vortex-array patches.rs:278, which runs only for patch-producing inputs. It reproduces with stock vortex APIs and a bare PrimitiveArray.
The apparent aarch64 vs x86 split was two different checks with two different profiles: ingestGateA runs cargo run (dev, debug assertions on), and workspace-test runs crane nextest (release, off). All release builds agree with each other on any arch. What diverges is a debug-built tool against a release-built one. That is still an S15 violation, since the bytes must be a function of the data for any build of a version, but it does not hit two shipped binaries.
#472 (aarch64 ingest_parquet_floats) is a separate, genuine same-profile cross-arch difference and stays open.
changed the title [-]determinism: i64 full-span column [MIN, 0, MAX] seals differently in debug vs release (btrblocks picks a different scheme) — breaks S15 / ADR-0057 §7[/-][+]determinism: vortex writes different container bytes under debug_assertions (side-effecting is_sorted assert on patch indices) — debug- vs release-built tools seal differently[/+]on Sep 28, 2026
Activity
[-]determinism: sealed table payload depends on float codegen (arch × opt-level) — breaks S15 / ADR-0057 §7[/-][+]determinism: i64 full-span column [MIN, 0, MAX] seals differently in debug vs release (btrblocks picks a different scheme) — breaks S15 / ADR-0057 §7[/+]gerchowl commentedon Sep 28, 2026
Correction from bisection: this is NOT float codegen. A single-column probe through
tessera_io::table::encode(no arrow, no ingest code) shows exactly one column differs between debug and release:[i64::MIN, 0, i64::MAX]: debug len=2940, release len=2916. A different encoding is chosen.This explains the x86_64 failure of
ingest_parquet_scalars, which carries the i64 extremes. The aarch64 failure ofingest_parquet_floatsis not explained by it. It depends on the arch, not the profile, and is tracked separately.Already ruled out by reading vortex 0.75.0: the sampling RNG uses a fixed seed; the scheme order is a fixed slice;
best_bit_widthis integer-only;max_minus_mingoes through i128; candidate ratios are single exact int→f64 divisions; and nodebug_assertionsorcatch_unwindpaths exist. The selector compares f64 ratios with a strict>(estimate.rs:156), but no codegen-sensitive ratio has been found yet. A TRACE probe of the per-candidate verdicts is next.gerchowl commentedon Sep 28, 2026
Root cause narrowed further. It is not scheme selection. A TRACE of
vortex_compressor::encodeshows compression decisions that are byte-identical across profiles (zigzag → bitpacking, the same ratios and winner, and the same stats). The 24-byte difference is in vortex file container/layout metadata (the first differing byte is at offset 101, a flatbuffer length of 440 vs 416). It correlates with an extraexecute_parentrewrite (vortex.cast(u8, len=1) → vortex.primitive) in the debug executor.Trigger: i64/u64 with at least 3 rows, spanning nearly the full width and containing 0 (e.g.
[MIN,0,MAX],[MIN+1,0,MAX],u64 [MAX,0,MAX-1]). Stable:[MIN,1,MAX],[MIN,-1,MAX],[0,1,MAX].Impact: CI shows aarch64 release equals x86 debug, which differs from x86 release. So the released x86 and aarch64 binaries seal this shape differently. That makes it a cross-arch S15 hole for such data. The dev corpus is profile-stable and nothing on dev is mis-sealed: gen_corpus is identical in both profiles and matches the committed corpus.json.
Interim:
The upstream vortex report is being drafted, after a time-boxed hunt for the exact condition (UB, overflow or uninitialised-read hypotheses).
test(ingest): keep the scalars fixture off the shape #468 seals non-r…
gerchowl commentedon Sep 28, 2026
Correction to my earlier comment: there is no cross-arch component. The discriminator is
debug_assertionsalone (measured by toggling only that knob, in both directions). The cause is a side-effecting#[cfg(debug_assertions)] assert!(is_sorted(&indices…))in vortex-arraypatches.rs:278, which runs only for patch-producing inputs. It reproduces with stock vortex APIs and a barePrimitiveArray.The apparent aarch64 vs x86 split was two different checks with two different profiles:
ingestGateArunscargo run(dev, debug assertions on), andworkspace-testruns crane nextest (release, off). All release builds agree with each other on any arch. What diverges is a debug-built tool against a release-built one. That is still an S15 violation, since the bytes must be a function of the data for any build of a version, but it does not hit two shipped binaries.#472 (aarch64
ingest_parquet_floats) is a separate, genuine same-profile cross-arch difference and stays open.[-]determinism: i64 full-span column [MIN, 0, MAX] seals differently in debug vs release (btrblocks picks a different scheme) — breaks S15 / ADR-0057 §7[/-][+]determinism: vortex writes different container bytes under debug_assertions (side-effecting is_sorted assert on patch indices) — debug- vs release-built tools seal differently[/+]spike: probe the #472 float cross-arch divergence on both arches
spike: probe the #472 float cross-arch divergence on both arches
docs(adr): a guarantee that depends on an absence is not a guarantee
chore: merge dev (devkit 1.17.0, stacked-PR CI, NIfTI units, trace race)
chore(ingest): merge the table lane (and dev through it), regeneratin…
docs(adr): say only what is true on dev, and scope the release-build …
7 remaining items
chore(ingest): merge dev with the vortex pin, and record that #468/#472…
chore(deps): bump dicom-rs to 0.10 to clear the three jxl advisories (#…
chore: release v0.1.0-alpha.2 (#639)
chore: release v0.1.0-alpha.3 (#652)
ci: run the aarch64 nix flake check nightly, on release tags and by h…
build(deps)!: move vortex from the 0.75 fork to 0.93.0 on a rebased f…
build(deps)!: move vortex from the 0.75 fork to 0.93.0 on a rebased f…
build(deps)!: move vortex from the 0.75 fork to 0.93.0 on a rebased f…
build(deps)!: move vortex from the 0.75 fork to 0.93.0 on a rebased f…
test(nix): fail the #468 and #472 guards unless exactly one test ran
build(deps)!: move vortex from the 0.75 fork to 0.93.0 on a rebased f…
test(nix): fail the #468 and #472 guards unless exactly one test ran
build(deps)!: move vortex from the 0.75 fork to 0.93.0 on a rebased f…
test(nix): fail the #468 and #472 guards unless exactly one test ran
build(deps)!: move vortex from the 0.75 fork to 0.93.0 on a rebased f…