You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
determinism: ingest_parquet_floats seals differently on aarch64 vs x86_64 in the SAME (debug) profile — genuine cross-arch float divergence #472
On CI, nix flake check (aarch64-linux) computes content_hash 33b065… for the #460 fixture ingest_parquet_floats, while x86_64 in both debug and release computes 2ee5bf… (the committed golden). Only that fixture moves on aarch64. It holds NaN 0x7ff8_0000_dead_beef, ±inf, −0.0, MIN_POSITIVE and the largest subnormal.
This depends on the arch, not the build profile, so it can't be reproduced on x86 and is a separate mechanism from #468's i64 debug/release split. On x86, the same f64 values encode identically in both profiles.
Next: bisect per column and per value on an aarch64 runner (workflow_dispatch with a probe), and check NaN-payload canonicalisation and the subnormal/FTZ behaviour of any float→int path in the encoder.
changed the title [-]determinism: ingest_parquet_floats seals a different content_hash on aarch64 release (arch-dependent, separate from #468)[/-][+]determinism: ingest_parquet_floats seals differently on aarch64 vs x86_64 in the SAME (debug) profile — genuine cross-arch float divergence[/+]on Sep 28, 2026
Mechanism found and confirmed: the persisted Stat::Sum, not a codec
It is one bit in the whole file — the sign bit of a NaN that Vortex computes and writes as the
column's Sum statistic. It is neither Pco nor a scheme, so deterministic_table_compressor's
ALP/ALPRD exclusion cannot address it: that exclusion is aimed at a different layer.
The chain
Stat::Sum is in PRUNING_STATS (vortex-array-0.75.0/src/stats/mod.rs), documented as the
stats "we want to ensure … are computed when compressing/writing", and StatsSet::write_flatbuffer (src/stats/flatbuffers.rs) serialises it. So the sum is in the
file bytes on every write.
vortex-array-0.75.0/src/aggregate_fn/fns/sum/primitive.rs:63 skips NaNs, then accumulates:
pub(super)fnsum_float_all<T:NativePType>(acc:&mutf64,slice:&[T]){for&v in slice {if !v.is_nan(){*acc += ToPrimitive::to_f64(&v).vortex_expect("float to f64");}}}
Because NaNs are skipped, the column's +inf and -inf meet in the accumulator and perform inf + (-inf) — an IEEE 754 invalid operation. IEEE does not specify the resulting NaN's bit
pattern, and the two architectures differ:
target
inf + (-inf)
x86_64 (SSE)
0xfff8_0000_0000_0000 — sign bit set
aarch64
0x7ff8_0000_0000_0000 — sign bit clear
That value is written into the stats → the file differs in one bit → different content_hash.
The NaN-skipping is what exposes this. Without it the accumulator would be NaN from element 1 and
would propagate that element's payload, which both architectures do identically (measured below).
Skipping is what lets the two infinities meet and generate a fresh default NaN.
Evidence
Both arches in one workflow matrix, same pinned rustc 1.96.0, debug profile, so the target
architecture is the only variable. Encoding the fixture's column through table::encode per
cumulative prefix:
case
len (both)
verdict
control_1_2_3 (ordinary finite)
2564
byte-identical
each of the 7 values alone
2204 / 2548
byte-identical
prefix_1 … prefix_4
2204–2572
byte-identical
prefix_5 (adds -inf)
2580
1 byte, 1 bit: ff→7f at offset 223
prefix_6, prefix_7, full column
2588
1 byte, 1 bit: ff→7f at offset 231
The differing byte is the top byte of an 8-byte little-endian field:
prefix_4 already contains +inf; prefix_5 is the first to add -inf. So the trigger is exactly
"both infinities in one column", which is precisely what makes inf + (-inf) reachable. The
offset moves 223 → 231 only because the array grew by one 8-byte element.
Direct measurement of the hardware semantics, same build:
So NaN payload propagation is identical on both (the fixture's 0xdeadbeef payload survives
arithmetic either way), and f64::min/max correctly ignore NaN on both. The only divergence is the
default NaN generated by an invalid operation.
Why this explains everything that made it look strange
Unreproducible on x86 by construction — a single machine can never show it. Note, though, that
it is diagnosable on x86 alone: the encoded bytes contain exactly one fff8000000000000, a value
that appears nowhere in the input data. That was the tell, before any ARM runner was involved.
Only this fixture moves: it is the only one carrying both infinities.
#468's ingest_parquet_scalars moves by profile (the side-effecting is_sorted assert on patch
indices). This one moves by arch, in a different layer. Both are real; neither fix covers the
other.
Next
Not proposing a fix in this comment, per the agreed order. The upstream vortex issue is drafted and
held, with three candidate fixes — the cheapest being to canonicalise a NaN sum before persisting it
(the value is "not a number", so a platform-chosen payload carries no information). A tessera-side
mitigation is also possible without waiting for upstream. Both need an owner decision, and no golden
on dev should move without one.
Probe and the throwaway dual-arch workflow are on spike/472-float-cross-arch-probe (not for merge).
Fix is to report any NaN sum as the canonical quiet NaN. On develop this needed two sites, not one — sum and the newer sum_v2 each finalise floats through their own path — so the rebase widened the change. A useful precedent: sum_v2 already writes f64::NAN explicitly on its non-skip_nans poisoning path, so canonicalising makes the computed NaN agree with the one it already hardcodes.
Verified end to end under the pin in #481: all 21 byte-carrying probe lines are byte-identical across x86_64 and aarch64, covering this issue's float shapes and #468's int shapes. The raw hardware NaN probe still differs per arch, as it should — only the persisted statistic is normalised.
On the ingest lane (tsra-386's run), ingest_parquet_floats moves 2ee5bf4d1b33 → 33b065510f5e, which is exactly the value aarch64 was already producing. So the fix brings x86 into agreement with ARM rather than inventing a third value. Every other ingest fixture is unchanged.
Fixed on dev via the pinned vortex fork (#481). CI guards cover it on both arches. Upstream fixes: vortex-data/vortex#10121 (#468) and #10122 (#472). Removing the pin once upstream releases is tracked in #480.
Activity
[-]determinism: ingest_parquet_floats seals a different content_hash on aarch64 release (arch-dependent, separate from #468)[/-][+]determinism: ingest_parquet_floats seals differently on aarch64 vs x86_64 in the SAME (debug) profile — genuine cross-arch float divergence[/+]spike: probe the #472 float cross-arch divergence on both arches
spike: probe the #472 float cross-arch divergence on both arches
gerchowl commentedon Sep 28, 2026
Mechanism found and confirmed: the persisted
Stat::Sum, not a codecIt is one bit in the whole file — the sign bit of a NaN that Vortex computes and writes as the
column's
Sumstatistic. It is neither Pco nor a scheme, sodeterministic_table_compressor'sALP/ALPRD exclusion cannot address it: that exclusion is aimed at a different layer.
The chain
Stat::Sumis inPRUNING_STATS(vortex-array-0.75.0/src/stats/mod.rs), documented as thestats "we want to ensure … are computed when compressing/writing", and
StatsSet::write_flatbuffer(src/stats/flatbuffers.rs) serialises it. So the sum is in thefile bytes on every write.
vortex-array-0.75.0/src/aggregate_fn/fns/sum/primitive.rs:63skips NaNs, then accumulates:Because NaNs are skipped, the column's
+infand-infmeet in the accumulator and performinf + (-inf)— an IEEE 754 invalid operation. IEEE does not specify the resulting NaN's bitpattern, and the two architectures differ:
inf + (-inf)0xfff8_0000_0000_0000— sign bit set0x7ff8_0000_0000_0000— sign bit clearThat value is written into the stats → the file differs in one bit → different
content_hash.The NaN-skipping is what exposes this. Without it the accumulator would be NaN from element 1 and
would propagate that element's payload, which both architectures do identically (measured below).
Skipping is what lets the two infinities meet and generate a fresh default NaN.
Evidence
Both arches in one workflow matrix, same pinned rustc 1.96.0, debug profile, so the target
architecture is the only variable. Encoding the fixture's column through
table::encodepercumulative prefix:
control_1_2_3(ordinary finite)prefix_1…prefix_4prefix_5(adds-inf)ff→7fat offset 223prefix_6,prefix_7, full columnff→7fat offset 231The differing byte is the top byte of an 8-byte little-endian field:
prefix_4already contains+inf;prefix_5is the first to add-inf. So the trigger is exactly"both infinities in one column", which is precisely what makes
inf + (-inf)reachable. Theoffset moves 223 → 231 only because the array grew by one 8-byte element.
Direct measurement of the hardware semantics, same build:
So NaN payload propagation is identical on both (the fixture's
0xdeadbeefpayload survivesarithmetic either way), and
f64::min/maxcorrectly ignore NaN on both. The only divergence is thedefault NaN generated by an invalid operation.
Why this explains everything that made it look strange
+=, so no vectorisation isinvolved and there is no
[profile.dev.package."*"]opt-level override in the workspace. This ispure hardware semantics, which is why x86_64 debug and release agree and both differ from aarch64.
it is diagnosable on x86 alone: the encoded bytes contain exactly one
fff8000000000000, a valuethat appears nowhere in the input data. That was the tell, before any ARM runner was involved.
Not the same bug as #468
#468's
ingest_parquet_scalarsmoves by profile (the side-effectingis_sortedassert on patchindices). This one moves by arch, in a different layer. Both are real; neither fix covers the
other.
Next
Not proposing a fix in this comment, per the agreed order. The upstream vortex issue is drafted and
held, with three candidate fixes — the cheapest being to canonicalise a NaN sum before persisting it
(the value is "not a number", so a platform-chosen payload carries no information). A tessera-side
mitigation is also possible without waiting for upstream. Both need an owner decision, and no golden
on
devshould move without one.Probe and the throwaway dual-arch workflow are on
spike/472-float-cross-arch-probe(not for merge).Refs #468, #460
docs(adr): a guarantee that depends on an absence is not a guarantee
chore(ingest): merge dev (devkit 1.17.0, stacked-PR CI, NIfTI units, …
docs(adr): say only what is true on dev, and scope the release-build …
gerchowl commentedon Sep 29, 2026
Filed upstream: vortex-data/vortex#10120 (issue) and vortex-data/vortex#10122 (fix PR).
Fix is to report any NaN sum as the canonical quiet NaN. On
developthis needed two sites, not one —sumand the newersum_v2each finalise floats through their own path — so the rebase widened the change. A useful precedent:sum_v2already writesf64::NANexplicitly on its non-skip_nanspoisoning path, so canonicalising makes the computed NaN agree with the one it already hardcodes.Verified end to end under the pin in #481: all 21 byte-carrying probe lines are byte-identical across x86_64 and aarch64, covering this issue's float shapes and #468's int shapes. The raw hardware NaN probe still differs per arch, as it should — only the persisted statistic is normalised.
On the ingest lane (tsra-386's run),
ingest_parquet_floatsmoves2ee5bf4d1b33 → 33b065510f5e, which is exactly the value aarch64 was already producing. So the fix brings x86 into agreement with ARM rather than inventing a third value. Every other ingest fixture is unchanged.#480 tracks removing the pin once released.
test(io): guard the canonical float-sum NaN on this branch, not only …
chore: merge dev after #474 squash-merged
build(deps): pin vortex to a patched fork for #468/#472 (#481)
gerchowl commentedon Sep 29, 2026
Fixed on dev via the pinned vortex fork (#481). CI guards cover it on both arches. Upstream fixes: vortex-data/vortex#10121 (#468) and #10122 (#472). Removing the pin once upstream releases is tracked in #480.
7 remaining items
chore: release v0.1.0-alpha.3 (#652)
ci: run the aarch64 nix flake check nightly, on release tags and by h…
build(deps)!: move vortex from the 0.75 fork to 0.93.0 on a rebased f…
build(deps)!: move vortex from the 0.75 fork to 0.93.0 on a rebased f…
build(deps)!: move vortex from the 0.75 fork to 0.93.0 on a rebased f…
build(deps)!: move vortex from the 0.75 fork to 0.93.0 on a rebased f…
test(nix): fail the #468 and #472 guards unless exactly one test ran
test(io): make the #472 guard's messages match what it asserts
build(deps)!: move vortex from the 0.75 fork to 0.93.0 on a rebased f…
test(nix): fail the #468 and #472 guards unless exactly one test ran
test(io): make the #472 guard's messages match what it asserts
build(deps)!: move vortex from the 0.75 fork to 0.93.0 on a rebased f…
test(nix): fail the #468 and #472 guards unless exactly one test ran
test(io): make the #472 guard's messages match what it asserts
build(deps)!: move vortex from the 0.75 fork to 0.93.0 on a rebased f…