Skip to content

fmax optimization on aarch64 unexpectedly working for signaling NaN #151286

@youknowone

Description

@youknowone

Related LLVM issue: llvm/llvm-project#176624

I tried this code:

#[inline]
pub fn hypot(coords: &[f64]) -> f64 {
    let mut max = 0.0_f64;
    let mut found_nan = false;
    let mut abs_coords = Vec::with_capacity(10);

    for &x in coords {
        let ax = x.abs();
        abs_coords.push(ax);
        found_nan |= ax.is_nan();
        if ax > max {
            max = ax;
        }
    }

    vector_norm(&abs_coords, max, found_nan)
}

unsafe extern "C" {
    pub fn frexp(n: f64, exp: *mut i32) -> f64;
}

/// Compute the Euclidean norm of a vector with high precision.
pub fn vector_norm(vec: &[f64], max: f64, found_nan: bool) -> f64 {
    let n = vec.len();

    if max.is_infinite() {
        return max;
    }
    if found_nan {
        return f64::NAN;
    }

    let mut max_e: i32 = 0;
    unsafe { frexp(max, &mut max_e) };

    // the code below is totally unreachable, but making recursive call is important
    if max_e < -1023 {
        // When max_e < -1023, ldexp(1.0, -max_e) would overflow.
        // TODO: This can be in-place ops, but we allocate a copy since we take &[f64].
        // This is acceptable because subnormal inputs are extremely rare in practice.
        let vec_copy: Vec<f64> = vec.iter().map(|&x| x / f64::MIN_POSITIVE).collect();
        let r = f64::MIN_POSITIVE * vector_norm(&vec_copy, max / f64::MIN_POSITIVE, found_nan);
        unreachable!();
        return r;
    }

    unreachable!();
}

#[cfg(test)]
mod tests {
    use super::*;

    pub const SNAN: f64 = f64::from_bits(0x7FF0_0000_0000_0001);

    fn test_hypot(coords: &[f64]) {
        let rs_result = hypot(coords);

        // SNAN + INFINITY should return INFINITY
        if coords[0].to_bits() == SNAN.to_bits() && coords[1].to_bits() == f64::INFINITY.to_bits() {
            assert_eq!(
                rs_result.to_bits(),
                0x7ff0000000000000,
                "hypot([SNAN, INFINITY]) should return INFINITY, got 0x{:x}",
                rs_result.to_bits()
            );
        }

        // Minimal serde_json usage - required for bug to manifest
        // This code path is never executed for SNAN test case
        if rs_result.is_nan() {
            return;
        }

        let _json = serde_json::to_string(&rs_result).unwrap();
    }

    pub const EDGE_VALUES: &[f64] = &[SNAN, f64::INFINITY];

    #[test]
    fn edgetest_hypot() {
        for &x in EDGE_VALUES {
            for &y in EDGE_VALUES {
                test_hypot(&[x, y]);
            }
        }
    }
}

Full cargo: math_bug.zip

The code size is looking not small, but I couldn't cut it anymore.

I expected to see the debug profile and release profile takes same control path.
Instead, it hits different path on aarch64 with release profile

$ cargo +nightly test --lib
   Compiling math_bug v0.1.0 (/Users/user/Projects/rust/math_bug)
...
warning: `math_bug` (lib test) generated 3 warnings (run `cargo fix --lib -p math_bug --tests` to apply 2 suggestions)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 1.04s
     Running unittests src/lib.rs (target/debug/deps/math_bug-0ebeb571c288fb81)

running 1 test
test tests::edgetest_hypot ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
cargo +nightly test --lib --release

warning: `math_bug` (lib test) generated 3 warnings (run `cargo fix --lib -p math_bug --tests` to apply 2 suggestions)
    Finished `release` profile [optimized] target(s) in 0.06s
     Running unittests src/lib.rs (target/release/deps/math_bug-9708721fabfaa446)

running 1 test
test tests::edgetest_hypot ... FAILED

failures:

---- tests::edgetest_hypot stdout ----

thread 'tests::edgetest_hypot' (37720335) panicked at src/lib.rs:62:13:
assertion `left == right` failed: hypot([SNAN, INFINITY]) should return INFINITY, got 0x7ff8000000000000
  left: 9221120237041090560
 right: 9218868437227405312
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    tests::edgetest_hypot

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

error: test failed, to rerun pass `--lib`

Meta

rustc --version --verbose:

$ cargo +nightly --version --verbose
cargo 1.94.0-nightly (6d1bd93c4 2026-01-10)
release: 1.94.0-nightly
commit-hash: 6d1bd93c47f059ec1344cb31e68a2fb284cbc6b1
commit-date: 2026-01-10
host: aarch64-apple-darwin
libgit2: 1.9.2 (sys:0.20.3 vendored)
libcurl: 8.7.1 (sys:0.4.84+curl-8.17.0 system ssl:(SecureTransport) LibreSSL/3.3.6)
ssl: OpenSSL 3.5.4 30 Sep 2025
os: Mac OS 26.0.1 [64-bit]
Backtrace

<backtrace>

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-LLVMArea: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues.A-floating-pointArea: Floating point numbers and arithmeticC-optimizationCategory: An issue highlighting optimization opportunities or PRs implementing suchO-AArch64Armv8-A or later processors in AArch64 mode

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status

    To Triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions