Skip to content

Commit e254f82

Browse files
0z5apytorchmergebot
authored andcommitted
[AArch64] Fix BF16 reciprocal and BF16/Half rsqrt rounding errors (#203940)
## Linked issue or supporting maintainer Fixes #203939 ## BC-breaking? No. Some BF16 and Half results on AArch64 change by up to about 1 ULP, towards the exact value. ## Summary (human written only) BF16 `reciprocal` and BF16/Half `rsqrt` on AArch64 were more than 1 ULP off and disagreed with `1/x` and across CPU capabilities; this computes both in fp32 and rounds once, as AVX2 does, at the cost of some speed. ## Agent details (if applicable) > Drafted with Claude Code; the code, tests and the runs below were done with it. Pushed as `0z5a:fix/aarch64-bf16-reciprocal-rsqrt`. > > - `reciprocal` no longer has the `vrecpeq_f32` path, so it always goes through `Vectorized<float>::reciprocal`. `rsqrt` in the shared reduced-precision base now goes through `Vectorized<float>::rsqrt`, which covers BF16 and Half. Half `reciprocal` is unchanged. > - New `ReciprocalReducedFloat` tests in `vec_test_all_types.cpp` check every normal input against the exact result, allowing 0.501 ULP for double rounding. They only build for the AArch64 DEFAULT and SVE128 paths: x86 DEFAULT still has the `rsqrt` error (see below), and SVE256 could not be run. > > Largest error on a CPU build of `d60701e`, GH200 (Neoverse V2): > > | Operation | SVE128 before | SVE128 after | DEFAULT before | DEFAULT after | > |---|---|---|---|---| > | BF16 `reciprocal` | 1.18 ULP | 0.50 ULP | 0.50 ULP | 0.50 ULP | > | BF16 `rsqrt` | 1.70 ULP | 0.50 ULP | 1.14 ULP | 0.50 ULP | > | Half `rsqrt` | 1.33 ULP | 0.50 ULP | 1.33 ULP | 0.50 ULP | > > After the fix, `torch.reciprocal(x)` equals `tensor(1.)/x` on all finite BF16 and Half inputs, and DEFAULT and SVE128 give identical results for `reciprocal` and `rsqrt` on those inputs (`rsqrt`: positive finite). Those results are also identical to `2.16.0.dev20261107+cpu` on x86 AVX2 and AVX512 (Xeon Gold 5120, hash of all outputs compared). > > Time per element, one thread, 1M elements: > > | Operation | SVE128 before | SVE128 after | DEFAULT before | DEFAULT after | > |---|---|---|---|---| > | BF16 `reciprocal` | 0.176 ns | 0.257 ns | 0.414 ns | 0.418 ns | > | BF16 `rsqrt` | 0.337 ns | 0.519 ns | 0.930 ns | 0.618 ns | > | Half `rsqrt` | 0.371 ns | 0.515 ns | 0.541 ns | 0.512 ns | > > `reciprocal` could keep more of its speed with `vrecpeq_f32` plus one `vrecpsq_f32` step: 0.180 ns against 0.228 ns for the divide in a standalone intrinsics program, with the same 0.50 ULP on normal inputs. I did not use it because it returned `-inf` for very small positive subnormals, where the divide returns `inf`. `rsqrt` has the same option: `vrsqrteq_f32` plus one step runs at 0.213 ns, faster than today, with the same 0.50 ULP on normal inputs, but it returns `-inf` for very small subnormals whose answer is about 1e20. > > `vec_test_all_types`: > > - The new tests fail before the fix (DEFAULT: `RSqrt` for BF16 and Half; SVE128: also BF16 `Reciprocal`) and pass after it. > - The same tests fail before and after the fix, so they are unrelated: `Arithmetic/3.Multiplication` on DEFAULT and SVE128, and `BitwiseFloatsAdditional/3.{Fmadd,FmaddVecN,Fmsub}` on DEFAULT. > > Not covered: x86 DEFAULT `rsqrt` measures 1.14 ULP for BF16 and 1.33 ULP for Half. It uses the generic path, which this PR does not change. x86 was only run from a nightly wheel, so the new tests and the fixed build were not run there. SVE256 was not run: this machine has 128-bit SVE, so the SVE256 test binary fails and crashes with or without the fix, and the new tests are now excluded from it. That exclusion was added after the runs above; it does not change what the DEFAULT and SVE128 binaries compile. `spin fixlint` was not run. Pull Request resolved: #203940 Approved by: https://github.com/Nicoshev, https://github.com/ezyang
1 parent 392f184 commit e254f82

3 files changed

Lines changed: 57 additions & 10 deletions

File tree

‎aten/src/ATen/cpu/vec/vec128/vec128_bfloat16_neon.h‎

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -387,23 +387,15 @@ class Vectorized<c10::BFloat16> : public Vectorized16<
387387
Vectorized<c10::BFloat16> neg() const {
388388
return vreinterpretq_bf16_s16(vreinterpretq_s16_bf16(values) ^ (-32768));
389389
}
390-
// Fast reciprocal is fine because we are truncating results
391-
Vectorized<c10::BFloat16> reciprocal() const {
392-
auto x = vcvtq_low_f32_bf16(values);
393-
auto y = vcvtq_high_f32_bf16(values);
394-
x = vrecpeq_f32(x);
395-
y = vrecpeq_f32(y);
396-
return vcvtq_high_bf16_f32(vcvtq_low_bf16_f32(x), y);
397-
}
398390
// Clearing the sign bit
399391
Vectorized<c10::BFloat16> abs() const {
400392
return vreinterpretq_bf16_u16(vreinterpretq_u16_bf16(values) & 0x7FFF);
401393
}
402394
#else
403395
DEFINE_UNARY_ELEMENTWISE_FUNC_VIA_FLOAT_METHOD(abs)
404396
DEFINE_UNARY_ELEMENTWISE_FUNC_VIA_FLOAT_METHOD(neg)
405-
DEFINE_UNARY_ELEMENTWISE_FUNC_VIA_FLOAT_METHOD(reciprocal)
406397
#endif
398+
DEFINE_UNARY_ELEMENTWISE_FUNC_VIA_FLOAT_METHOD(reciprocal)
407399

408400
// These functions are optimized on clang-21+
409401
#if BF16_ARITHMETIC_SUPPORTED() && (__clang_major__ >= 21)

‎aten/src/ATen/cpu/vec/vec128/vec128_reduced_precision_common_neon.h‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -299,7 +299,8 @@ struct Vectorized16 {
299299
&Vectorized<float>::lgamma);
300300
}
301301
Derived rsqrt() const {
302-
return static_cast<const Derived*>(this)->sqrt().reciprocal();
302+
return static_cast<const Derived*>(this)->map_with_vec_float_method(
303+
&Vectorized<float>::rsqrt);
303304
}
304305
Derived pow(const Derived& exp) const {
305306
return static_cast<const Derived*>(this)->map2_with_vec_float_method(

‎aten/src/ATen/test/vec_test_all_types.cpp‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,10 @@ namespace {
2828
class SqrtAndReciprocal : public ::testing::Test {};
2929
template <typename T>
3030
class SqrtAndReciprocalReal : public ::testing::Test {};
31+
#if defined(__aarch64__) && !defined(CPU_CAPABILITY_SVE256)
32+
template <typename T>
33+
class ReciprocalReducedFloat : public ::testing::Test {};
34+
#endif
3135
template <typename T>
3236
class FractionAndRemainderReal : public ::testing::Test {};
3337
template <typename T>
@@ -106,6 +110,9 @@ namespace {
106110
TYPED_TEST_SUITE(Rounding, RealFloatTestedTypes);
107111
TYPED_TEST_SUITE(SqrtAndReciprocal, FloatTestedTypes);
108112
TYPED_TEST_SUITE(SqrtAndReciprocalReal, RealFloatTestedTypes);
113+
#if defined(__aarch64__) && !defined(CPU_CAPABILITY_SVE256)
114+
TYPED_TEST_SUITE(ReciprocalReducedFloat, ReducedFloatTestedTypes);
115+
#endif
109116
TYPED_TEST_SUITE(FractionAndRemainderReal, RealFloatTestedTypes);
110117
TYPED_TEST_SUITE(Trigonometric, RealFloatTestedTypes);
111118
TYPED_TEST_SUITE(ErrorFunctions, RealFloatTestedTypes);
@@ -360,6 +367,53 @@ namespace {
360367
createDefaultUnaryTestCase<vec>(TestSeed()),
361368
RESOLVE_OVERLOAD(filter_zero));
362369
}
370+
#if defined(__aarch64__) && !defined(CPU_CAPABILITY_SVE256)
371+
// Checks every positive and negative normal input against the exact result.
372+
// Results are computed in fp32 and rounded once, so they can exceed half a
373+
// ULP only by the small double rounding margin allowed here.
374+
template <typename vec, typename Exact, typename Op>
375+
void test_reduced_float_accuracy(bool positive_only, Exact exact, Op op) {
376+
using T = ValueType<vec>;
377+
constexpr int digits = std::numeric_limits<T>::digits;
378+
const double min_normal = static_cast<float>(std::numeric_limits<T>::min());
379+
const double max_finite = static_cast<float>(std::numeric_limits<T>::max());
380+
const auto n = static_cast<size_t>(vec::size());
381+
std::vector<uint16_t> bits(n);
382+
std::vector<T> out(n);
383+
for (uint32_t i = 0; i < 65536; i++) {
384+
std::fill(bits.begin(), bits.end(), static_cast<uint16_t>(i));
385+
T in;
386+
std::memcpy(&in, bits.data(), sizeof(T));
387+
const double x = static_cast<float>(in);
388+
if (!std::isfinite(x) || std::abs(x) < min_normal || (positive_only && x < 0)) {
389+
continue;
390+
}
391+
const double expected = exact(x);
392+
if (std::abs(expected) < min_normal || std::abs(expected) > max_finite) {
393+
continue;
394+
}
395+
op(vec::loadu(bits.data())).store(out.data());
396+
const double got = static_cast<float>(out[0]);
397+
const double ulp = std::ldexp(1.0, std::ilogb(expected) - (digits - 1));
398+
ASSERT_LE(std::abs(got - expected), (0.5 + 1e-3) * ulp)
399+
<< "input " << x << " got " << got << " expected " << expected;
400+
}
401+
}
402+
TYPED_TEST(ReciprocalReducedFloat, Reciprocal) {
403+
using vec = TypeParam;
404+
test_reduced_float_accuracy<vec>(
405+
false,
406+
[](double x) { return 1.0 / x; },
407+
[](vec v) { return v.reciprocal(); });
408+
}
409+
TYPED_TEST(ReciprocalReducedFloat, RSqrt) {
410+
using vec = TypeParam;
411+
test_reduced_float_accuracy<vec>(
412+
true,
413+
[](double x) { return 1.0 / std::sqrt(x); },
414+
[](vec v) { return v.rsqrt(); });
415+
}
416+
#endif
363417
TYPED_TEST(FractionAndRemainderReal, Frac) {
364418
using vec = TypeParam;
365419
test_unary<vec>(

0 commit comments

Comments
 (0)