Repository navigation
Commit e254f82
[AArch64] Fix BF16 reciprocal and BF16/Half rsqrt rounding errors (#203940)
## Linked issue or supporting maintainer
Fixes #203939
## BC-breaking?
No. Some BF16 and Half results on AArch64 change by up to about 1 ULP, towards the exact value.
## Summary (human written only)
BF16 `reciprocal` and BF16/Half `rsqrt` on AArch64 were more than 1 ULP off and disagreed with `1/x` and across CPU capabilities; this computes both in fp32 and rounds once, as AVX2 does, at the cost of some speed.
## Agent details (if applicable)
> Drafted with Claude Code; the code, tests and the runs below were done with it. Pushed as `0z5a:fix/aarch64-bf16-reciprocal-rsqrt`.
>
> - `reciprocal` no longer has the `vrecpeq_f32` path, so it always goes through `Vectorized<float>::reciprocal`. `rsqrt` in the shared reduced-precision base now goes through `Vectorized<float>::rsqrt`, which covers BF16 and Half. Half `reciprocal` is unchanged.
> - New `ReciprocalReducedFloat` tests in `vec_test_all_types.cpp` check every normal input against the exact result, allowing 0.501 ULP for double rounding. They only build for the AArch64 DEFAULT and SVE128 paths: x86 DEFAULT still has the `rsqrt` error (see below), and SVE256 could not be run.
>
> Largest error on a CPU build of `d60701e`, GH200 (Neoverse V2):
>
> | Operation | SVE128 before | SVE128 after | DEFAULT before | DEFAULT after |
> |---|---|---|---|---|
> | BF16 `reciprocal` | 1.18 ULP | 0.50 ULP | 0.50 ULP | 0.50 ULP |
> | BF16 `rsqrt` | 1.70 ULP | 0.50 ULP | 1.14 ULP | 0.50 ULP |
> | Half `rsqrt` | 1.33 ULP | 0.50 ULP | 1.33 ULP | 0.50 ULP |
>
> After the fix, `torch.reciprocal(x)` equals `tensor(1.)/x` on all finite BF16 and Half inputs, and DEFAULT and SVE128 give identical results for `reciprocal` and `rsqrt` on those inputs (`rsqrt`: positive finite). Those results are also identical to `2.16.0.dev20261107+cpu` on x86 AVX2 and AVX512 (Xeon Gold 5120, hash of all outputs compared).
>
> Time per element, one thread, 1M elements:
>
> | Operation | SVE128 before | SVE128 after | DEFAULT before | DEFAULT after |
> |---|---|---|---|---|
> | BF16 `reciprocal` | 0.176 ns | 0.257 ns | 0.414 ns | 0.418 ns |
> | BF16 `rsqrt` | 0.337 ns | 0.519 ns | 0.930 ns | 0.618 ns |
> | Half `rsqrt` | 0.371 ns | 0.515 ns | 0.541 ns | 0.512 ns |
>
> `reciprocal` could keep more of its speed with `vrecpeq_f32` plus one `vrecpsq_f32` step: 0.180 ns against 0.228 ns for the divide in a standalone intrinsics program, with the same 0.50 ULP on normal inputs. I did not use it because it returned `-inf` for very small positive subnormals, where the divide returns `inf`. `rsqrt` has the same option: `vrsqrteq_f32` plus one step runs at 0.213 ns, faster than today, with the same 0.50 ULP on normal inputs, but it returns `-inf` for very small subnormals whose answer is about 1e20.
>
> `vec_test_all_types`:
>
> - The new tests fail before the fix (DEFAULT: `RSqrt` for BF16 and Half; SVE128: also BF16 `Reciprocal`) and pass after it.
> - The same tests fail before and after the fix, so they are unrelated: `Arithmetic/3.Multiplication` on DEFAULT and SVE128, and `BitwiseFloatsAdditional/3.{Fmadd,FmaddVecN,Fmsub}` on DEFAULT.
>
> Not covered: x86 DEFAULT `rsqrt` measures 1.14 ULP for BF16 and 1.33 ULP for Half. It uses the generic path, which this PR does not change. x86 was only run from a nightly wheel, so the new tests and the fixed build were not run there. SVE256 was not run: this machine has 128-bit SVE, so the SVE256 test binary fails and crashes with or without the fix, and the new tests are now excluded from it. That exclusion was added after the runs above; it does not change what the DEFAULT and SVE128 binaries compile. `spin fixlint` was not run.
Pull Request resolved: #203940
Approved by: https://github.com/Nicoshev, https://github.com/ezyang1 parent 392f184 commit e254f82
3 files changed
Lines changed: 57 additions & 10 deletions
File tree
- aten/src/ATen
- cpu/vec/vec128
- test
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
387 | 387 | | |
388 | 388 | | |
389 | 389 | | |
390 | | - | |
391 | | - | |
392 | | - | |
393 | | - | |
394 | | - | |
395 | | - | |
396 | | - | |
397 | | - | |
398 | 390 | | |
399 | 391 | | |
400 | 392 | | |
401 | 393 | | |
402 | 394 | | |
403 | 395 | | |
404 | 396 | | |
405 | | - | |
406 | 397 | | |
| 398 | + | |
407 | 399 | | |
408 | 400 | | |
409 | 401 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
299 | 299 | | |
300 | 300 | | |
301 | 301 | | |
302 | | - | |
| 302 | + | |
| 303 | + | |
303 | 304 | | |
304 | 305 | | |
305 | 306 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
31 | 35 | | |
32 | 36 | | |
33 | 37 | | |
| |||
106 | 110 | | |
107 | 111 | | |
108 | 112 | | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
109 | 116 | | |
110 | 117 | | |
111 | 118 | | |
| |||
360 | 367 | | |
361 | 368 | | |
362 | 369 | | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
363 | 417 | | |
364 | 418 | | |
365 | 419 | | |
| |||
0 commit comments