Repository navigation
docs: document the reduced-precision float32 default and MLX_ENABLE_TF32 - #3894
Conversation
| reduced-precision path is never taken, not because the setting is | ||
| ignored. | ||
|
|
||
| TF32-class math keeps the ``float32`` exponent range but rounds the |
There was a problem hiding this comment.
The technical details should be ignored since Apple silicon might not do exactly same.
There was a problem hiding this comment.
Dropped, along with the TF32-format paragraph. Agreed that the docs
shouldn't assert equivalence between the Apple silicon path and NVIDIA's
TF32 -- that was inferred from measurements rather than from anything
either vendor specifies, and it is exactly the kind of claim that goes
stale silently.
|
|
||
| # float32 matmuls now run at full float32 precision | ||
|
|
||
| A few details worth knowing: |
There was a problem hiding this comment.
I prefer omitting the details because they could change at any time and we might not remember to update the docs here.
There was a problem hiding this comment.
Dropped. I also removed the in-process os.environ example, which is what
made the latching caveat necessary in the first place -- the page now
shows only the launch-time form, so there is no first-use ordering left
to explain.
Review feedback on ml-explore#3894: per-backend hardware descriptions and implementation details (first-use latching, the matvec shape exception, complex64) can change without this page being updated. Keep only what holds regardless of backend -- that float32 matmul-family ops may run at reduced precision by default, and that MLX_ENABLE_TF32=0 opts out. Removing the in-process os.environ example makes the latching caveat unnecessary: the launch-time form has no first-use ordering to explain. Co-Authored-By: Claude Opus 5 <[email protected]>
stoyoda0012-cyber
changed the title
|
Reduced the page to the two things that hold regardless of backend and |
float32 matmul-family ops run at TF32-class reduced precision by default on both current-generation backends (CUDA tensor cores; the M5 NAX — earlier Apple silicon runs full float32), and the MLX_ENABLE_TF32 switch was previously undocumented. Adds a short Numerical Precision usage page: what the default costs (measured ~1e-4..1e-3 vs ~2e-7 relative error), how to opt out, the first-use latching, the matvec shape exception, the CUDA complex64 case, and that half-precision inputs are unaffected. Follow-up promised in ml-explore#3860.
- Metal: reduced precision applies on the M5 generation and newer on macOS 26.2 or later; state explicitly that pre-gen-17 machines never take the reduced-precision path, so the opt-out reads as a no-op there without being ignored - Add a bullet noting the default applies to anything that lowers to matmul-family ops, e.g. attention fallbacks composed from plain GEMMs Scope measurements: ml-explore#3860 (three-generation matrix by mabaeyens, PhilipJohnBasile, katlun-lgtm). Co-Authored-By: Claude Fable 5 <[email protected]>
The bullet stating that float16/bfloat16 inputs are unaffected could be read as "those dtypes have no reduced-precision behaviour at all". State the boundary instead: the setting applies to float32 only, and reduced-precision behaviour involving fp16/bf16 inputs, where it exists, has other causes and is not controlled by MLX_ENABLE_TF32. Suggested by mabaeyens in ml-explore#3860. Co-Authored-By: Claude Fable 5 <[email protected]>
Review feedback on ml-explore#3894: per-backend hardware descriptions and implementation details (first-use latching, the matvec shape exception, complex64) can change without this page being updated. Keep only what holds regardless of backend -- that float32 matmul-family ops may run at reduced precision by default, and that MLX_ENABLE_TF32=0 opts out. Removing the in-process os.environ example makes the latching caveat unnecessary: the launch-time form has no first-use ordering to explain. Co-Authored-By: Claude Opus 5 <[email protected]>
a19966a to
261a349
Compare
Documents the reduced-precision default discussed in #3860.
Adds a short "Numerical Precision" usage page stating two things:
matmul, convolution and attention) may run at reduced precision by
default on hardware with dedicated matrix-multiplication units;
MLX_ENABLE_TF32=0at launch keeps them in full float32.Per review, per-backend descriptions and implementation-specific
behaviour are left out, so the page does not need updating when either
changes.
Testing
Built the docs with
sphinx-build -b htmlagainstdocs/requirements.txt: the page renders, is reachable from the Usagetoctree, and none of the six build warnings originate from it (they are
the pre-existing
nn.distributedautosummary imports against the pipwheel, the doxygen-generated C++ group, and the
printoptionsstub).