Repository navigation
Using mixed precision causes incorrect loss, metric values #45568
Description
Tensorflow version : 2.4.0-rc3, compiled from source
GPU : RTX 3080 10GB
CUDA / CUDNN : 11.1 / 8
bazel version : 3.1.0
Windows 10
I decided to mixed precision to speed up the training, but some issues were.
I used this code:
policy = keras.mixed_precision.Policy('mixed_float16')
keras.mixed_precision.set_global_policy(policy)
...
model.compile(optimizer = keras.mixed_precision.LossScaleOptimizer(optimizers.Adam(learning_rate=1e-2))
loss = dice_loss
metric = [dice_coefficient]
)
Training didn't show any nan loss and metric. But when evaluating the model or each epoch ended, val loss and metric were incorrect(they were fixed over epochs). The colab notebook is here
etc:
- The values of
model.predict(x)andmodel(x)are different. When the former was nan, the latter seemed normal.
Activity
[-]Using mixed precision cause incorrect loss, metric values[/-][+]Using mixed precision causes incorrect loss, metric values[/+]google-ml-butler commentedon Dec 10, 2020
Are you satisfied with the resolution of your issue?
Yes
No
amahendrakar commentedon Dec 10, 2020
On running the code with TF v2.2, the
val_lossvalue is1.0001.Was able to reproduce the issue with TF v2.3, with which the
val_lossisnan.Whereas with TF v2.4.0-rc3 and TF-nightly, the code runs indefinitely. Please check the linked gist for reference. Thanks!
25 remaining items
PR #45597: [XLA:GPU] Add SKIP_PROFILED command buffer update mode
PR #45597: [XLA:GPU] Add SKIP_PROFILED command buffer update mode
PR #45597: [XLA:GPU] Add SKIP_PROFILED command buffer update mode
PR #45597: [XLA:GPU] Add SKIP_PROFILED command buffer update mode
PR #45597: [XLA:GPU] Add SKIP_PROFILED command buffer update mode
PR #45597: [XLA:GPU] Add SKIP_PROFILED command buffer update mode
PR #45597: [XLA:GPU] Add SKIP_PROFILED command buffer update mode
PR #45597: [XLA:GPU] Add SKIP_PROFILED command buffer update mode
PR #45597: [XLA:GPU] Add SKIP_PROFILED command buffer update mode
PR #45597: [XLA:GPU] Add SKIP_PROFILED command buffer update mode
PR #45597: [XLA:GPU] Add SKIP_PROFILED command buffer update mode
PR #45597: [XLA:GPU] Add SKIP_PROFILED command buffer update mode
PR #45597: [XLA:GPU] Add SKIP_PROFILED command buffer update mode
PR #45597: [XLA:GPU] Add SKIP_PROFILED command buffer update mode
PR #45597: [XLA:GPU] Add SKIP_PROFILED command buffer update mode