These -1e9 constants are too large for fp16 training https://github.com/huggingface/transformers/blob/df735d1317994e366ab0edff6c55930e18912b7c/src/transformers/models/t5/modeling_tf_t5.py#L728 https://github.com/huggingface/transformers/blob/df735d1317994e366ab0edff6c55930e18912b7c/src/transformers/models/t5/modeling_tf_t5.py#L746 Maybe they should be made configurable?
These -1e9 constants are too large for fp16 training
transformers/src/transformers/models/t5/modeling_tf_t5.py
Line 728 in df735d1
transformers/src/transformers/models/t5/modeling_tf_t5.py
Line 746 in df735d1
Maybe they should be made configurable?