Feature request
store and log gradient norm in trainer
Motivation
Gradient norm is an important metric but currently the gradient norm is discarded during clipping.
Your contribution
I checked and all of the following grad_norm_clip functions return the gradient norm. We can get gradient norm without extra compute by storing the return value, and we can then log it. If this sounds good i am happy to prepare a PR.
|
if is_sagemaker_mp_enabled() and args.fp16: |
|
self.optimizer.clip_master_grads(args.max_grad_norm) |
|
elif hasattr(self.optimizer, "clip_grad_norm"): |
|
# Some optimizers (like the sharded optimizer) have a specific way to do gradient clipping |
|
self.optimizer.clip_grad_norm(args.max_grad_norm) |
|
elif hasattr(model, "clip_grad_norm_"): |
|
# Some models (like FullyShardedDDP) have a specific way to do gradient clipping |
|
model.clip_grad_norm_(args.max_grad_norm) |
|
elif self.use_apex: |
|
# Revert to normal clipping otherwise, handling Apex or full precision |
|
nn.utils.clip_grad_norm_( |
|
amp.master_params(self.optimizer), |
|
args.max_grad_norm, |
|
) |
|
else: |
|
self.accelerator.clip_grad_norm_( |
|
model.parameters(), |
|
args.max_grad_norm, |
|
) |
Feature request
store and log gradient norm in trainer
Motivation
Gradient norm is an important metric but currently the gradient norm is discarded during clipping.
Your contribution
I checked and all of the following grad_norm_clip functions return the gradient norm. We can get gradient norm without extra compute by storing the return value, and we can then log it. If this sounds good i am happy to prepare a PR.
transformers/src/transformers/trainer.py
Lines 1918 to 1936 in 32ec734