Fix grad_norm unserializable tensor log failure#29212
Conversation
|
The issue seems to be that deepspeeds get_global_grad_norm() unexpectedly returns a tensor instead of a float. |
There was a problem hiding this comment.
met same error too, here is the variable print log
json_string = json.dumps(dataclasses.asdict(self), indent=2, sort_keys=True) + "\n"
"print(dataclasses.asdict(self))" results:
{...
'log_history': [
{'loss': 24.6648, 'grad_norm': tensor(1091.0826), 'learning_rate': 1e-05, 'epoch': 0.04, 'step': 5},
{'loss': 8.7945, 'grad_norm': tensor(248.9097), 'learning_rate': 1e-05, 'epoch': 0.08, 'step': 10}
], ...}
show grad_norm is a tensor
muellerzr
left a comment
muellerzr
left a comment
There was a problem hiding this comment.
Thanks, looks good to me!
|
Actually, this seems to be in direct conflict with #29158 |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
@muellerzr In this case, which PR should stand? |
That other PR seems to affect the |
|
This one should stand and be okay, the other PR will need to change, so this is fine to me |
6 participants
What does this PR do?
Fixes Issue #29207
@muellerzr
@pacman100