-
Notifications
You must be signed in to change notification settings - Fork 4.7k
[wall_clock_breakdown] always log stats when enabled #7617
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Signed-off-by: Stas Bekman <[email protected]>
|
I think your point is that what we show with |
That's correct, Masahiro. It's a data dump.
I'm not sure what you mean, please clarify or exemplify what you mean - perhaps in the context of this PR if it'd help? |
Sorry for the typo, I meant making a new function |
|
Thank you for clarifying, Masahiro. I'd be happy to improve this further. Since (or using and using it at the calling point? unless you object to the mod to Either way works for me. |
|
Thank you for your response, @stas00. I think the first approach should be sufficient though the second approach is cleaner. |
|
If we don't do it now, we won't do it, let's complete this while it's "hot" - @sfc-gh-truwase, do you have a preference for the 2 options I proposed 2 comments up as a follow up to Masahiro's suggestion? |
|
@stas00 let's go with the second approach, which I agree is cleaner. |
|
please have a look #7621 |
currently when main logger is WARN level, `wall_clock_breakdown: true` never logs - which is invalid as it disables this crucial at times functionality. Plus I think we have a disconnect somewhere since the recently added `--log_level` flag doesn't seem to change this logger's level. The future plan is to be able to have different log levels for different modules, but for now just use `print` if `wall_clock_breakdown` is `True`, so this functionality is not log-level dependent. `print` is also less noisy than the logger, because of the long prefix generated by the latter, which is of no value to the user since we print stats and not code related logs, so the printed results are easier to digest. Signed-off-by: Stas Bekman <[email protected]>
a refactor follow up to #7617 as suggested by @tohtana to create 2 independent utils, sharing the main logic via another util. Signed-off-by: Stas Bekman <[email protected]>
a refactor follow up to deepspeedai#7617 as suggested by @tohtana to create 2 independent utils, sharing the main logic via another util. Signed-off-by: Stas Bekman <[email protected]> Signed-off-by: Ma, Liangliang <[email protected]>
currently when main logger is WARN level,
wall_clock_breakdown: truenever logs - which is invalid as it disables this crucial at times functionality. Plus I think we have a disconnect somewhere since the recently added--log_levelflag doesn't seem to change this logger's level.The future plan is to be able to have different log levels for different modules, but for now just use
printifwall_clock_breakdownisTrue, so this functionality is not log-level dependent.printis also less noisy than the logger, because of the long prefix generated by the latter, which is of no value to the user since we print stats and not code related logs, so the printed results are easier to digest.