Skip to content

Conversation

@stas00
Copy link
Collaborator

@stas00 stas00 commented Oct 2, 2025

currently when main logger is WARN level, wall_clock_breakdown: true never logs - which is invalid as it disables this crucial at times functionality. Plus I think we have a disconnect somewhere since the recently added --log_level flag doesn't seem to change this logger's level.

The future plan is to be able to have different log levels for different modules, but for now just use print if wall_clock_breakdown is True, so this functionality is not log-level dependent.

print is also less noisy than the logger, because of the long prefix generated by the latter, which is of no value to the user since we print stats and not code related logs, so the printed results are easier to digest.

@stas00 stas00 enabled auto-merge (squash) October 2, 2025 19:54
@tohtana
Copy link
Collaborator

tohtana commented Oct 2, 2025

I think your point is that what we show with wall_clock_breakdown: true is not a log. That’s why we don’t want it to be affected by the log level. If that’s the case, how about introducing something like print_dict?

@stas00
Copy link
Collaborator Author

stas00 commented Oct 2, 2025

I think your point is that what we show with wall_clock_breakdown: true is not a log. That’s why we don’t want it to be affected by the log level.

That's correct, Masahiro. It's a data dump.

If that’s the case, how about introducing something like print_dict?

I'm not sure what you mean, please clarify or exemplify what you mean - perhaps in the context of this PR if it'd help?

@stas00 stas00 merged commit 9cbd3ed into master Oct 2, 2025
14 of 16 checks passed
@stas00 stas00 deleted the stas/wall_clock_breakdown_log branch October 2, 2025 23:08
@tohtana
Copy link
Collaborator

tohtana commented Oct 2, 2025

I'm not sure what you mean, please clarify or exemplify what you mean - perhaps in the context of this PR if it'd help?

Sorry for the typo, I meant making a new function print_dist instead of log_dist to clarify it is not a log output.
If log_dist's default behavior is using logger, it is not very clear what use_logger==False means.

@stas00
Copy link
Collaborator Author

stas00 commented Oct 2, 2025

Thank you for clarifying, Masahiro. I'd be happy to improve this further.

Since log_dist has so much logic in it, replicating it is probably not a great idea, perhaps adding a wrapper then? so keeping the code as I suggested and then adding:

def print_dist(*args, **kwargs):
    """ same logic as log_dist but uses print instead of logger """
    log_dist(*args, **kwargs, use_logger==False)

(or using partial)

and using it at the calling point?

unless you object to the mod to log_dist, in which case we should rip out all of the log_dist logic except the last line into a new helper util and then add 2 wrappers instead, that will use this util?

def print_dist(...):
    string = helper(...)
    if string is not None: print(string)

def log_dist(...):
    string = helper(...)
    if string is not None: logger.log(string)

Either way works for me.

@tohtana
Copy link
Collaborator

tohtana commented Oct 3, 2025

Thank you for your response, @stas00.

I think the first approach should be sufficient though the second approach is cleaner.
Also, I don't think this is urgent.

@stas00
Copy link
Collaborator Author

stas00 commented Oct 3, 2025

If we don't do it now, we won't do it, let's complete this while it's "hot" - @sfc-gh-truwase, do you have a preference for the 2 options I proposed 2 comments up as a follow up to Masahiro's suggestion?

@sfc-gh-truwase
Copy link
Collaborator

@stas00 let's go with the second approach, which I agree is cleaner.

@stas00 stas00 mentioned this pull request Oct 3, 2025
@stas00
Copy link
Collaborator Author

stas00 commented Oct 3, 2025

please have a look #7621

mauryaavinash95 pushed a commit to DataStates/DeepSpeed that referenced this pull request Oct 4, 2025
currently when main logger is WARN level, `wall_clock_breakdown: true`
never logs - which is invalid as it disables this crucial at times
functionality. Plus I think we have a disconnect somewhere since the
recently added `--log_level` flag doesn't seem to change this logger's
level.

The future plan is to be able to have different log levels for different
modules, but for now just use `print` if `wall_clock_breakdown` is
`True`, so this functionality is not log-level dependent.

`print` is also less noisy than the logger, because of the long prefix
generated by the latter, which is of no value to the user since we print
stats and not code related logs, so the printed results are easier to
digest.

Signed-off-by: Stas Bekman <[email protected]>
stas00 added a commit that referenced this pull request Oct 4, 2025
a refactor follow up to
#7617 as suggested by
@tohtana to create 2 independent utils, sharing the main logic via
another util.

Signed-off-by: Stas Bekman <[email protected]>
Liangliang-Ma pushed a commit to Liangliang-Ma/DeepSpeed that referenced this pull request Oct 13, 2025
a refactor follow up to
deepspeedai#7617 as suggested by
@tohtana to create 2 independent utils, sharing the main logic via
another util.

Signed-off-by: Stas Bekman <[email protected]>
Signed-off-by: Ma, Liangliang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants