mds: emit warning with estimated replay time by vshankar · Pull Request #55616 · ceph/ceph

vshankar · 2024-02-16T12:05:52Z

It is helpful to get an estimate of how much time it might take to finish replaying the journal.

Fixes: https://tracker.ceph.com/issues/61863

Supersedes #52527

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows
jenkins test rook e2e

vshankar · 2024-02-17T09:13:48Z

jenkins test make check

vshankar · 2024-02-27T04:36:30Z

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20240226.124304

vshankar · 2024-02-27T05:59:36Z

https://pulpito.ceph.com/vshankar-2024-02-27_04:05:06-fs-wip-vshankar-testing-20240226.124304-testing-default-smithi/

* refs/pull/55616/head: mds: emit warning with estimated replay time

vshankar · 2024-03-04T10:15:46Z

Reruning tests with #55909 and #55908 included. See - https://pulpito.ceph.com/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/

vshankar · 2024-03-06T04:29:53Z

Test failure - https://pulpito.ceph.com/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7580900

vshankar · 2024-03-06T04:30:03Z

(dropping from current batch)

src/mds/MDLog.cc

src/mds/mdstypes.h

src/mds/Beacon.cc

qa/tasks/cephfs/test_failover.py

batrick

Also, I'd like to see the commit broken into:

mds: ...
qa: ...
PendingReleaseNotes,doc: ...

(we need release note)

vshankar · 2024-04-30T04:32:48Z

Also, I'd like to see the commit broken into:

mds: ...

qa: ...

PendingReleaseNotes,doc: ...

(we need release note)

@batrick - This PR is mostly @manishym changes that I took over, so many comments from the original PR probably aren't incorporated. I'll plan to visit this soon.

github-actions · 2024-05-07T19:58:38Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

vshankar · 2024-06-05T05:02:30Z

jenkins retest this please

batrick · 2024-06-10T20:34:06Z

jenkins test make check

batrick · 2024-06-10T20:34:11Z

jenkins test api

github-actions · 2024-08-11T15:01:36Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

github-actions · 2024-09-10T17:01:30Z

This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution!

Comment addressed.

vshankar · 2024-11-18T05:55:25Z

This PR is under test in https://tracker.ceph.com/issues/68968.

* refs/pull/55616/head: PendingReleaseNotes: add note for replay completion warning qa: test to verify `MDS_ESTIMATED_REPLAY_TIME` warning doc: add a note for `MDS_ESTIMATED_REPLAY_TIME` MDS warning mds: emit warning for estinated replay time Reviewed-by: Patrick Donnelly <[email protected]>

github-actions · 2024-11-29T10:50:52Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

If replay runs more than 30 seconds, emit a warning with estimated replay completion time. Fixes: https://tracker.ceph.com/issues/61863 Signed-off-by: Manish M Yathnalli <[email protected]> Signed-off-by: Venky Shankar <[email protected]>

Signed-off-by: Venky Shankar <[email protected]>

* refs/pull/55616/head: PendingReleaseNotes: add note for replay completion warning qa: test to verify `MDS_ESTIMATED_REPLAY_TIME` warning doc: add a note for `MDS_ESTIMATED_REPLAY_TIME` MDS warning mds: emit warning for estinated replay time Reviewed-by: Patrick Donnelly <[email protected]>

vshankar · 2024-12-24T10:24:40Z

https://tracker.ceph.com/projects/cephfs/wiki/Main#wip-vshankar-testing-20241219063429-debug

mchangir · 2024-12-26T12:08:55Z

src/mds/MDLog.cc

+  auto total_bytes = write_pos - trimmed_pos;
+  double percent_complete = ((double)(read_pos - trimmed_pos)) / (double)total_bytes;
+  auto elapsed_time = std::chrono::duration_cast<std::chrono::seconds>
+    (ceph::coarse_mono_clock::now() - replay_start_time);
+  auto time = ((1 - percent_complete) / percent_complete) * elapsed_time;
+
+  dout(20) << __func__ << "percent_complete=" << percent_complete
+	   << ", elapsed_time=" << elapsed_time
+	   << ", estimated_time=" << std::chrono::round<std::chrono::seconds>(time)
+	   << dendl;
+
+  estimated_time.percent_complete = percent_complete * 100;
+  estimated_time.elapsed_time = elapsed_time;
+  estimated_time.estimated_time = std::chrono::round<std::chrono::seconds>(time);
+  dout(20) << __func__ << "estimated_time.percent_complete=" << estimated_time.percent_complete
+	   << ", estimated_time.elapsed_time=" << estimated_time.elapsed_time
+	   << ", estimated_time.estimated_time=" << estimated_time.estimated_time
+	   << dendl;


Why are there 2 log lines with almost the same info ?

For debugging when I was trying to get the regex prefect. I'll remove it in a follow up PR once this change is merge (of course if nothing else stands out).

mchangir

LGTM

vshankar added the cephfs Ceph File System label Feb 16, 2024

vshankar requested a review from a team February 16, 2024 12:05

vshankar requested a review from a team as a code owner February 16, 2024 12:05

github-actions bot added common documentation tests labels Feb 16, 2024

vshankar mentioned this pull request Feb 16, 2024

mds: emit warning with estimated replay time #52527

Closed

14 tasks

vshankar force-pushed the log_estimated_journal_replay_time branch from fbc84d0 to b5246b4 Compare February 17, 2024 08:19

vshankar force-pushed the log_estimated_journal_replay_time branch from b5246b4 to 7f63a76 Compare February 19, 2024 04:20

vshankar added the wip-vshankar-testing3 label Feb 19, 2024

vshankar added a commit to vshankar/ceph that referenced this pull request Mar 4, 2024

Merge PR ceph#55616 into wip-vshankar-testing-20240304.042522

bd92dc1

* refs/pull/55616/head: mds: emit warning with estimated replay time

vshankar removed the wip-vshankar-testing3 label Mar 6, 2024

batrick requested changes Apr 23, 2024

View reviewed changes

src/mds/MDLog.cc Outdated Show resolved Hide resolved

src/mds/mdstypes.h Outdated Show resolved Hide resolved

src/mds/mdstypes.h Outdated Show resolved Hide resolved

src/mds/Beacon.cc Outdated Show resolved Hide resolved

qa/tasks/cephfs/test_failover.py Outdated Show resolved Hide resolved

batrick requested changes Apr 23, 2024

View reviewed changes

github-actions bot added the needs-rebase label May 7, 2024

vshankar force-pushed the log_estimated_journal_replay_time branch from 7f63a76 to 9450bab Compare June 4, 2024 07:30

github-actions bot removed the needs-rebase label Jun 4, 2024

vshankar requested a review from batrick June 5, 2024 05:02

batrick approved these changes Jun 10, 2024

View reviewed changes

github-actions bot added the stale label Aug 11, 2024

github-actions bot closed this Sep 10, 2024

vshankar reopened this Nov 12, 2024

github-actions bot removed the stale label Nov 12, 2024

vshankar force-pushed the log_estimated_journal_replay_time branch from 2f78af0 to 0aa487c Compare November 12, 2024 07:50

github-actions bot removed the needs-rebase label Nov 12, 2024

vshankar added the wip-vshankar-testing label Nov 18, 2024

vshankar force-pushed the log_estimated_journal_replay_time branch from 0aa487c to e5a9be4 Compare November 29, 2024 10:50

vshankar requested review from a team as code owners November 29, 2024 10:50

github-actions bot added the needs-rebase label Nov 29, 2024

github-actions bot added the core label Nov 29, 2024

vshankar added 4 commits November 29, 2024 16:21

doc: add a note for MDS_ESTIMATED_REPLAY_TIME MDS warning

7614893

Signed-off-by: Venky Shankar <[email protected]>

qa: test to verify MDS_ESTIMATED_REPLAY_TIME warning

19db699

Signed-off-by: Venky Shankar <[email protected]>

PendingReleaseNotes: add note for replay completion warning

65acd39

Signed-off-by: Venky Shankar <[email protected]>

vshankar force-pushed the log_estimated_journal_replay_time branch from e5a9be4 to 65acd39 Compare November 29, 2024 10:51

github-actions bot removed the needs-rebase label Nov 29, 2024

mchangir reviewed Dec 26, 2024

View reviewed changes

mchangir approved these changes Dec 27, 2024

View reviewed changes

vshankar merged commit 2215d55 into ceph:main Dec 27, 2024

vshankar removed the wip-vshankar-testing label Dec 27, 2024

Comments

Conversation

vshankar commented Feb 16, 2024

Contribution Guidelines

Checklist

Uh oh!

vshankar commented Feb 17, 2024

Uh oh!

vshankar commented Feb 27, 2024

Uh oh!

vshankar commented Feb 27, 2024

Uh oh!

vshankar commented Mar 4, 2024

Uh oh!

vshankar commented Mar 6, 2024

Uh oh!

vshankar commented Mar 6, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

batrick left a comment

Choose a reason for hiding this comment

Uh oh!

vshankar commented Apr 30, 2024

Uh oh!

github-actions bot commented May 7, 2024

Uh oh!

vshankar commented Jun 5, 2024

Uh oh!

batrick commented Jun 10, 2024

Uh oh!

batrick commented Jun 10, 2024

Uh oh!

github-actions bot commented Aug 11, 2024

Uh oh!

github-actions bot commented Sep 10, 2024

Uh oh!

vshankar commented Nov 18, 2024

Uh oh!

github-actions bot commented Nov 29, 2024

Uh oh!

vshankar commented Dec 24, 2024

Uh oh!

mchangir Dec 26, 2024

Choose a reason for hiding this comment

Uh oh!

vshankar Dec 27, 2024

Choose a reason for hiding this comment

Uh oh!

mchangir left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants