Skip to content

Add timestamp to flaky restart monitor test#5727

Merged
estesp merged 2 commits intocontainerd:mainfrom
dmcgowan:test-monitor-restart-grace
Jul 13, 2021
Merged

Add timestamp to flaky restart monitor test#5727
estesp merged 2 commits intocontainerd:mainfrom
dmcgowan:test-monitor-restart-grace

Conversation

@dmcgowan
Copy link
Copy Markdown
Member

@dmcgowan dmcgowan commented Jul 13, 2021

This adds the timestamp to the final log to help debug this flaky test. See logs below

    default:     restart_monitor_linux_test.go:92: 2021-07-13 16:21:49.450742838 +0000 UTC m=+145.976532745: status={"stopped" '\u0089' "2021-07-13 16:21:49.446496309 +0000 UTC"}
    default:     restart_monitor_linux_test.go:92: 2021-07-13 16:21:50.459980678 +0000 UTC m=+146.985770567: status={"stopped" '\u0089' "2021-07-13 16:21:49.446496309 +0000 UTC"}
    default:     restart_monitor_linux_test.go:92: 2021-07-13 16:21:51.467907321 +0000 UTC m=+147.993697215: status={"stopped" '\u0089' "2021-07-13 16:21:49.446496309 +0000 UTC"}
    default:     restart_monitor_linux_test.go:92: 2021-07-13 16:21:52.47103632 +0000 UTC m=+148.996826201: status={"stopped" '\u0089' "2021-07-13 16:21:49.446496309 +0000 UTC"}
    default:     restart_monitor_linux_test.go:92: 2021-07-13 16:21:53.478105735 +0000 UTC m=+150.003895616: status={"stopped" '\u0089' "2021-07-13 16:21:49.446496309 +0000 UTC"}
    default:     restart_monitor_linux_test.go:92: 2021-07-13 16:21:54.507516616 +0000 UTC m=+151.033306499: status={"stopped" '\u0089' "2021-07-13 16:21:49.446496309 +0000 UTC"}
    default:     restart_monitor_linux_test.go:92: 2021-07-13 16:21:55.511818588 +0000 UTC m=+152.037608464: status={"stopped" '\u0089' "2021-07-13 16:21:49.446496309 +0000 UTC"}
    default:     restart_monitor_linux_test.go:92: 2021-07-13 16:21:56.520061366 +0000 UTC m=+153.045851249: status={"stopped" '\u0089' "2021-07-13 16:21:49.446496309 +0000 UTC"}
    default:     restart_monitor_linux_test.go:102: the task was not restarted in 10s + 1s

The last log should not have run until after 16:22:0 and additional checks at 1 second intervals in between. This was seen on the VM test which sometimes has had pauses, but this will help diagnose how long the delay is.

@theopenlab-ci
Copy link
Copy Markdown

theopenlab-ci Bot commented Jul 13, 2021

Build succeeded.

@dmcgowan dmcgowan force-pushed the test-monitor-restart-grace branch from 8d1f8b9 to 18321f5 Compare July 13, 2021 19:08
@dmcgowan
Copy link
Copy Markdown
Member Author

dmcgowan commented Jul 13, 2021

Added a commit to move the check before the sleep, ensures this will also help ensure that the loop gets run once after the deadline even if there is a pause

@theopenlab-ci
Copy link
Copy Markdown

theopenlab-ci Bot commented Jul 13, 2021

Build succeeded.

estesp
estesp approved these changes Jul 13, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@estesp estesp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@estesp estesp merged commit 34fb8d8 into containerd:main Jul 13, 2021
@dmcgowan dmcgowan deleted the test-monitor-restart-grace branch April 20, 2024 00:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kzys kzys kzys approved these changes

@estesp estesp estesp approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dmcgowan @kzys @estesp