qa/tasks/cephadm: enable mon_cluster_log_to_file#54312
Conversation
Without cluster_log_to_file we have nothing to grep for errors: 2023-10-27T16:06:59.111 DEBUG:teuthology.orchestra.run.smithi150:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/38cc7fce-74d9-11ee-8db9-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-10-27T16:06:59.141 INFO:teuthology.orchestra.run.smithi150.stderr:grep: /var/log/ceph/38cc7fce-74d9-11ee-8db9-212e2dc638e7/ceph.log: No such file or directory Set mon_cluster_log_to_file = true. Fixes: https://tracker.ceph.com/issues/63425 Signed-off-by: Dan van der Ster <[email protected]>
| log_to_file = true | ||
| log_to_stderr = false | ||
| log to journald = false | ||
| mon cluster log to file = true |
There was a problem hiding this comment.
Here's another example this time with a list of the out files in the /var/log/ceph/ dir: http://qa-proxy.ceph.com/teuthology/teuthology-2023-10-27_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7438907/teuthology.log
You can see that ceph.log is missing.
I couldn't confirm if this cephadm.conf change makes the ceph.log appear -- can you please help testing that?
There was a problem hiding this comment.
We still need #48539 in case something else goes sideways.
@adk3798 Please prioritize adding necessary ignorelists to cephadm suite -- it's what that PR is blocked on and why I have been unstaling it for months now. Chris and I didn't want to merge something that would turn one of the major suites red, but there a limit to how long something like that can wait...
There was a problem hiding this comment.
I couldn't confirm if this cephadm.conf change makes the ceph.log appear -- can you please help testing that?
@dvanders If it does, it would turn cephadm suite red because, unlike in other suites, there are no ignorelists there.
There was a problem hiding this comment.
@idryomov At this point, if it's causing failures to be missed elsewhere, I'm okay with it being merged. I have a partially complete ignorelist PR and this will at least force me to finish it. I can deal with sifting through the failures for the time before I finish that as well.
There was a problem hiding this comment.
@adk3798 I have asked @chrisphoffman to move forward with #48539.
There was a problem hiding this comment.
@adk3798 I have asked @chrisphoffman to move forward with #48539.
This hasn't happened. I have unstaled #48539 again, but would suggest moving forward by picking up this PR and seeing if it actually makes ceph.log file appear and causes those ignorelist-related failures.
There was a problem hiding this comment.
If it does, it would turn cephadm suite red
Adam is asking for another run, but this is definitely working:
Let's get this merged!
|
This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days. |
|
jenkins test api |
|
There are a lot of failures in the rados suite from this PR. Until a PR is raised to whitelist expected warnings, a lot of main test batches are blocked. Should be a relatively straightfoward fix from the rados team, but pasting here to raise awareness. |
5 participants
Without cluster_log_to_file we have nothing to grep for errors:
2023-10-27T16:06:59.111 DEBUG:teuthology.orchestra.run.smithi150:> sudo egrep '[ERR]|[WRN]|[SEC]' /var/log/ceph/38cc7fce-74d9-11ee-8db9-212e2dc638e7/ceph.log | egrep -v '(MDS_ALL_DOWN)' | egrep -v '(MDS_UP_LESS_THAN_MAX)' | head -n 1 2023-10-27T16:06:59.141 INFO:teuthology.orchestra.run.smithi150.stderr:grep: /var/log/ceph/38cc7fce-74d9-11ee-8db9-212e2dc638e7/ceph.log: No such file or directory
Set mon_cluster_log_to_file = true.
Fixes: https://tracker.ceph.com/issues/63425
See https://pulpito.ceph.com/teuthology-2023-10-28_14:23:03-upgrade:quincy-x-reef-distro-default-smithi/7439369/ for a broken example