Skip to content

[DNM] cephadm: Has the podman journald log driver a problem with line breaks?#39547

Closed
sebastian-philipp wants to merge 1 commit intoceph:masterfrom
sebastian-philipp:cephadm-podman-jornald-line-break-bug
Closed

[DNM] cephadm: Has the podman journald log driver a problem with line breaks?#39547
sebastian-philipp wants to merge 1 commit intoceph:masterfrom
sebastian-philipp:cephadm-podman-jornald-line-break-bug

Conversation

@sebastian-philipp
Copy link
Contributor

@sebastian-philipp sebastian-philipp commented Feb 18, 2021

Revert "cephadm: configure journald as the logdriver"

This reverts commit 7c44f21.

Details: https://tracker.ceph.com/issues/49551

Checklist

  • References tracker ticket
  • Updates documentation if necessary
  • Includes tests for new functionality or reproducer for bug

Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox

… breaks?

Revert "cephadm: configure journald as the logdriver"

This reverts commit 7c44f21.
@sebastian-philipp
Copy link
Contributor Author

let's see if this is related to this strange logging behavior:

2021-02-18T01:28:45.704 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]:
2021-02-18T01:28:45.704 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: pgmap v258: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s
2021-02-18T01:28:45.705 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]:
2021-02-18T01:28:45.705 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: cluster
2021-02-18T01:28:45.705 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]:
2021-02-18T01:28:45.705 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: 2021
2021-02-18T01:28:45.705 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: -
2021-02-18T01:28:45.706 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: 02
2021-02-18T01:28:45.706 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: -
2021-02-18T01:28:45.706 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: 18
2021-02-18T01:28:45.706 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: T
2021-02-18T01:28:45.707 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: 01
2021-02-18T01:28:45.707 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: :
2021-02-18T01:28:45.707 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: 28
2021-02-18T01:28:45.707 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: :
2021-02-18T01:28:45.707 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: 40
2021-02-18T01:28:45.708 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: .
2021-02-18T01:28:45.708 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: 815423
2021-02-18T01:28:45.708 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: +0000
2021-02-18T01:28:45.708 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]:
2021-02-18T01:28:45.708 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: osd.7
2021-02-18T01:28:45.709 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]:  (
2021-02-18T01:28:45.709 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: osd
2021-02-18T01:28:45.709 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: .
2021-02-18T01:28:45.709 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: 7
2021-02-18T01:28:45.709 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: ) 26 : cluster [DBG] 1.0 scrub ok
2021-02-18T01:28:45.710 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: audit 2021-02-18T01:28:40.860457+0000 mon.a
2021-02-18T01:28:45.710 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]:  (mon.0) 702 : audit [INF] from='client.? 172.21.15.81:0/1384921965' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished
2021-02-18T01:28:45.710 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: cluster 2021-02-18
2021-02-18T01:28:45.710 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: T01:28:40.860715+0000 mon.a (mon.
2021-02-18T01:28:45.711 INFO:[email protected]:Feb 18 01:28:41 smithi081 conmon[30241]: 0) 703 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in
2021-02-18T01:28:45.711 INFO:[email protected]:Feb 18 01:28:42 smithi081 conmon[30241]: cluster 2021-02-18T01:28:41.862847+0000 mon.a (mon.0) 704 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in
2021-02-18T01:28:45.711 INFO:[email protected]:Feb 18 01:28:42 smithi081 conmon[30241]: cluster 2021-02-18T01:28:42.790258+0000 mgr.y (mgr.14146) 290 : cluster [DBG] pgmap v261: 148 pgs: 4 creating+peering, 8 unknown, 136 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s
2021-02-18T01:28:45.711 INFO:[email protected]:Feb 18 01:28:42 smithi081 conmon[30241]: cluster 2021-02-18T01:28:42.864142+0000 mon.a (mon.0) 705 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in
2021-02-18T01:28:45.711 INFO:[email protected]:Feb 18 01:28:43 smithi081 conmon[30241]: audit 2021-02-18T01:28:43.231828+0000 mon.c (mon.1) 23
2021-02-18T01:28:45.712 INFO:[email protected]:Feb 18 01:28:43 smithi081 conmon[30241]:  : audit [INF] from='client.? 172.21.15.81:0/3879181000' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch
2021-02-18T01:28:45.712 INFO:[email protected]:Feb 18 01:28:43 smithi081 conmon[30241]: audit 2021-02-18T01:28:43.232119+0000 mon.a (mon.0) 706 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch
2021-02-18T01:28:45.712 INFO:[email protected]:Feb 18 01:28:43 smithi081 conmon[30241]: audit 2021-02-18T01:28:43.448295+0000 mon.a (mon.0) 707 : audit [INF] from='client.? 172.21.15.81:0/1074727989' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]: dispatch
2021-02-18T01:28:45.712 INFO:[email protected]:Feb 18 01:28:44 smithi081 conmon[30241]: cluster 2021-02-18T01:28:43.837145+0000 osd.2 (osd.2) 21 : cluster [DBG] 2.e scrub ok
2021-02-18T01:28:45.712 INFO:[email protected]:Feb 18 01:28:44 smithi081 conmon[30241]: cluster
2021-02-18T01:28:45.713 INFO:[email protected]:Feb 18 01:28:44 smithi081 conmon[30241]:  2021-02-18T01:28:43.871381+0000 osd.5 (osd.5) 24 : cluster [DBG] purged_snaps scrub starts
2021-02-18T01:28:45.713 INFO:[email protected]:Feb 18 01:28:44 smithi081 conmon[30241]: cluster 2021-02-18T01:28:43.871500+0000 osd.5 (osd.5) 25 : cluster [DBG] purged_snaps scrub ok
2021-02-18T01:28:45.713 INFO:[email protected]:Feb 18 01:28:44 smithi081 conmon[30241]: cluster 2021-02-18T01:28:43.882919+0000 mon.a (mon.0) 708 :
2021-02-18T01:28:45.713 INFO:[email protected]:Feb 18 01:28:44 smithi081 conmon[30241]: cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS)
2021-02-18T01:28:45.713 INFO:[email protected]:Feb 18 01:28:44 smithi081 conmon[30241]: audit 2021-02-18T01:28
2021-02-18T01:28:45.714 INFO:[email protected]:Feb 18 01:28:44 smithi081 conmon[30241]: :43.891641+0000 mon.a (
2021-02-18T01:28:45.714 INFO:[email protected]:Feb 18 01:28:44 smithi081 conmon[30241]: mon.0) 709 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished
2021-02-18T01:28:45.714 INFO:[email protected]:Feb 18 01:28:44 smithi081 conmon[30241]: audit 2021-
2021-02-18T01:28:45.714 INFO:[email protected]:Feb 18 01:28:44 smithi081 conmon[30241]: 02-18T01:28:43.891744+0000 mon.a (mon
2021-02-18T01:28:45.715 INFO:[email protected]:Feb 18 01:28:44 smithi081 conmon[30241]: .0) 710 : audit [INF] from='client.? 172.21.15.81:0/1074727989' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]': finished
2021-02-18T01:28:45.715 INFO:[email protected]:Feb 18 01:28:44 smithi081 conmon[30241]: cluster 2021-
2021-02-18T01:28:45.715 INFO:[email protected]:Feb 18 01:28:44 smithi081 conmon[30241]: 02-18T01:28:43.891802+0000
2021-02-18T01:28:45.715 INFO:[email protected]:Feb 18 01:28:44 smithi081 conmon[30241]: mon.a (mon.0) 711 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in
2021-02-18T01:28:45.715 INFO:[email protected]:Feb 18 01:28:44 smithi081 conmon[30241]:
2021-02-18T01:28:45.716 INFO:[email protected]:Feb 18 01:28:44 smithi081 conmon[30241]: cluster 2021-02-18T
2021-02-18T01:28:45.716 INFO:[email protected]:Feb 18 01:28:44 smithi081 conmon[30241]: 01:28:44.790700+0000 mgr.y (mgr
2021-02-18T01:28:45.716 INFO:[email protected]:Feb 18 01:28:44 smithi081 conmon[30241]: .14146) 291 : cluster [DBG] pgmap v264: 148 pgs: 4 creating+peering, 1 unknown, 143 active+clean; 25 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s
2021-02-18T01:28:45.717 INFO:teuthology.orchestra.run.smithi081.stderr:set osd.2 primary-affinity to 0 (802)
20

@sebastian-philipp
Copy link
Contributor Author

with this PR, the output looks like so:

2021-02-24T12:11:42.496 INFO:[email protected]:-- Logs begin at Wed 2021-02-24 12:05:36 UTC. --
2021-02-24T12:13:07.200 INFO:[email protected]:Feb 24 12:13:07 smithi097 systemd[1]: Starting Ceph mon.a for 7435360a-7699-11eb-9036-001a4aab830c...
2021-02-24T12:13:07.699 INFO:[email protected]:Feb 24 12:13:07 smithi097 podman[28467]: 2021-02-24 12:13:07.296574903 +0000 UTC m=+0.136321470 container create a8ffa589d8a960971ec412d79a32e8545ed25e49d4f31c1077f75d8289864d7a (image=quay.ceph.io/ceph-ci/ceph:ff3f3310f941f746953c15ed490a43e60592fdfc, name=ce
2021-02-24T12:13:07.700 INFO:[email protected]:Feb 24 12:13:07 smithi097 podman[28467]: 2021-02-24 12:13:07.46329627 +0000 UTC m=+0.303042835 container init a8ffa589d8a960971ec412d79a32e8545ed25e49d4f31c1077f75d8289864d7a (image=quay.ceph.io/ceph-ci/ceph:ff3f3310f941f746953c15ed490a43e60592fdfc, name=ceph-
2021-02-24T12:13:07.700 INFO:[email protected]:Feb 24 12:13:07 smithi097 podman[28467]: 2021-02-24 12:13:07.49647931 +0000 UTC m=+0.336225878 container start a8ffa589d8a960971ec412d79a32e8545ed25e49d4f31c1077f75d8289864d7a (image=quay.ceph.io/ceph-ci/ceph:ff3f3310f941f746953c15ed490a43e60592fdfc, name=ceph
2021-02-24T12:13:07.700 INFO:[email protected]:Feb 24 12:13:07 smithi097 bash[28393]: a8ffa589d8a960971ec412d79a32e8545ed25e49d4f31c1077f75d8289864d7a
2021-02-24T12:13:07.701 INFO:[email protected]:Feb 24 12:13:07 smithi097 systemd[1]: Started Ceph mon.a for 7435360a-7699-11eb-9036-001a4aab830c.
2021-02-24T12:13:11.299 INFO:[email protected]:Feb 24 12:13:11 smithi097 systemd[1]: Stopping Ceph mon.a for 7435360a-7699-11eb-9036-001a4aab830c...
2021-02-24T12:13:11.699 INFO:[email protected]:Feb 24 12:13:11 smithi097 podman[29001]: 2021-02-24 12:13:11.297846288 +0000 UTC m=+0.245713927 container stop a8ffa589d8a960971ec412d79a32e8545ed25e49d4f31c1077f75d8289864d7a (image=quay.ceph.io/ceph-ci/ceph:ff3f3310f941f746953c15ed490a43e60592fdfc, name=ceph
2021-02-24T12:13:11.700 INFO:[email protected]:Feb 24 12:13:11 smithi097 podman[29001]: 2021-02-24 12:13:11.356187712 +0000 UTC m=+0.304055340 container died a8ffa589d8a960971ec412d79a32e8545ed25e49d4f31c1077f75d8289864d7a (image=quay.ceph.io/ceph-ci/ceph:ff3f3310f941f746953c15ed490a43e60592fdfc, name=ceph
2021-02-24T12:13:11.700 INFO:[email protected]:Feb 24 12:13:11 smithi097 podman[29001]: a8ffa589d8a960971ec412d79a32e8545ed25e49d4f31c1077f75d8289864d7a
2021-02-24T12:13:12.131 INFO:[email protected]:Feb 24 12:13:11 smithi097 podman[29048]: 2021-02-24 12:13:11.862616158 +0000 UTC m=+0.559399522 container remove a8ffa589d8a960971ec412d79a32e8545ed25e49d4f31c1077f75d8289864d7a (image=quay.ceph.io/ceph-ci/ceph:ff3f3310f941f746953c15ed490a43e60592fdfc, name=ce
2021-02-24T12:13:12.131 INFO:[email protected]:Feb 24 12:13:11 smithi097 systemd[1]: Stopped Ceph mon.a for 7435360a-7699-11eb-9036-001a4aab830c.
2021-02-24T12:13:12.131 INFO:[email protected]:Feb 24 12:13:11 smithi097 systemd[1]: Starting Ceph mon.a for 7435360a-7699-11eb-9036-001a4aab830c...
2021-02-24T12:13:12.132 INFO:[email protected]:Feb 24 12:13:12 smithi097 podman[29183]: 2021-02-24 12:13:12.129170926 +0000 UTC m=+0.141957213 container create 6f76e2866585e6a95760ad3936ff0a517f91bf1823213f8fa7f6cd3bb3547b94 (image=quay.ceph.io/ceph-ci/ceph:ff3f3310f941f746953c15ed490a43e60592fdfc, name=ce
2021-02-24T12:13:12.449 INFO:[email protected]:Feb 24 12:13:12 smithi097 podman[29183]: 2021-02-24 12:13:12.270882596 +0000 UTC m=+0.283668872 container init 6f76e2866585e6a95760ad3936ff0a517f91bf1823213f8fa7f6cd3bb3547b94 (image=quay.ceph.io/ceph-ci/ceph:ff3f3310f941f746953c15ed490a43e60592fdfc, name=ceph
2021-02-24T12:13:12.450 INFO:[email protected]:Feb 24 12:13:12 smithi097 podman[29183]: 2021-02-24 12:13:12.304149269 +0000 UTC m=+0.316935553 container start 6f76e2866585e6a95760ad3936ff0a517f91bf1823213f8fa7f6cd3bb3547b94 (image=quay.ceph.io/ceph-ci/ceph:ff3f3310f941f746953c15ed490a43e60592fdfc, name=cep
2021-02-24T12:13:12.450 INFO:[email protected]:Feb 24 12:13:12 smithi097 bash[29109]: 6f76e2866585e6a95760ad3936ff0a517f91bf1823213f8fa7f6cd3bb3547b94
2021-02-24T12:13:12.450 INFO:[email protected]:Feb 24 12:13:12 smithi097 systemd[1]: Started Ceph mon.a for 7435360a-7699-11eb-9036-001a4aab830c.
2021-02-24T12:20:23.065 INFO:[email protected]:Feb 24 12:20:22 smithi097 systemd[1]: Stopping Ceph mon.a for 7435360a-7699-11eb-9036-001a4aab830c...
2021-02-24T12:20:23.448 INFO:[email protected]:Feb 24 12:20:23 smithi097 podman[3865]: 2021-02-24 12:20:23.097074407 +0000 UTC m=+0.148753699 container stop 6f76e2866585e6a95760ad3936ff0a517f91bf1823213f8fa7f6cd3bb3547b94 (image=quay.ceph.io/ceph-ci/ceph:ff3f3310f941f746953c15ed490a43e60592fdfc, name=ceph-
2021-02-24T12:20:23.449 INFO:[email protected]:Feb 24 12:20:23 smithi097 podman[3865]: 2021-02-24 12:20:23.15718332 +0000 UTC m=+0.208862685 container died 6f76e2866585e6a95760ad3936ff0a517f91bf1823213f8fa7f6cd3bb3547b94 (image=quay.ceph.io/ceph-ci/ceph:ff3f3310f941f746953c15ed490a43e60592fdfc, name=ceph-7
2021-02-24T12:20:23.449 INFO:[email protected]:Feb 24 12:20:23 smithi097 podman[3865]: 6f76e2866585e6a95760ad3936ff0a517f91bf1823213f8fa7f6cd3bb3547b94
~

and that's all we get from journald. Thus the messages that are broken in the logs are completely gone otherwise!:

Without this PR, the output looks like so:

2021-02-24T11:29:47.993 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: debug
2021-02-24T11:29:47.994 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: 2021-02-24T11:29:47.748+0000 7f7c005b2700 20 expand_channel_meta expand map: {default=info}
2021-02-24T11:29:47.994 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: debug 2021-02-24T11:29:47.748+0000 7f7c005b2700 20 expand_channel_meta from 'info' to 'info'
2021-02-24T11:29:47.994 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: debug 2021-02-24T11:29:47.748+0000 7f7c005b2700 20 expand_channel_meta expanded map: {default=info}
2021-02-24T11:29:47.994 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: debug 2021-02-24T11:29:47.748+0000 7f7c005b2700 20 expand_channel_meta expand map: {default=daemon}
2021-02-24T11:29:47.994 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: debug 2021-02-24T11:29:47.748+0000 7f7c005b2700 20 expand_channel_meta from 'daemon' to 'daemon'
2021-02-24T11:29:47.995 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: debug 2021-02-24T11:29:47.748+0000 7f7c005b2700 20 expand_channel_meta expanded map: {default=daemon}
2021-02-24T11:29:47.995 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: debug 2021-02-24T11:29:47.748+0000 7f7c005b2700 20 expand_channel_meta expand map: {default=debug}
2021-02-24T11:29:47.995 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: debug 2021-02-24T11:29:47.748+0000 7f7c005b2700 20 expand_channel_meta from 'debug' to 'debug'
2021-02-24T11:29:47.995 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: debug 2021-02-24T11:29:47.748+0000 7f7c005b2700 20 expand_channel_meta expanded map: {default=debug}
2021-02-24T11:29:47.995 INFO:[email protected]:Feb 24 11:29:47 smithi036 systemd[1]: Started Ceph mon.a for 76535e7c-7693-11eb-9035-001a4aab830c.
2021-02-24T11:29:47.996 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: cluster 2021-02-24T11:29:47.752629+0000
2021-02-24T11:29:47.996 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0)
2021-02-24T11:29:47.996 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: cluster 2021-02-24T11:29:47.
2021-02-24T11:29:47.996 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: 752703+0000 mon.a (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.36:3300/0,v1:172.21.15.36:6789/0]}
2021-02-24T11:29:47.996 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: cluster 2021-
2021-02-24T11:29:47.997 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: 02-24T11:29:47
2021-02-24T11:29:47.997 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: .752748+0000 mon.a (mon.0) 3 : cluster
2021-02-24T11:29:47.997 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: [DBG] fsmap
2021-02-24T11:29:47.997 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: cluster 2021-02
2021-02-24T11:29:47.997 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: -24T11:29:47
2021-02-24T11:29:47.998 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: .752787+0000 mon.a (mon.0) 4
2021-02-24T11:29:47.998 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]:  : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in
2021-02-24T11:29:47.998 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: cluster 2021-
2021-02-24T11:29:47.998 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: 02-24T11:29:47.753240
2021-02-24T11:29:47.999 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: +0000 mon.a (mon.0) 5 : cluster
2021-02-24T11:29:47.999 INFO:[email protected]:Feb 24 11:29:47 smithi036 conmon[31056]: [DBG] mgrmap e1: no daemons active
2021-02-24T11:29:49.787 INFO:[email protected]:Feb 24 11:29:49 smithi036 conmon[31056]: audit 2021-02-24T11:29:48.480398+0000 mon.a (mon.0) 6 : audit [INF] from='client.? 172.21.15.36:0/2864638363' entity='client.admin'
2021-02-24T11:29:54.029 INFO:[email protected]:Feb 24 11:29:50 smithi036 conmon[31056]: audit 2021-02-24T11:29:50.650956+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.36:0/1987400015' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch
2021-02-24T11:29:54.287 INFO:[email protected]:Feb 24 11:29:53 smithi036 conmon[31056]: audit 2021-02-24T11:29:53.913430+0000 mon.a (mon.0) 8 : audit [DBG] from='client.? 172.21.15.36:0/3603124846' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch
2021-02-24T11:29:56.037 INFO:[email protected]:Feb 24 11:29:55 smithi036 conmon[31056]: cluster 2021-02-24T11:29:55.712209+0000 mon.a (mon.
2021-02-24T11:29:56.038 INFO:[email protected]:Feb 24 11:29:55 smithi036 conmon[31056]: 0) 9 : cluster [INF] Activating manager daemon a
2021-02-24T11:29:56.038 INFO:[email protected]:Feb 24 11:29:55 smithi036 conmon[31056]: cluster 2021-02-24T11:29:55.714770+0000 mon.a

@huww98
Copy link
Contributor

huww98 commented Feb 25, 2021

I see the same behavior too. This may because ceph daemon issues multiple write system calls for a single line of log. And conmon does not buffer until a newline.

Can we enable ceph daemons to talk to journald directly? We can take advantage of journald structured log. e.g.:

  • Translate ceph log level to journald priority
  • Omit time in message as journalctl will show the time
  • More structured output for easier analysis.

I would like to contribute if this idea looks good.

@huww98 huww98 mentioned this pull request Feb 27, 2021
3 tasks
@sebastian-philipp
Copy link
Contributor Author

containers/conmon#242

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants