Skip to content

Signal 23 (SIGURG) Docker event flood #315

@pothos

Description

@pothos

Description
Beginning with release 2605.9.0 Docker containers generate many signal 23 events which flood monitoring systems (example: kubernetes/kops#10388). The SIGURG signal does not kill the process but is generated by Go runtime scheduling (https://go.googlesource.com/proposal/+/master/design/24543-non-cooperative-preemption.md). Because the Go runtime does not know if the process expects external SIGURG signals, the signal is not filtered out but reported to the process (golang/go#37942). The process has to filter this signal out itself before forwarding it to, e.g,. children processes or logs.
This change was introduced with the Go 1.15 update (actually Go 1.14 but Flatcar skipped that for Stable). The Go 1.15 compiler was used for the Docker, containerd, and runc binaries. However, while containerd has some workaround in place (containerd/containerd#4532) this is not the case for every part of it.
I suggest trying to downgrade the Docker/containerd binaries to use the Go 1.13 compiler.

Impact

For example, many entries appear in the output of docker events -f container=ID

Environment and steps to reproduce

  1. Set-up: Run an alpine container, and in a second terminal monitor the events as written above (can omit -f …).
  2. Task Hit Ctrl-C a few times and Enter
  3. Action(s): ↑
  4. Error: Observe log entries, maybe also attach with sudo strace -tt -ff -p PID to the container process and also to the containerd-shim process to see 12:27:44.156649 poll([{fd=0, events=POLLIN}], 1, -1) = ? ERESTART_RESTARTBLOCK (Interrupted by signal) 12:27:44.206100 --- SIGURG {si_signo=SIGURG, si_code=SI_USER, si_pid=0, si_uid=0} --- for the container process and [pid 2120] futex(0xa70750, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000} <unfinished ...> [pid 2189] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=2114, si_uid=0} --- for the shim process

Expected behavior
The signal 23 from the Go runtime does not appear in the log anymore

Additional information
Created an upstream issue at containerd/containerd#4935

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions