-
Notifications
You must be signed in to change notification settings - Fork 46
Description
Description
Beginning with release 2605.9.0 Docker containers generate many signal 23 events which flood monitoring systems (example: kubernetes/kops#10388). The SIGURG signal does not kill the process but is generated by Go runtime scheduling (https://go.googlesource.com/proposal/+/master/design/24543-non-cooperative-preemption.md). Because the Go runtime does not know if the process expects external SIGURG signals, the signal is not filtered out but reported to the process (golang/go#37942). The process has to filter this signal out itself before forwarding it to, e.g,. children processes or logs.
This change was introduced with the Go 1.15 update (actually Go 1.14 but Flatcar skipped that for Stable). The Go 1.15 compiler was used for the Docker, containerd, and runc binaries. However, while containerd has some workaround in place (containerd/containerd#4532) this is not the case for every part of it.
I suggest trying to downgrade the Docker/containerd binaries to use the Go 1.13 compiler.
Impact
For example, many entries appear in the output of docker events -f container=ID
Environment and steps to reproduce
- Set-up: Run an alpine container, and in a second terminal monitor the events as written above (can omit
-f …). - Task Hit Ctrl-C a few times and Enter
- Action(s): ↑
- Error: Observe log entries, maybe also attach with
sudo strace -tt -ff -p PIDto the container process and also to thecontainerd-shimprocess to see12:27:44.156649 poll([{fd=0, events=POLLIN}], 1, -1) = ? ERESTART_RESTARTBLOCK (Interrupted by signal) 12:27:44.206100 --- SIGURG {si_signo=SIGURG, si_code=SI_USER, si_pid=0, si_uid=0} ---for the container process and[pid 2120] futex(0xa70750, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000} <unfinished ...> [pid 2189] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=2114, si_uid=0} ---for the shim process
Expected behavior
The signal 23 from the Go runtime does not appear in the log anymore
Additional information
Created an upstream issue at containerd/containerd#4935