ateom-gvisor: runsc delete fails after most checkpoints because cleanup runs before the sentry is reaped #2180

Problem

After runsc checkpoint, cleaning up the actor's containers often fails with the error:

    runsc ... delete -force glutton
    W container.go:1001] stopping container: destroying container "glutton":
      connecting to control server at PID <pid>: connection refused
    WARN failed to terminate workload after checkpoint
      err: ... while deleting "glutton" container: while running `runsc delete`: exit status 128

#2107 stops leaking actor cgroups after checkpoint and delete. However, that change only contains the damage: the delete still fails and logs a WARN, and the race below is still there.

Cause

1. runsc checkpoint _pause makes the sentry exit and close its control socket.
2. The sentry is a child of ateom, which runs as the pod's init process. It stays a zombie until ateom's reaper (internal/childreap) collects it.
3. The reaper never reaps while a reaper.RunCommand subcommand is running, so it can't steal that subcommand's exit status.
4. terminateWorkload runs about ten runsc subcommands back to back, roughly 8 ms apart, so the reaper often gets no chance to run.
5. Each subcommand gets connection refused and then checks whether the sandbox PID exists. A zombie's PID still exists, so runsc decides the sandbox is running. delete -force <app container> then tries to destroy the container through the dead socket and exits with status 128.

When the reaper does get a turn before the delete, the log shows Sandbox isn't running anymore and every delete succeeds.

Reproduce

Suspend a few running gVisor actors, then count cleanup failures in each worker's log:

kubectl ate suspend actor <actor> -a <atespace>
kubectl -n <ns> logs <worker> --since=3m | grep -c 'failed to terminate workload after checkpoint'