I dont think you can SIGTERM a paused container? but someone correct me if I am wrong

Correct, it won't do anything since it's been frozen and as such can't respond to signals.
From what I understand you can SIGKILL then unfreeze and the process will be forced to exit immediately and won't have a chance to wreck any havoc.

@cpuguy83 is right, if you want SIGKILL or SIGTERM to a frozen process, you should send
SIGKILL or SIGTERM to the process and unfreeze it, the process will be forced to exit
immediately

can we have an integration test to prove that is true.

@jfrazelle Had add a integration test

@coolljt0725 GitHub says it needs a rebase.

@thaJeztah yeah, thanks

since #12400 is merged, if the design of this is ok, I'll rebase this. ping @cpuguy83 @LK4D4 @jfrazelle @calavera @thaJeztah

Hi all, if the design is ok, I'll rebase this

ping @cpuguy83 @calavera @LK4D4 PTAL

Also, @coolljt0725 does this solve #12797?

design looks ok for me

Should we try to fix this upon daemon startup instead? What if the daemon crashes w/o daemon.Shutdown() func having a chance to be called. I'm assuming we'll run into the original issue, right?

@duglin I think fix this on docker daemon shutdown is also to do some cleanup of containers to avoid orphaned virt interfaces and unmounted rootfs...

rebased , ping @LK4D4 @cpuguy83 @jfrazelle @duglin @calavera

@coolljt0725 yes but my point is that I don't think we can count on that code always being run. What if someone just pulls the plug on the machine and it dies? Don't we need to do this logic upon daemon startup otherwise things will still be in a bad state?

@duglin I understand you point , I think this is a common problem not only for docker, other software have the same problem, abnormal shutdown of software will with some information missing and hard to recover or cleanup on next startup.
I think to do this logic upon daemon startup need some journal file, with a journal file we can know if the daemon is shutdown normally and we can do some recover base on the journal file.
This PR is to make normal shutdown of daemon cleanup what has created.

@coolljt0725 I haven't had a chance to look into why its hanging, has the investigation been done? Do we know what the fix is to if this cleanup func isn't run? In other words, what can the user do to get out of this bad state?

@duglin I have investigated this. The hang will only happened when use native.cgroupdriver=cgroupfs, this happened because daemon shutdown leave the paused container cgroup still exitst with freezer.state=FROZEN, so on daemon next startup, when go to https://github.com/docker/libcontainer/blob/master/process_linux.go#L196 the process of the container will join this FROZEN cgroup and become paused, so it hangs.
for native.cgroupdriver=systemd it will return an error [8] System error: Unit docker-xxxxx.scope already exists.

@coolljt0725 is there any way to kill the frozen process? (w/o actually rebooting the machine)

@duglin Sorry for the delay. I think the way to kill the frozen process is to send a SIGTERM or SIGKILL to the process and then remove it from the frozen cgroup or thaw the frozen cgroup

@coolljt0725 is this something we can do for those processes during daemon startup to avoid a hang?

Ping @coolljt0725

@tiborvass I don't understand your comment above. Do you mean we should add a empty SignalMap in signal_windows.go?
and we get the signal by sig, ok := signal.SignalMap["TERM"] ?

ping @tiborvass

so sorry needs a rebase

@jfrazelle @tiborvass rebased

so so sorry needs another rebase

@jfrazelle @tiborvass rebased

@jfrazelle @cpuguy83 rebased

code LGTM

LGTM, tested locally.

We should probably open a separate issue to discuss handling of paused containers when someone calls stop instead of raising a conflict.