|
if ctr.RemovalInProgress || ctr.Dead { |
Because the "Dead" set to "true" under container lock (between container.Lock() and container.Unlock()), there is a race in case If the 'update' method will be called concurrently with container removing. Not sure for the "RemovalInProgress" flag, but very like there is the same issue with him.
Possible solution: make these conditions checking after following ctrl.Lock().
|
ctr.CheckpointTo(daemon.containersReplica) |
Write container state raised without checking for container state. If the container is in 'Dead == true && RemovalInProgress == true' it will cause unexpected files on filesystem after finished container remove.
Script to reproduce failing conditions:
#! /bin/bash
cont=test_1
trap 'kill $(jobs -p)' SIGINT
trap 'kill $(jobs -p)' EXIT
(
while true; do
#echo "update..."
for c in $(docker ps --filter name=$cont -q); do
docker update --cpus 3 $c
done
for c in $(docker ps --filter name=$cont -q); do
docker update --cpus 2 $c
done
sleep 0.1
done
) &
while true; do
echo "wait..."
while true; do
test -z "$(docker ps --filter name=$cont -q)" && break
done
echo "run..."
docker run --rm --cpus 4 --sig-proxy=false --name $cont --workdir $(pwd) -v $(pwd):$(pwd) -v $HOME/.ssh:$HOME/.ssh ubuntu /bin/bash -c "echo 'started'; ls -la; sleep 2; echo 'stopped'"
err=$?
if [ $err -ne 0 ]; then
echo "FATAL ERROR: $err"
break
fi
done
In couple of minutes of script work we have:
ERRO[0005] Error waiting for container: unable to remove filesystem for 91bced6a61bed607b3858e61fcd583a72ca83009cb17e15e540aa7ce8e3cb272: unlinkat /spool/docker/containers/91bced6a61bed607b3858e61fcd583a72ca83009cb17e15e540aa7ce8e3cb272: directory not empty
docker version:
Client: Docker Engine - Community
Version: 20.10.2
API version: 1.41
Go version: go1.13.15
Git commit: 2291f61
Built: Mon Dec 28 16:17:43 2020
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.2
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 8891c58
Built: Mon Dec 28 16:15:19 2020
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: 1.4.3
GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info:
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 917
Server Version: 20.10.2
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.4.0-65-generic
Operating System: Ubuntu 20.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.698GiB
Name: dmytro-Aspire-S7-393
ID: GQS7:MFEA:EHZE:NLX5:OHVX:OULM:FBQM:QW3D:LRF2:5CSZ:2QOD:T367
Docker Root Dir: /var/lib/docker
Debug Mode: false
Username: zodiacbuild
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
WARNING: No blkio weight support
WARNING: No blkio weight_device support
moby/daemon/update.go
Line 54 in 0af8ed4
Because the "Dead" set to "true" under container lock (between container.Lock() and container.Unlock()), there is a race in case If the 'update' method will be called concurrently with container removing. Not sure for the "RemovalInProgress" flag, but very like there is the same issue with him.
Possible solution: make these conditions checking after following ctrl.Lock().
moby/daemon/update.go
Line 49 in 0af8ed4
Write container state raised without checking for container state. If the container is in 'Dead == true && RemovalInProgress == true' it will cause unexpected files on filesystem after finished container remove.
Script to reproduce failing conditions:
In couple of minutes of script work we have:
ERRO[0005] Error waiting for container: unable to remove filesystem for 91bced6a61bed607b3858e61fcd583a72ca83009cb17e15e540aa7ce8e3cb272: unlinkat /spool/docker/containers/91bced6a61bed607b3858e61fcd583a72ca83009cb17e15e540aa7ce8e3cb272: directory not empty
docker version:
docker info: