Description
When using the devmapper snapshotter, any disk space used by a container during it's execution will remain allocated in the underlying device after the container exits. In development environments where the devmapper snapshotter docs suggest loopback devices, users will see what appears to be disk space leak in their root filesystem when space is not freed after exiting a container.
As an example, I ran a test where I launched a container, wrote 500MB of random data, and saw that the 500MB remained allocated after the container exited (details in steps to reproduce)
# Before the container is run, the host has 5 GB of space available
$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/nvme4n1p1 16G 9.9G 5.0G 67% /
# After the container exits, the host has 4.6 GB
$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/nvme4n1p1 16G 11G 4.6G 70% /
The underlying space does get reused by the thin pool - if I repeatedly launch a container, write 500MB, then exit the container, no additional space is allocated in the underlying device.
The space can be returned to the root filesystem by completely removing the thin-pool and loopback devices, but that means deleting all containerd data as well.
It looks like docker had this same issue in moby/moby#3182 which was tracked down to be that dmsetup does not issue discards to the underlying device when a thin device is removed (https://bugzilla.redhat.com/show_bug.cgi?id=1043527). The docker workaround was to issue BLKDISCARD ioctls to thin devices on removal. Would a similar approach be appropriate here?
Steps to reproduce the issue:
- Setup a new thin-pool using the loopback device script provided by the devmapper snapshotter readme
- Start containerd with the following config
oom_score = -999
[debug]
level = "debug"
[metrics]
address = "127.0.0.1:1338"
[plugins.linux]
runtime = "runc"
shim_debug = true
[plugins.devmapper]
pool_name = "devpool"
root_path = "/var/lib/containerd/devmapper"
base_image_size = "10GB"
async_remove = false
- Pull an image
$ sudo ctr image pull --snapshotter devmapper docker.io/library/busybox:latest
- Check available host disk space
- run a container
sudo ctr run --snapshotter devmapper --rm --tty --net-host docker.io/library/busybox:latest busybox-test
- inside the container, write 500MB of random data to a file
# head -c 500000000 </dev/urandom > random.txt
- outside the container verify that 500 MB of disk space have been utilized
- exit the container
- verify that the 500 MB of used disk space was not released
Describe the results you received:
The devmapper snapshotter did not release disk space when the container exited.
Describe the results you expected:
I expected the devmapper snapshotter to release disk space when the container exited.
What version of containerd are you using:
$ containerd --version
containerd containerd.io 1.4.6 d71fcd7d8303cbf684402823e425e9dd2e99285d
Any other relevant information (runC version, CRI configuration, OS/Kernel version, etc.):
runc --version
$ runc --version
runc version 1.0.0-rc95
commit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
spec: 1.0.2-dev
go: go1.13.15
libseccomp: 2.3.3
uname -a
$ uname -a
Linux cloudinstance 4.19.0-17-cloud-amd64 #1 SMP Debian 4.19.194-1 (2021-06-10) x86_64 GNU/Linux
Description
When using the devmapper snapshotter, any disk space used by a container during it's execution will remain allocated in the underlying device after the container exits. In development environments where the devmapper snapshotter docs suggest loopback devices, users will see what appears to be disk space leak in their root filesystem when space is not freed after exiting a container.
As an example, I ran a test where I launched a container, wrote 500MB of random data, and saw that the 500MB remained allocated after the container exited (details in steps to reproduce)
The underlying space does get reused by the thin pool - if I repeatedly launch a container, write 500MB, then exit the container, no additional space is allocated in the underlying device.
The space can be returned to the root filesystem by completely removing the thin-pool and loopback devices, but that means deleting all containerd data as well.
It looks like docker had this same issue in moby/moby#3182 which was tracked down to be that dmsetup does not issue discards to the underlying device when a thin device is removed (https://bugzilla.redhat.com/show_bug.cgi?id=1043527). The docker workaround was to issue BLKDISCARD ioctls to thin devices on removal. Would a similar approach be appropriate here?
Steps to reproduce the issue:
Describe the results you received:
The devmapper snapshotter did not release disk space when the container exited.
Describe the results you expected:
I expected the devmapper snapshotter to release disk space when the container exited.
What version of containerd are you using:
Any other relevant information (runC version, CRI configuration, OS/Kernel version, etc.):
runc --versionuname -a