Containerd snapshotter saves layers that can't be cleared easily #48909
Description
Description
I have the following enabled on my machine for docker:
"features": {
"containerd-snapshotter": true
}
If I do a pull and cancel it part way through it will save the layers.
If I never complete this pull the data will persist under /contianerd/io.containerd.snapshotter.v1.overlayfs and /containerd/io.containerd.content.v1.content
Running commands like docker system prune -a or docker image prune -a do not work for removing this data and there doesn't seem to be any way to manually trigger garbage collection of these files.
Additionally I tried to see if I could find anything related to these files through ctr images, ctr leases, and ctr snapshots with no luck.
Was told this is the best place for this bug here:
containerd/containerd#10548
docker/cli#5315
Reproduce
- Enable snapshotter for docker with code block from description.
- Note reported usage of docker images with docker system df
- docker pull <any_image>
- Cancel the pull before it completes, but after some layers have been pulled and extracted.
- Confirm with docker system df (or du) that the usage has increased.
- Confirm new data is unable to be removed through normal commands (or even through gc over time)
Expected behavior
Have some way of clearing this data without running an rm command on the directory.
Either through ctr or docker; or if garbage collection cleans it up then a way to manually force that data to be removed.
docker version
Client: Docker Engine - Community
Version: 27.1.1
API version: 1.46
Go version: go1.21.12
Git commit: 6312585
Built: Tue Jul 23 19:57:01 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 27.1.1
API version: 1.46 (minimum version 1.24)
Go version: go1.21.12
Git commit: cc13f95
Built: Tue Jul 23 19:57:01 2024
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: 1.7.19
GitCommit: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
runc:
Version: 1.7.19
GitCommit: v1.1.13-0-g58aa920
docker-init:
Version: 0.19.0
GitCommit: de40ad0docker info
Client: Docker Engine - Community
Version: 27.1.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.16.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.29.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 27.1.1
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 nvidia runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
runc version: v1.1.13-0-g58aa920
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.5.0-44-generic
Operating System: Ubuntu 22.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 20
Total Memory: 62.46GiB
Name:
ID: 797409c4-1056-4a1f-946d-93f3a7004068
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: true
Insecure Registries:
localhost:5000
Live Restore Enabled: falseAdditional Info
No response