Description
After upgrade from 19.03.x to 20.10.x we noticed that a raising amount of GB of disk space was occupied within folder "docker/buildkit/content/blobs/sha256". This only happens when building with "DOCKER_BUILDKIT=1". Although we are building with "--no-cache" the space gets occupied.
We tried to cleanup calling "docker system prune -af" and "docker builder prune -af" or "docker buildx prune -af". Nothing removed the data.
Calling "docker system df" or "docker buildx du" afterwards reported 0B of any occupied space which is surprising since there were more than 30 GB occupied in docker/buildkit/content/blobs/sha256.
Steps to reproduce the issue:
- Upgrade docker-ce to 20.10.x.
- Check occupied space in 'buildkit' subfolder: "sudo du -h -d 1 /var/lib/docker/buildkit"
- Build an image deriving from CUDA development image: "FROM nvidia/cuda:11.0-devel-ubuntu20.04" with "DOCKER_BUILDKIT=1 docker build --no-cache"
- Check increased occupied space in 'buildkit' subfolder: "sudo du -h -d 1 /srv/docker/buildkit"
- Clean up: "sudo docker prune -af"
- Check done cleanup: "sudo docker system df"
- Check that there is no change in occupied space in 'buildkit' subfolder: "sudo du -h -d 1 /srv/docker/buildkit"
Describe the results you received:
The occupied space in /var/lib/docker/buildkit was not freed.
Describe the results you expected:
Occupied space in /var/lib/docker/buildkit was freed.
Additional information you deem important (e.g. issue happens only occasionally):
Issue always happens.
Output of docker version:
Client: Docker Engine - Community
Version: 20.10.3
API version: 1.41
Go version: go1.13.15
Git commit: 48d30b5
Built: Fri Jan 29 14:33:21 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.3
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 46229ca
Built: Fri Jan 29 14:31:32 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.3
GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Output of docker info:
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 20.10.3
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.4.0-65-generic
Operating System: Ubuntu 20.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.941GiB
Name: friday-agent-1
ID: TL7W:VGHA:PUTC:W3TV:R5MT:I4PL:QSR5:SMO3:LMJP:TLYU:DQCI:PVSL
Docker Root Dir: /srv/docker
Debug Mode: false
Username: fraunhofermevis
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
WARNING: No blkio weight support
WARNING: No blkio weight_device support
Description
After upgrade from 19.03.x to 20.10.x we noticed that a raising amount of GB of disk space was occupied within folder "docker/buildkit/content/blobs/sha256". This only happens when building with "DOCKER_BUILDKIT=1". Although we are building with "--no-cache" the space gets occupied.
We tried to cleanup calling "docker system prune -af" and "docker builder prune -af" or "docker buildx prune -af". Nothing removed the data.
Calling "docker system df" or "docker buildx du" afterwards reported 0B of any occupied space which is surprising since there were more than 30 GB occupied in docker/buildkit/content/blobs/sha256.
Steps to reproduce the issue:
Describe the results you received:
The occupied space in /var/lib/docker/buildkit was not freed.
Describe the results you expected:
Occupied space in /var/lib/docker/buildkit was freed.
Additional information you deem important (e.g. issue happens only occasionally):
Issue always happens.
Output of
docker version:Output of
docker info: