Building large images with provenance attestation causes ResourceExhausted error when building to tar file or infinite hang when building to a folder #2208
Description
Contributing guidelines
- I've read the contributing guidelines and wholeheartedly agree
I've found a bug and checked that ...
- ... the documentation does not mention anything about my problem
- ... there are no open or closed issues that are related to my problem
Description
Building a large image with a provenance attestation and outputting to a tar file fails with an error:
ERROR: failed to solve: ResourceExhausted: failed to write file header root/.cshrc: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (7693509 vs. 4194304)
When building the same image with a provenance attestation and outputting to a local folder, the build command copies some files (a few MB) but hangs and never exits.
Smaller images work fine. Building a Docker image works fine (output=type=docker). Just these two output methods fail.
My goal is to find the parent image of the new image somehow. Adding a provenance attestation and producing a tar file or local folder is the only way I can find to find that information (because the metadata file no longer contains this info). Pushing my image to Docker Hub is not possible and docker buildx imagetools inspect will not inspect an unpushed image. If I'm missing something and there is a way to find the parent image (name, tag and digest), please please please let me know!
Expected behaviour
I expect the image to be exported as a tar file or local folder, regardless of the image size.
Actual behaviour
The command fails with the above error message or by hanging forever.
Buildx version
github.com/docker/buildx v0.12.1 30feaa1
Docker info
Client: Docker Engine - Community
Version: 25.0.0
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.12.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.24.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
scan: Docker Scan (Docker Inc.)
Version: v0.23.0
Path: /usr/libexec/docker/cli-plugins/docker-scan
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 13
Server Version: 25.0.0
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
runc version: v1.1.11-0-g4bccb38
init version: de40ad0
Security Options:
seccomp
Profile: builtin
Kernel Version: 4.18.0-513.11.1.el8_9.x86_64
Operating System: Rocky Linux 8.9 (Green Obsidian)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 15.63GiB
Name: rocky8
ID: QMWV:XVVY:NGBT:2EWN:GXSB:ZASB:SR7E:HGXS:2NIN:ZZ55:J27Y:UNA7
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Builders list
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
default * docker
default default running v0.12.4+3b6880d2a00f linux/amd64, linux/amd64/v2, linux/amd64/v3
Configuration
Dockerfile:
FROM rockylinux:9
# Install a bunch of stuff to produce a large image
RUN [ "/usr/bin/yum", "install", "-y", "gcc", "gcc-toolset-12", "gcc-toolset-13", "golang", "httpd", "clang.i686", "clang.x86_64", "gnome-shell" ]Try to build the image to a tar file:
DOCKER_BUILDKIT=1 /usr/bin/docker buildx build -t test --output=type=tar,dest=image.tar --attest type=provenance,mode=max,filename=provenance.json .
Then try to build the image to a local folder:
DOCKER_BUILDKIT=1 /usr/bin/docker buildx build -t test --output=type=local,dest=foldername --attest type=provenance,mode=max,filename=provenance.json .
Build logs
root@rocky8:~# DOCKER_BUILDKIT=1 /usr/bin/docker buildx build -t test --output=type=tar,dest=image.tar --attest type=provenance,mode=max,filename=provenance.json .
[+] Building 173.2s (6/6) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 205B 0.0s
=> [internal] load metadata for docker.io/library/rockylinux:9 0.2s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> CACHED [1/2] FROM docker.io/library/rockylinux:9@sha256:c944604c0c759f5d164ffbdf0bbab2fac582b739938937403c067ab634a0518a 0.0s
=> => resolve docker.io/library/rockylinux:9@sha256:c944604c0c759f5d164ffbdf0bbab2fac582b739938937403c067ab634a0518a 0.0s
=> [2/2] RUN [ "/usr/bin/yum", "install", "-y", "gcc", "gcc-toolset-12", "gcc-toolset-13", "golang", "httpd", "clang.i686", "clang.x86_64" 153.4s
=> ERROR exporting to client tarball 19.1s
=> => sending tarball 4.7s
------
> exporting to client tarball:
------
ERROR: failed to solve: ResourceExhausted: failed to write file header root/.cshrc: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (7693509 vs. 4194304)
This ran for 5 minutes before I killed it:
root@rocky8:~# DOCKER_BUILDKIT=1 /usr/bin/docker buildx build -t test --output=type=local,dest=output --attest type=provenance,mode=max,filename=provenance.json .
[+] Building 330.5s (5/6) docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 256B 0.0s
=> [internal] load metadata for docker.io/library/rockylinux:9 0.5s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [1/2] FROM docker.io/library/rockylinux:9@sha256:c944604c0c759f5d164ffbdf0bbab2fac582b739938937403c067ab634a0518a 1.9s
=> => resolve docker.io/library/rockylinux:9@sha256:c944604c0c759f5d164ffbdf0bbab2fac582b739938937403c067ab634a0518a 0.0s
=> => extracting sha256:489e1be6ce56f590a5a31bdf814671cac006421930c1175cb62e1763bf51a3f9 1.9s
=> [2/2] RUN [ "/usr/bin/yum", "install", "-y", "gcc", "gcc-toolset-12", "gcc-toolset-13", "golang", "httpd", "clang.i686", "clang.x86_64" 169.1s
=> exporting to client directory 330.0s
=> => copying files 7.78MB 320.7s
^Croot@rocky8:~#
root@rocky8:~# du -sch output/*
0 output/afs
0 output/bin
0 output/dev
5.6M output/etc
0 output/home
0 output/lib
0 output/lib64
0 output/lost+found
0 output/media
0 output/mnt
3.4M output/opt
0 output/proc
0 output/provenance.json
0 output/root
4.0K output/run
0 output/sbin
0 output/srv
0 output/sys
0 output/tmp
2.2M output/usr
12M total
Additional info
No response