Repository navigation
panic: runtime error: invalid memory address or nil pointer dereference #5767
Description
Contributing guidelines and issue reporting guide
- I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.
Well-formed report checklist
- I have found a bug that the documentation does not mention anything about my problem
- I have found a bug that there are no open or closed issues that are related to my problem
- I have provided version/information about my environment and done my best to provide a reproducer
Description of bug
Bug description
today a very strange error appeared, buildx runs in gitlab runner and creates a build, but today suddenly a problem appeared
I tried different versions, all without success
docker info
Client:
Version: 25.0.5
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.13.1
Path: /usr/local/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.25.0
Path: /usr/local/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 28.0.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
runc version: v1.2.5-0-g599[23](****/39292#L23)ef
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.1.0-31-amd64
Operating System: Alpine Linux v3.21 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 48
Total Memory: 1[25](****/-/jobs/39292#L25).5GiB
Name: f5dc9ade0ae0
ID: d1f2f41e-e30b-434b-a789-b3ddefada255
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
::1/128
1[27](****39292#L27).0.0.0/8
Live Restore Enabled: false
Product License: Community EngineReproduction
image:
name: docker:git
services:
- name: docker:dind
alias: thedockerhost
command: ['--tls=false']
variables:
DOCKER_HOST: tcp://thedockerhost:2375/
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ''
CONTAINER_RELEASE_IMAGE: $CI_REGISTRY_IMAGE:latest
DOCKER_BUILDKIT: 1
build-image:
stage: build
tags:
- docker-207
before_script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
script:
- docker buildx version
- docker buildx create --driver=docker-container --name=buildkit-builder --use
- docker buildx inspect buildkit-builder --bootstrapVersion information
v0.13.1
$ docker buildx version
github.com/docker/buildx v0.13.1 788433953af10f2a698f5c07611dddce2e08c7a0
$ docker buildx create --driver=docker-container --name=buildkit-builder --use
buildkit-builder
$ docker buildx inspect buildkit-builder --bootstrap
#1 [internal] booting buildkit
#1 pulling image moby/buildkit:buildx-stable-1
#1 pulling image moby/buildkit:buildx-stable-1 15.8s done
#1 creating container buildx_buildkit_buildkit-builder0
#1 31.31 panic: runtime error: invalid memory address or nil pointer dereference
�#1 creating container buildx_buildkit_buildkit-builder0 15.5s done
panic: runtime error: invalid memory address or nil pointer dereference
#1 31.31
#1 31.31
#1 31.31 github.com/fsnotify/fsnotify.(*Watcher).igithub.com/fsnotify/fsnotify.(*Watcher).isClosed(...)
#1 31.31 github.com/fsnotify/fsnotify.(*Watcher).AddWith(0x0, {0x1ce301f, github.com/fsnotify/fsnotify.(*Watcher).AddWith(0x0, {0x1ce301f, 0xc}, {0x0, 0x0, 0xc0005129e8?})
#1 31.31
Activity
crazy-max commentedon Feb 21, 2025
Seems you're using an old client with latest Docker 28 but don't think that's the issue.
This error seems returned by the Docker engine and not BuildKit. Can you run this command with debug enabled?
And this one with
docker runfor creating the container builder without Buildx?:Also cc @thaJeztah @vvoland if this is smth you might be aware of.
entense commentedon Feb 21, 2025
@crazy-max
docker --debug buildx create --driver=docker-container --name=buildkit-builder --use --bootstrapfull output
docker --debug run -d --privileged --name testbk moby/buildkit:latest --debugthaJeztah commentedon Feb 21, 2025
@crazy-max panic seems to be in buildkitd (CDI?)
(edit: removed control chars for readability)
elezar commentedon Feb 21, 2025
@klihub do you know of any reason that
fsnotifywould panic like this?klihub commentedon Feb 21, 2025
No, I have never seen it doing that. Did you check if there are any related issues open ?
crazy-max commentedon Feb 21, 2025
@entense Thanks for the logs
In any case do you have a default buildkitd config file in your home?:
$BUILDX_CONFIG/buildkitd.default.toml$DOCKER_CONFIG/buildx/buildkitd.default.toml~/.docker/buildx/buildkitd.default.tomlI would say no as it seems to run the build in a fresh
docker:dindcontainer.crazy-max commentedon Feb 21, 2025
@entense Can you try with this command so it shows the outputs?
entense commentedon Feb 21, 2025
@crazy-max
crazy-max commentedon Feb 21, 2025
@elezar We are vendoring fsnotify 1.7.0 https://github.com/moby/buildkit/blob/v0.20.0/go.mod#L150 while in your case it's 1.5.1: https://github.com/cncf-tags/container-device-interface/blob/v0.8.0/go.mod#L6. Wonder if this is related.
crazy-max commentedon Feb 21, 2025
@entense Thanks! So not something wrong with buildx.
crazy-max commentedon Feb 21, 2025
@entense Can you try with this command and post the output please?:
and this one:
entense commentedon Feb 21, 2025
@crazy-max
docker --debug run --rm -t -e FSNOTIFY_DEBUG=1 --privileged moby/buildkit:latest --debug
docker --debug run --rm -t -e FSNOTIFY_DEBUG=1 --privileged crazymax/buildkit:5767 --debug
tonistiigi commentedon Feb 21, 2025
@entense Could you run again with
tonistiigi/buildkit:v0.20.0-fsnotify-error-debugimage and paste the output sourceAfaics the likely issue is in https://github.com/cncf-tags/container-device-interface/blob/main/pkg/cdi/cache.go#L506 where it ignores the error return from
NewWatcher. SIGSEGV happens ifwatch.watcheris nil inisClosed().5 remaining items
jrscholey commentedon Feb 22, 2025
As of Friday 21st Feb, we are also experiencing the same issue in our gitlab build pipelines using docker-dind:20.10 with our k8s gitlab runners, with no configuration change from our side.
On Friday afternoon the error appeared to be intermittent, but as of the evening (CET) until now it continues to return the error 100%. Clearing runner caches / artifacts and restarting our runner deployments has no effect.
I wll try to provide more logs when I can. But initially we are seeing the same output as OP in the build logs of gitlab.
entense commentedon Feb 22, 2025
I managed to solve this temporarily by increasing the limits on the server.
increase hard ulimit from
1048576to4096000paste
session required pam_limits.soput
DefaultLimitNOFILE=4096000in
/etc/sysctl.confconfig.tomland reboot system
klihub commentedon Feb 22, 2025
@tonistiigi @elezar Here is a minimal footprint fix cncf-tags/container-device-interface#254. Provided that's deemed good enough and gets merged, would you like us to roll a v0.8.1 patch release with only the fix cherry-picked ?
tonistiigi commentedon Feb 22, 2025
Pushed test PR #5769 with image
tonistiigi/buildkit:cdi-fsnotify-fixwith that patch.Yes please.
Anyone hitting this(in gitlab), you can try increasing the limits as suggested above or use
moby/buildkit:v0.19.0ortonistiigi/buildkit:cdi-fsnotify-fixas a workaround. If you have more details of what may cause reaching the open file limit in that Gitlab configuration you can also post it. Still not clear to me why this specifically would reach this limit.@klihub By default the CDI directories we listen on, do not even exist in the container image configuration. Would that cause file watcher on the parent? (and result in many open files). If true, why would that be as even if fsnotify needs to listen on parent it only cares about the root and not about the events from subfiles of that parent dir.
klihub commentedon Feb 22, 2025
@tonistiigi No, we don't do that. We don't walk up the directory tree and install a set of watches to detect when all missing directories are created. Instead we simply administer if we haven't managed to create a watch for a (missing) directory we need to track, and then re-try creating the watch when we do a refresh.
@entense mentioned in this comment earlier that it looks like the problem arises, whenever you have 120 docker containers running on the server. Stopping one makes the problem go away and then creating one brings it back.
Could we try to figure out if you really hit a limit which is set too low, or if some component (maybe us in CDI) leaks file descriptors somewhere, by driving the system to the 120 container state and then checking the
/proc/$pid/fdof the failing process to see what kind of descriptors we see there ? Could help a bit narrowing down where we are looking for the culprit.klihub commentedon Feb 24, 2025
@tonistiigi Tagged a v0.8.1 release.
thaJeztah commentedon Feb 24, 2025
@klihub it's only
tags.cncf.io/container-device-interface, and no updates intags.cncf.io/container-device-interface/specs-go, correct?klihub commentedon Feb 24, 2025
Yes, no other changes than the bug fix (and test cases).
thaJeztah commentedon Feb 24, 2025
Thanks!
jrscholey commentedon Feb 24, 2025
I can confirm that passing
--driver-opt image=tonistiigi/buildkit:cdi-fsnotify-fixor--driver-opt image=moby/buildkit:v0.19.0tobuildx createin our gitlab-ci.yml resolves the issue for us. Many thanks! @tonistiigientense commentedon Feb 24, 2025
It works the same for me