Skip to content

Race condition building with new/unstarted docker-container driver #49236

@milas

Description

@milas

Description

I build in GitLab CI using an ephemeral docker-container driver builder. This builder is itself running on an ephemeral/fresh DinD Moby engine.

There appears to be a rare race condition such that it's possible for the build to fail because despite having pulled/created the container, it's not yet visible.

Reproduce

  1. docker buildx create --use --driver=docker-container
  2. docker buildx build

NOTE: This is not hit every time!

Expected behavior

No response

docker version

dind // 27.4.1:

docker.io/library/docker:dind@sha256:6864c0479398023a5180cdf87e8e67ec9dcb706a21def1684f47b4ad66624d59

docker info

Client:
 Version:    27.4.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.3
    Path:     /usr/local/libexec/docker/cli-plugins/docker-buildx
Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 27.4.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 88bf19b2105c8b17560993bee28a01ddc2f97182
 runc version: v1.2.3-0-g0d37cfd
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.75+
 Operating System: Alpine Linux v3.21 (containerized)
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 31.3GiB
 Name: d18e0d94fd78
 ID: 6baace31-03b6-4426-9e73-cd20cc1a8567
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Additional Info

My builder:

$ docker buildx inspect
Name:          gitlab-ci-0
Driver:        docker-container
Last Activity: 2025-01-08 16:12:11 +0000 UTC
Nodes:
Name:                  gitlab-ci-00
Endpoint:              tcp://docker:2375
Status:                inactive
BuildKit daemon flags: --allow-insecure-entitlement=network.host

Buildx output:

#0 building with "gitlab-ci-0" instance using docker-container driver
#1 [internal] booting buildkit
#1 pulling image moby/buildkit:buildx-stable-1
#1 pulling image moby/buildkit:buildx-stable-1 2.0s done
#1 creating container buildx_buildkit_gitlab-ci-00 done
#1 ERROR: Error response from daemon: No such image: moby/buildkit:buildx-stable-1
------
 > [internal] booting buildkit:
------
ERROR: Error response from daemon: No such image: moby/buildkit:buildx-stable-1

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.status/0-triage

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions