Skip to content

Error (Unable to complete atomic operation, key modified) deleting object [endpoint_count ], retrying #51054

@rkready

Description

@rkready

Description

Restarted the docker daemon, seeing the following warning message in the dockerd log:

Sep 26 09:43:52 redacted.redacted dockerd[39178]: time="2025-09-26T09:43:52.446171446-07:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint_count 009a8baba4bdd54c8cedd64e7b2a61ea85572671cb8d8a21e675f389ae64c70e], retrying...."

This message is confusing to our users, as it appears the docker daemon failed, although that is not the case, as docker commands continue to operate as expected.

Reproduce

systemctl start docker
docker network ls (note the ID of the bridge network)
systemctl restart docker
journalctl -b -u docker|tail -n20 (note that the Error deleting object message contains the bridge network ID saved above.)

Expected behavior

No warning message about error deleting object for the default bridge network,

docker version

Client: Docker Engine - Community
 Version:           28.3.3
 API version:       1.51
 Go version:        go1.24.5
 Git commit:        980b856
 Built:             Fri Jul 25 11:36:28 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.3.3
  API version:      1.51 (minimum version 1.24)
  Go version:       go1.24.5
  Git commit:       bea959c
  Built:            Fri Jul 25 11:33:28 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    28.3.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.26.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 28.3.3
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.14.0-570.33.2.el9_6.x86_64
 Operating System: Red Hat Enterprise Linux 9.6 (Plow)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 1.73GiB
 Name: dockerzcxtest1.fyre.ibm.com
 ID: e835aa2c-5d24-4b67-bf16-c778e7360fc5
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

[root@redacted ~]# systemctl start docker

[root@redacted ~]# docker network ls
NETWORK ID     NAME      DRIVER    SCOPE
009a8baba4bd   bridge    bridge    local
e091ecf333fa   host      host      local
5386d4496101   none      null      local

[root@redacted ~]# systemctl restart docker

[root@redacted ~]# journalctl -b -u docker|tail -n20
Sep 26 09:43:51 redacted.redacted dockerd[38860]: time="2025-09-26T09:43:51.654791311-07:00" level=info msg="Daemon shutdown complete"
Sep 26 09:43:51 redacted.redacted systemd[1]: docker.service: Deactivated successfully.
Sep 26 09:43:51 redacted.redacted systemd[1]: Stopped Docker Application Container Engine.
Sep 26 09:43:51 redacted.redacted systemd[1]: docker.service: Consumed 1.022s CPU time.
Sep 26 09:43:51 redacted.redacted systemd[1]: Starting Docker Application Container Engine...
Sep 26 09:43:51 redacted.redacted dockerd[39178]: time="2025-09-26T09:43:51.754548038-07:00" level=info msg="Starting up"
Sep 26 09:43:51 redacted.redacted dockerd[39178]: time="2025-09-26T09:43:51.758095984-07:00" level=info msg="OTEL tracing is not configured, using no-op tracer provider"
Sep 26 09:43:51 redacted.redacted dockerd[39178]: time="2025-09-26T09:43:51.758520826-07:00" level=info msg="CDI directory does not exist, skipping: failed to monitor for changes: no such file or directory" dir=/var/run/cdi
Sep 26 09:43:51 redacted.redacted dockerd[39178]: time="2025-09-26T09:43:51.758736584-07:00" level=info msg="CDI directory does not exist, skipping: failed to monitor for changes: no such file or directory" dir=/etc/cdi
Sep 26 09:43:51 redacted.redacted dockerd[39178]: time="2025-09-26T09:43:51.781841933-07:00" level=info msg="Creating a containerd client" address=/run/containerd/containerd.sock timeout=1m0s
Sep 26 09:43:51 redacted.redacted dockerd[39178]: time="2025-09-26T09:43:51.811600967-07:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Sep 26 09:43:51 redacted.redacted dockerd[39178]: time="2025-09-26T09:43:51.811952746-07:00" level=info msg="Loading containers: start."
Sep 26 09:43:52 redacted.redacted dockerd[39178]: time="2025-09-26T09:43:52.446171446-07:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint_count 009a8baba4bdd54c8cedd64e7b2a61ea85572671cb8d8a21e675f389ae64c70e], retrying...."
Sep 26 09:43:52 redacted.redacted dockerd[39178]: time="2025-09-26T09:43:52.526740529-07:00" level=info msg="Loading containers: done."
Sep 26 09:43:52 redacted.redacted dockerd[39178]: time="2025-09-26T09:43:52.545601386-07:00" level=info msg="Docker daemon" commit=bea959c containerd-snapshotter=false storage-driver=overlay2 version=28.3.3
Sep 26 09:43:52 redacted.redacted dockerd[39178]: time="2025-09-26T09:43:52.545675174-07:00" level=info msg="Initializing buildkit"
Sep 26 09:43:52 redacted.redacted dockerd[39178]: time="2025-09-26T09:43:52.581756681-07:00" level=info msg="Completed buildkit initialization"
Sep 26 09:43:52 redacted.redacted dockerd[39178]: time="2025-09-26T09:43:52.591647030-07:00" level=info msg="Daemon has completed initialization"
Sep 26 09:43:52 redacted.redacted systemd[1]: Started Docker Application Container Engine.
Sep 26 09:43:52 redacted.redacted dockerd[39178]: time="2025-09-26T09:43:52.592963088-07:00" level=info msg="API listen on /run/docker.sock"

Metadata

Metadata

Assignees

Labels

area/networkingNetworkingkind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.version/28.3

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions