Skip to content

Getting error with overlayfs: device or resource busy #37970

@sargun

Description

@sargun

This seems to be related to #34672.

Description
We have containers fail to start up, with the error: Error response from daemon: error creating overlay mount to /mnt/docker/10000.10000/overlay2/8cf4d4d1bcfa3c3983c260dae482513a7c10e2a38e76b5bcad35da05653abfaf/merged: device or resource busy

In the log, we get:

Oct 04 22:59:40 titusagent-mainvpc-r416xlarge004-i-06067ebd1d0792b8a dockerd[1870]: time="2018-10-04T22:59:40.200321961Z" level=debug msg="SetQuota(/mnt/docker/10000.10000/overlay2/8cf4d4d1bcfa3c3983c260dae482513a7c10e2a38e76b5bcad35da05653abfaf-init, 18122539008): projectID=1078"
Oct 04 22:59:41 titusagent-mainvpc-r416xlarge004-i-06067ebd1d0792b8a dockerd[1870]: time="2018-10-04T22:59:41.516160665Z" level=debug msg="SetQuota(/mnt/docker/10000.10000/overlay2/8cf4d4d1bcfa3c3983c260dae482513a7c10e2a38e76b5bcad35da05653abfaf, 18122539008): projectID=1079"
Oct 04 22:59:41 titusagent-mainvpc-r416xlarge004-i-06067ebd1d0792b8a dockerd[1870]: time="2018-10-04T22:59:41.540898661Z" level=debug msg="container mounted via layerStore: &{/mnt/docker/10000.10000/overlay2/8cf4d4d1bcfa3c3983c260dae482513a7c10e2a38e76b5bcad35da05653abfaf/merged 0x55da380489e0 0x55da380489e0}"
Oct 04 22:59:44 titusagent-mainvpc-r416xlarge004-i-06067ebd1d0792b8a dockerd[1870]: time="2018-10-04T22:59:44.243611858Z" level=debug msg="container mounted via layerStore: &{/mnt/docker/10000.10000/overlay2/8cf4d4d1bcfa3c3983c260dae482513a7c10e2a38e76b5bcad35da05653abfaf/merged 0x55da380489e0 0x55da380489e0}"
Oct 04 22:59:45 titusagent-mainvpc-r416xlarge004-i-06067ebd1d0792b8a dockerd[1870]: time="2018-10-04T22:59:45.810327774Z" level=debug msg="container mounted via layerStore: &{/mnt/docker/10000.10000/overlay2/8cf4d4d1bcfa3c3983c260dae482513a7c10e2a38e76b5bcad35da05653abfaf/merged 0x55da380489e0 0x55da380489e0}"
Oct 04 22:59:46 titusagent-mainvpc-r416xlarge004-i-06067ebd1d0792b8a dockerd[1870]: time="2018-10-04T22:59:46.602156580Z" level=debug msg="container mounted via layerStore: &{/mnt/docker/10000.10000/overlay2/8cf4d4d1bcfa3c3983c260dae482513a7c10e2a38e76b5bcad35da05653abfaf/merged 0x55da380489e0 0x55da380489e0}"
Oct 04 22:59:46 titusagent-mainvpc-r416xlarge004-i-06067ebd1d0792b8a dockerd[1870]: time="2018-10-04T22:59:46.865366329Z" level=error msg="error unmounting /mnt/docker/10000.10000/overlay2/8cf4d4d1bcfa3c3983c260dae482513a7c10e2a38e76b5bcad35da05653abfaf/merged: invalid argument"
Oct 04 22:59:46 titusagent-mainvpc-r416xlarge004-i-06067ebd1d0792b8a dockerd[1870]: time="2018-10-04T22:59:46.946538129Z" level=debug msg="Failed to unmount 8cf4d4d1bcfa3c3983c260dae482513a7c10e2a38e76b5bcad35da05653abfaf overlay: /mnt/docker/10000.10000/overlay2/8cf4d4d1bcfa3c3983c260dae482513a7c10e2a38e76b5bcad35da05653abfaf/merged - no such file or directory"
Oct 04 22:59:46 titusagent-mainvpc-r416xlarge004-i-06067ebd1d0792b8a dockerd[1870]: time="2018-10-04T22:59:46.948102510Z" level=debug msg="FIXME: Got an API for which error does not match any expected type!!!: error creating overlay mount to /mnt/docker/10000.10000/overlay2/8cf4d4d1bcfa3c3983c260dae482513a7c10e2a38e76b5bcad35da05653abfaf/merged: device or resource busy" error_type="*errors.errorString" module=api
Oct 04 22:59:46 titusagent-mainvpc-r416xlarge004-i-06067ebd1d0792b8a dockerd[1870]: time="2018-10-04T22:59:46.948117937Z" level=error msg="Handler for POST /v1.26/containers/cec5ef92af3ddd4e938cbbb95b11dc04bf03edc4c6ed76dcf0bba991d1c577ff/start returned error: error creating overlay mount to /mnt/docker/10000.10000/overlay2/8cf4d4d1bcfa3c3983c260dae482513a7c10e2a38e76b5bcad35da05653abfaf/merged: device or resource busy"
Oct 04 22:59:46 titusagent-mainvpc-r416xlarge004-i-06067ebd1d0792b8a dockerd[1870]: time="2018-10-04T22:59:46.948134415Z" level=debug msg="FIXME: Got an API for which error does not match any expected type!!!: error creating overlay mount to /mnt/docker/10000.10000/overlay2/8cf4d4d1bcfa3c3983c260dae482513a7c10e2a38e76b5bcad35da05653abfaf/merged: device or resource busy" error_type="*errors.errorString" module=api

Additional information you deem important (e.g. issue happens only occasionally):
We're still trying to figure out the root cause of this, as it's happening quite infrequently.

Output of docker version:

(nfsuper) ~ $ docker version
Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:17:20 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:15:30 2018
  OS/Arch:      linux/amd64
  Experimental: false

Output of docker info:

(nfsuper) ~ $ docker info
Containers: 11
 Running: 11
 Paused: 0
 Stopped: 0
Images: 14
Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: /apps/titus-executor/bin/tini-static
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: ac32c23 (expected: 949e6fa)
Security Options:
 apparmor
 seccomp
  Profile: default
 userns
Kernel Version: 4.18.10netflix-gf008452
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 64
Total Memory: 480.3GiB
Name: titusagent-mainvpc-r416xlarge004-i-06067ebd1d0792b8a
ID: 4PE2:AIBM:QNEO:CHO3:AVML:WK6P:4H6T:LESA:CTES:W7RU:Z35H:E7HH
Docker Root Dir: /mnt/docker/10000.10000
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions