Skip to content

Peer operation failed:could not delete fdb entry into the sandbox and other errors #35239

Closed
Closed
Peer operation failed:could not delete fdb entry into the sandbox and other errors#35239
Labels
area/networkingNetworkingarea/swarmversion/17.06
@sathieu

Description

@sathieu
sathieu
opened on Oct 18, 2017

Description

Docker swarm is feeding the log with ton of messages:

Steps to reproduce the issue: I don't know
1.
2.
3.

Describe the results you received:

Oct 18 12:00:37 uvinct638 dockerd[23134]: time="2017-10-18T12:00:37.287564129+02:00" level=info msg="Node join event for uvinct636-206bf082d1ba/10.201.2.37"
Oct 18 12:00:39 uvinct638 dockerd[23134]: time="2017-10-18T12:00:39.311571426+02:00" level=warning msg="Deleting bridge mac mac 02:42:0a:00:01:07 failed, no such file or directory"
Oct 18 12:00:39 uvinct638 dockerd[23134]: time="2017-10-18T12:00:39.511290232+02:00" level=warning msg="Deleting bridge mac mac 02:42:0a:00:00:08 failed, no such file or directory"
Oct 18 12:00:41 uvinct638 dockerd[23134]: time="2017-10-18T12:00:41.119774822+02:00" level=warning msg="Deleting bridge mac mac 02:42:0a:ff:00:0b failed, no such file or directory"
Oct 18 12:01:00 uvinct638 dockerd[23134]: time="2017-10-18T12:01:00.851896965+02:00" level=info msg="Node join event for uvinct636-206bf082d1ba/10.201.2.37"
Oct 18 12:01:03 uvinct638 dockerd[23134]: time="2017-10-18T12:01:03.018329602+02:00" level=info msg="Node join event for uvinct637-60c8706643e8/10.201.2.44"
Oct 18 12:01:07 uvinct638 dockerd[23134]: time="2017-10-18T12:01:07.289408143+02:00" level=info msg="Node join event for uvinct637-60c8706643e8/10.201.2.44"
Oct 18 12:01:23 uvinct638 dockerd[23134]: time="2017-10-18T12:01:23.926907656+02:00" level=warning msg="Deleting bridge mac mac 02:42:0a:00:01:03 failed, no such file or directory"
Oct 18 12:01:24 uvinct638 dockerd[23134]: time="2017-10-18T12:01:24.557966863+02:00" level=warning msg="Deleting bridge mac mac 02:42:0a:00:00:05 failed, no such file or directory"
Oct 18 12:01:25 uvinct638 dockerd[23134]: time="2017-10-18T12:01:25.111865314+02:00" level=warning msg="Deleting bridge mac mac 02:42:0a:00:01:07 failed, no such file or directory"
Oct 18 12:01:25 uvinct638 dockerd[23134]: time="2017-10-18T12:01:25.322984150+02:00" level=warning msg="Deleting bridge mac mac 02:42:0a:00:00:08 failed, no such file or directory"
Oct 18 12:01:26 uvinct638 dockerd[23134]: time="2017-10-18T12:01:26.112205180+02:00" level=warning msg="Deleting bridge mac mac 02:42:0a:ff:00:08 failed, no such file or directory"
Oct 18 12:01:26 uvinct638 dockerd[23134]: time="2017-10-18T12:01:26.711065756+02:00" level=warning msg="Deleting bridge mac mac 02:42:0a:ff:00:0b failed, no such file or directory"
Oct 18 12:01:32 uvinct638 dockerd[23134]: time="2017-10-18T12:01:32.911132258+02:00" level=warning msg="Deleting bridge mac mac 02:42:0a:ff:00:08 failed, no such file or directory"
Oct 18 12:01:37 uvinct638 dockerd[23134]: time="2017-10-18T12:01:37.308287809+02:00" level=info msg="Node join event for uvinct637-60c8706643e8/10.201.2.44"
Oct 18 12:01:58 uvinct638 dockerd[23134]: time="2017-10-18T12:01:58.113213493+02:00" level=warning msg="Deleting bridge mac mac 02:42:0a:00:01:07 failed, no such file or directory"
Oct 18 12:01:58 uvinct638 dockerd[23134]: time="2017-10-18T12:01:58.511353025+02:00" level=warning msg="Deleting bridge mac mac 02:42:0a:00:00:08 failed, no such file or directory"
Oct 18 12:01:59 uvinct638 dockerd[23134]: time="2017-10-18T12:01:59.311016393+02:00" level=warning msg="Deleting bridge mac mac 02:42:0a:ff:00:0b failed, no such file or directory"
[...]
Oct 18 11:48:47 uvinct638 dockerd[23134]: time="2017-10-18T11:48:47.503053675+02:00" level=warning msg="rmServiceBinding handleEpTableEvent chat-i_mongo 0b4829f2d1a597a997882cb67725bd2910134dbbf27536b419df21f202cf3956 aborted c.serviceBindings[skey] !ok"
Oct 18 11:48:47 uvinct638 dockerd[23134]: time="2017-10-18T11:48:47.503949504+02:00" level=warning msg="rmServiceBinding handleEpTableEvent chat-i_rocketchat 0b4e3fff58fba83dfa44331555e06ff74bab13b0bd740de0c93f01558346dda2 aborted c.serviceBindings[skey] !ok"
Oct 18 11:48:47 uvinct638 dockerd[23134]: time="2017-10-18T11:48:47.504829213+02:00" level=warning msg="rmServiceBinding handleEpTableEvent chat-i_mongo 0b54178494cfde0ca6c848ed13355e71d73947ee625880e459ac309cdf2e713d aborted c.serviceBindings[skey] !ok"
Oct 18 11:48:47 uvinct638 dockerd[23134]: time="2017-10-18T11:48:47.505675639+02:00" level=warning msg="rmServiceBinding handleEpTableEvent chat-i_rocketchat 0b576b12b88cd8281bdd6bd4bf7c5658896bbd43fa6b7935e5ff56f112997dd6 aborted c.serviceBindings[skey] !ok"
Oct 18 11:48:47 uvinct638 dockerd[23134]: time="2017-10-18T11:48:47.506549141+02:00" level=warning msg="rmServiceBinding handleEpTableEvent chat-i_rocketchat 0b62fa3abaaf9cad767db895b35aad1f13ba8ab64180a61b5504bb9c520afdad aborted c.serviceBindings[skey] !ok"
Oct 18 11:48:47 uvinct638 dockerd[23134]: time="2017-10-18T11:48:47.507438961+02:00" level=warning msg="rmServiceBinding handleEpTableEvent chat-i_mongo 0b72e279acbd11f35af9bbd1e825e97d5fa7a10ba931855590428e07677d9717 aborted c.serviceBindings[skey] !ok"
[...]
Oct 18 11:43:50 uvinct638 dockerd[23134]: time="2017-10-18T11:43:50.192687109+02:00" level=warning msg="Peer operation failed:could not delete fdb entry into the sandbox: could not find the neighbor entry to delete op:&{2 pss2mr0slcazui1cwi4fceizf a44e991045a93533ad86e6670700e7e948bbb2d0737b44059485831c1a94d873 [0 0 0 0 0 0 0 0 0 0 255 255 10 0 1 3] [255 255 255 0] [2 66 10 0 1 3] [0 0 0 0 0 0 0 0 0 0 255 255 10 201 2 37] true false false false EventNotify}"
Oct 18 11:43:50 uvinct638 dockerd[23134]: time="2017-10-18T11:43:50.193398745+02:00" level=warning msg="Peer operation failed:could not delete fdb entry into the sandbox: could not find the neighbor entry to delete op:&{2 pss2mr0slcazui1cwi4fceizf a4531fca31b0504ed6583940582e4ab1bb8c6e771d130f80b7e7085b25d2aad3 [0 0 0 0 0 0 0 0 0 0 255 255 10 0 1 7] [255 255 255 0] [2 66 10 0 1 7] [0 0 0 0 0 0 0 0 0 0 255 255 10 201 2 37] true false false false EventNotify}"
Oct 18 11:43:50 uvinct638 dockerd[23134]: time="2017-10-18T11:43:50.194105401+02:00" level=warning msg="Peer operation failed:could not delete fdb entry into the sandbox: could not find the neighbor entry to delete op:&{2 pss2mr0slcazui1cwi4fceizf a471639bddc94e6cff8dc2062e3077cb0bd34f0f8406b2c73475c4462864bf50 [0 0 0 0 0 0 0 0 0 0 255 255 10 0 1 7] [255 255 255 0] [2 66 10 0 1 7] [0 0 0 0 0 0 0 0 0 0 255 255 10 201 2 37] true false false false EventNotify}"
Oct 18 11:43:50 uvinct638 dockerd[23134]: time="2017-10-18T11:43:50.194855865+02:00" level=warning msg="Peer operation failed:could not delete fdb entry into the sandbox: could not find the neighbor entry to delete op:&{2 pss2mr0slcazui1cwi4fceizf a47d5326131e7faaa05fe89bad70d3e69e8c06ea9afa71efd225cd850b9faae2 [0 0 0 0 0 0 0 0 0 0 255 255 10 0 1 3] [255 255 255 0] [2 66 10 0 1 3] [0 0 0 0 0 0 0 0 0 0 255 255 10 201 2 37] true false false false EventNotify}"
Oct 18 11:43:50 uvinct638 dockerd[23134]: time="2017-10-18T11:43:50.195590399+02:00" level=warning msg="Peer operation failed:could not delete fdb entry into the sandbox: could not find the neighbor entry to delete op:&{2 pss2mr0slcazui1cwi4fceizf a47dc9a7bf002494046c662a5f5284b442f29bc05dc47d2ac508cc4d2431af4a [0 0 0 0 0 0 0 0 0 0 255 255 10 0 1 3] [255 255 255 0] [2 66 10 0 1 3] [0 0 0 0 0 0 0 0 0 0 255 255 10 201 2 37] true false false false EventNotify}"
Oct 18 11:43:50 uvinct638 dockerd[23134]: time="2017-10-18T11:43:50.196310291+02:00" level=warning msg="Peer operation failed:could not delete fdb entry into the sandbox: could not find the neighbor entry to delete op:&{2 pss2mr0slcazui1cwi4fceizf a488d7a7a7f0f76274f98a9d8a2d20ee2952be453fb3442c37781486ba582a66 [0 0 0 0 0 0 0 0 0 0 255 255 10 0 1 7] [255 255 255 0] [2 66 10 0 1 7] [0 0 0 0 0 0 0 0 0 0 255 255 10 201 2 37] true false false false EventNotify}"

[...]

Describe the results you expected:
Not that much logs!

Additional information you deem important (e.g. issue happens only occasionally):

I recently upgraded from 17.06.0 to 17.06.2

Output of docker version:

Client:
 Version:      17.06.2-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   cec0b72
 Built:        Tue Sep  5 20:00:06 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.2-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   cec0b72
 Built:        Tue Sep  5 19:58:57 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 14
 Running: 3
 Paused: 0
 Stopped: 11
Images: 7
Server Version: 17.06.2-ce
Storage Driver: devicemapper
 Pool Name: docker-thinpool
 Pool Blocksize: 524.3kB
 Base Device Size: 10.74GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 5.047GB
 Data Space Total: 40.8GB
 Data Space Available: 35.75GB
 Metadata Space Used: 1.88MB
 Metadata Space Total: 427.8MB
 Metadata Space Available: 425.9MB
 Thin Pool Minimum Free Space: 4.079GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.137 (2016-11-30)
Logging Driver: gelf
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: ibf4iqt5ng4w5fgxn1c24r68j
 Is Manager: true
 ClusterID: q2uv8gycua86cxnacbyam6kl4
 Managers: 3
 Nodes: 3
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Root Rotation In Progress: false
 Node Address: 10.201.2.51
 Manager Addresses:
  10.201.2.37:2377
  10.201.2.44:2377
  10.201.2.51:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.0-4-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 5.823GiB
Name: uvinct638
ID: Y6XX:4SE2:JSKF:YH2P:ZO7X:PT5K:TWCQ:VFUL:5GOY:EZU4:ECI4:VNLZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: http://proxy-pp.nantes.net:8080/
No Proxy: localhost,docker-registry.nantesmetropole.fr,gitlab.nantesmetropole.fr
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

VMware ESXi

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/networkingNetworkingarea/swarmversion/17.06

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions