Skip to content

Docker network connection time outs to container from host over time #48082

@Commifreak

Description

@Commifreak

Description

Seems I see a similar bug like docker/for-win#8861 and/or docker/for-mac#7116 but I dont know if they are really related.

However, I use docker CE 26.1.4 on Ubuntu 22.04 and see similar issue:

After some time, requests to (or even between) containers just time out like:

# curl -vvv 10.10.1.6:3000
*   Trying 10.10.1.6:3000...
* connect to 10.10.1.6 port 3000 failed: No route to host
* Failed to connect to 10.10.1.6 port 3000 after 569 ms: No route to host
* Closing connection 0
curl: (7) Failed to connect to 10.10.1.6 port 3000 after 569 ms: No route to host

Doing this multiple times mostly works but sometimes it just takes several seconds or is just not working.

A ping looks like:

 ping 10.10.1.6
PING 10.10.1.6 (10.10.1.6) 56(84) bytes of data.
From 10.10.1.2 icmp_seq=2 Redirect Host(New nexthop: 10.10.1.6)
From 10.10.1.2 icmp_seq=3 Redirect Host(New nexthop: 10.10.1.6)
From 10.10.1.2 icmp_seq=4 Redirect Host(New nexthop: 10.10.1.6)
From 10.10.1.2 icmp_seq=5 Redirect Host(New nexthop: 10.10.1.6)
From 10.10.1.2 icmp_seq=6 Redirect Host(New nexthop: 10.10.1.6)
From 10.10.1.2 icmp_seq=8 Redirect Host(New nexthop: 10.10.1.6)
64 bytes from 10.10.1.6: icmp_seq=10 ttl=64 time=0.170 ms
From 10.10.1.2 icmp_seq=11 Redirect Host(New nexthop: 10.10.1.6)
From 10.10.1.2 icmp_seq=9 Destination Host Unreachable
From 10.10.1.2 icmp_seq=12 Destination Host Unreachable
From 10.10.1.2 icmp_seq=13 Destination Host Unreachable
From 10.10.1.2 icmp_seq=14 Destination Host Unreachable
64 bytes from 10.10.1.6: icmp_seq=15 ttl=64 time=0.121 ms
64 bytes from 10.10.1.6: icmp_seq=17 ttl=64 time=0.115 ms
From 10.10.1.2 icmp_seq=16 Destination Host Unreachable
From 10.10.1.2 icmp_seq=18 Destination Host Unreachable
From 10.10.1.2 icmp_seq=19 Destination Host Unreachable
From 10.10.1.2 icmp_seq=20 Destination Host Unreachable
From 10.10.1.2 icmp_seq=21 Destination Host Unreachable
64 bytes from 10.10.1.6: icmp_seq=22 ttl=64 time=0.047 ms
From 10.10.1.2 icmp_seq=23 Destination Host Unreachable
From 10.10.1.2 icmp_seq=24 Destination Host Unreachable
From 10.10.1.2 icmp_seq=25 Destination Host Unreachable
64 bytes from 10.10.1.6: icmp_seq=26 ttl=64 time=0.083 ms
64 bytes from 10.10.1.6: icmp_seq=27 ttl=64 time=0.052 ms
64 bytes from 10.10.1.6: icmp_seq=29 ttl=64 time=0.106 ms
64 bytes from 10.10.1.6: icmp_seq=30 ttl=64 time=0.039 ms
From 10.10.1.2 icmp_seq=28 Destination Host Unreachable
64 bytes from 10.10.1.6: icmp_seq=32 ttl=64 time=0.093 ms
From 10.10.1.2 icmp_seq=31 Destination Host Unreachable
From 10.10.1.2 icmp_seq=33 Destination Host Unreachable
64 bytes from 10.10.1.6: icmp_seq=34 ttl=64 time=0.044 ms

Just while pasting this: 10.10.1.2 is another container. I dont know why ping is telling me something about it?

Reproduce

Honestly, I dont know.

Expected behavior

No response

docker version

Client: Docker Engine - Community
 Version:           26.1.4
 API version:       1.45
 Go version:        go1.21.11
 Git commit:        5650f9b
 Built:             Wed Jun  5 11:28:57 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.4
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.11
  Git commit:       de5c9cf
  Built:            Wed Jun  5 11:28:57 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.33
  GitCommit:        d2d58213f83a351ca8f528a95fbd145f5654e957
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    26.1.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 11
  Running: 10
  Paused: 0
  Stopped: 1
 Images: 44
 Server Version: 26.1.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d2d58213f83a351ca8f528a95fbd145f5654e957
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-113-generic
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 19.53GiB
 Name: docker2204
 ID: 21e7fd1e-f89b-4bc8-a786-affac52de37b
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http://192.168.205.10:8000/
 HTTPS Proxy: http://192.168.205.10:8000/
 No Proxy: localhost,127.0.0.1,.company.tld
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 10.10.0.0/16, Size: 24

Additional Info

The default bridge is NOT in use. I created a new custom bridge:

[
    {
        "Name": "myBridge",
        "Id": "04830e3ba77e4aff0821396070a24b0039d9b7ac349ae144734f2ab5cc6f4c09",
        "Created": "2024-06-18T07:27:07.685670272+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "10.10.1.0/24",
                    "Gateway": "10.10.1.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {....},
        "Options": {},
        "Labels": {}
    }
]

System route table:

default via 192.168.205.10 dev ens33 proto static
10.10.0.0/24 dev docker0 proto kernel scope link src 10.10.0.1 linkdown
10.10.1.0/24 dev br-04830e3ba77e proto kernel scope link src 10.10.1.1
10.10.3.0/24 dev br-4b3a2ceb3bc7 proto kernel scope link src 10.10.3.1
10.10.18.0/24 dev br-2286be86007b proto kernel scope link src 10.10.18.1
192.168.205.0/24 dev ens33 proto kernel scope link src 192.168.205.20

The mentioned container above (10.10.1.6) is a gitea. But its not just this container. Some days its another in the same bridge net.

I dont know where to start searching. docker logs does not mention any fault.

Thanks for taking some time to look onto this! ❤

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/networkingNetworkingkind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.version/26.1

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions