Description
With a node on 29.1.3 and another on 28.5.2 (or lower) it appears that encrypted overlay networks stop working i.e. no traffic can be sent between containers on the overlay network (DNS resolution of containers on the other node still works). If both are 29.1.3 or both are 28.5.2, traffic flows.
Reproduce
On node1
docker network create --driver overlay --attachable --opt encrypted encnet
docker run -d --name test1 --network encnet alpine sleep 1d
On node2
docker run --rm --name test2 --network encnet alpine sh -c "apk add -q iputils && ping -c3 test1"
PING test1 (10.0.2.2) 56(84) bytes of data.
--- test1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2066ms
Expected behavior
PING test1 (10.0.2.2) 56(84) bytes of data.
64 bytes from test1.encnet (10.0.2.2): icmp_seq=1 ttl=64 time=0.893 ms
64 bytes from test1.encnet (10.0.2.2): icmp_seq=2 ttl=64 time=1.94 ms
64 bytes from test1.encnet (10.0.2.2): icmp_seq=3 ttl=64 time=0.796 ms
docker version
Client: Docker Engine - Community
Version: 28.5.2
API version: 1.51
Go version: go1.25.3
Git commit: ecc6942
Built: Wed Nov 5 14:43:11 2025
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 28.5.2
API version: 1.51 (minimum version 1.24)
Go version: go1.25.3
Git commit: 89c5e8f
Built: Wed Nov 5 14:43:11 2025
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.7.28
GitCommit: b98a3aace656320842a23f4a392a33f46af97866
runc:
Version: 1.3.0
GitCommit: v1.3.0-0-g4ca628d1
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
Client: Docker Engine - Community
Version: 28.5.2
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.29.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.40.3
Path: /usr/libexec/docker/cli-plugins/docker-compose
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /root/.docker/cli-plugins/docker-sbom
Server:
Containers: 37
Running: 18
Paused: 0
Stopped: 19
Images: 30
Server Version: 28.5.2
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Swarm: active
NodeID: 8y1hquegv1kbinh82tijyteh1
Is Manager: true
ClusterID: zfdcacibg74wivs2j429pn45l
Managers: 1
Nodes: 2
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.15.13.189
Manager Addresses:
10.15.13.189:2377
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b98a3aace656320842a23f4a392a33f46af97866
runc version: v1.3.0-0-g4ca628d1
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.8.0-1044-aws
Operating System: Ubuntu 22.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 15.36GiB
Name: testhost
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: falseAdditional Info
Grateful for any information or workaround to make upgrades to 29 easier.
Description
With a node on 29.1.3 and another on 28.5.2 (or lower) it appears that encrypted overlay networks stop working i.e. no traffic can be sent between containers on the overlay network (DNS resolution of containers on the other node still works). If both are 29.1.3 or both are 28.5.2, traffic flows.
Reproduce
On node1
docker network create --driver overlay --attachable --opt encrypted encnetdocker run -d --name test1 --network encnet alpine sleep 1dOn node2
Expected behavior
docker version
Client: Docker Engine - Community Version: 28.5.2 API version: 1.51 Go version: go1.25.3 Git commit: ecc6942 Built: Wed Nov 5 14:43:11 2025 OS/Arch: linux/amd64 Context: default Server: Docker Engine - Community Engine: Version: 28.5.2 API version: 1.51 (minimum version 1.24) Go version: go1.25.3 Git commit: 89c5e8f Built: Wed Nov 5 14:43:11 2025 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.7.28 GitCommit: b98a3aace656320842a23f4a392a33f46af97866 runc: Version: 1.3.0 GitCommit: v1.3.0-0-g4ca628d1 docker-init: Version: 0.19.0 GitCommit: de40ad0docker info
Additional Info
Grateful for any information or workaround to make upgrades to 29 easier.