Repository navigation
Can't connect to docker container's exposed port with network's gateway ip from other container #5588
Description
- I have tried with the latest version of Docker Desktop
- I have tried disabling enabled experimental features
- I have uploaded Diagnostics
Expected behavior
When creating a docker container with an exposed port (not binding it), be able to reach the application running on it from another container in the same network using the network's gateway ip and the port the container was assigned on the host. HostPort in the docker container's HostConfig, PortBindings should be empty. This is the behaviour on Ubuntu.
Actual behavior
When creating a docker container with an exposed port (not binding it), can reach the container from the host with localhost and the newly assigned port but not from another container with their network's gateway ip and the newly assigned port. HostPort in the docker container's HostConfig, PortBindings` is set to "0". This is the behaviour on Mac.
Information
The problem is reproducible. Not sure if it's new, we just noticed it and the not sure whether it came with an update. Added steps to reproduce below. When the run a container and use the -p <port> option to map the container's port to a randomly assigned one on the host machine, we noticed that HostPort under HostConfig and PortBindings in the container's config is set to "0", on Ubuntu it is an empty string. If we were to specify the port we want to bind to it works fine -p <host_port>:<container_port>. In this case the HostPort under HostConfig/PortBindings would be the host_port we specified on both Mac and Ubuntu. We suspect this is causing the issue.
macOS version: 10.15.6
Docker Desktop Version: 3.3.1
Docker version: 20.10.5
Ubuntu host tested on: 18.04.5
Docker version: 20.10.6
Reproduce
The following script would reproduce the issue, it would fail on Mac and succeed on Ubuntu:
#!/usr/bin/env bash
echo "Starting PostgreSQL database in container"
CONTAINER=$(docker run -d -p 5432 -e POSTGRES_USER=test -e POSTGRES_PASSWORD=test -e POSTGRES_DB=test postgres:9.6)
echo "PostgreSQL database started in container: $CONTAINER"
sleep 5 # Making sure db will be responsive
PORT=$(docker port $CONTAINER | head -n 1 | rev | cut -d: -f1 | rev)
GATEWAY_IP=$(docker network inspect -f '{{range .IPAM.Config}}{{.Gateway}}{{end}}' bridge)
echo
echo "Gateway IP: $GATEWAY_IP, port: $PORT"
echo
echo "Querying database version from another container"
docker run postgres:9.6 psql "host=$GATEWAY_IP port=$PORT dbname=test user=test password=test" -c "select version()"
echo
echo "HostConfig PortBindings of container: $CONTAINER"
docker inspect -f '{{range $p, $conf := .HostConfig.PortBindings}} {{$p}} -> HostIp: {{(index $conf 0).HostIp}} HostPort: {{(index $conf 0).HostPort}} {{end}}' $CONTAINER
echo
echo "NetworkSettings Ports of container: $CONTAINER"
docker inspect -f '{{range $p, $conf := .NetworkSettings.Ports}} {{$p}} -> HostIp: {{(index $conf 0).HostIp}} HostPort: {{(index $conf 0).HostPort}} {{end}}' $CONTAINER
echo
echo "Stopping $CONTAINER"
docker stop $CONTAINER
Activity
nk412 commentedon Apr 16, 2021
+1 seeing the same issue
djs55 commentedon Apr 16, 2021
Thanks for your report.
For maximum compatibility/portability between systems I recommend
docker run -pto expose ports on the host (as you are doing)The networking implementation on macOS is different to native Linux as there are two kernels with separate network stacks. On Linux
docker run -phas the side-effect you observe, but on macOS the port has to be exposed differently, due to the different kernel.illesguy commentedon Apr 16, 2021
The issue we are having is within a Python library (https://github.com/testcontainers/testcontainers-python) that uses the approach I've described (network gateway ip + port) to communicate between containers it starts, so we can't change that. For now we were able to fix this, by explicitly binding the ports we want to expose instead letting docker do it.
But if you're saying using the internal DNS and IP is recommended, maybe that is a ticket to open on the repository of the library we are using.
ag-TJNII commentedon May 5, 2021
So is this a 'Won't Fix', then? This used to work fine on Docker for Mac. I support Dockerized utilities in both Linux and Mac environments and use this behavior often. I also tried hitting the listening container by it's IP when troubleshooting and that didn't work. when did this behavior change? I didn't see any notes about it in the changelogs.
I'm also unable to get back to a working state by uninstalling and installing an older version. I'm suspicious this is due to a recent networking components update, and I noticed the uninstall does not ask for privileged access like the install does. Does a uninstall also uninstall the networking components?
goatherder commentedon May 11, 2021
I see the same behaviour with https://github.com/testcontainers/testcontainers-go
docker-robott commentedon Aug 9, 2021
Issues go stale after 90 days of inactivity.
Mark the issue as fresh with
/remove-lifecycle stalecomment.Stale issues will be closed after an additional 30 days of inactivity.
Prevent issues from auto-closing with an
/lifecycle frozencomment.If this issue is safe to close now please do so.
Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale
rnorth commentedon Aug 26, 2021
We're looking into something that looks awfully similar with testcontainers-java, whereby GATEWAY_IP:PORT connections fail in Docker for Mac & Windows from one container to another.
testcontainers/testcontainers-java#4395
We're not quite sure where our investigations will lead us yet, but we'll update here if we find something useful.
rnorth commentedon Aug 26, 2021
/remove-lifecycle stale
^ on the basis that this used to work just fine and has become relied upon, so removal without warning is not without impacts.
rnorth commentedon Aug 26, 2021
Based on some initial tinkering there's something weird going on here, but it gives us some ideas for a workaround that could be applied to testcontainers-java and probably other similar libs:
I was always under the impression that
-p PORTand-p 0:PORTwould be effectively the same, but I'm sure there are many moving parts that make this not such a simple assertion 😆.Taking @illesguy's repro script above and simply sticking a
0:in the right place seems to fix it (with Docker for Mac 3.5.2, launching the script from inside a container).@djs55, for various use cases (particularly CI environments with 'docker wormhole' arrangements) the
GATEWAY_IP:PORTpattern has, for a long time, been really quite important to us and the network+named container pattern would be quite hard for users to migrate to.Could you perhaps give us a steer on:
-p 0:...approach, or whether there's an imminent risk that this might also break?Thank you in advance.
djs55 commentedon Oct 4, 2021
@rnorth sorry for the delay in replying. The observation about
0:8080working but8080not working is interesting. I'll add a ticket to our backlog to investigate: it might be fixable as you suggest.djs55 commentedon Oct 5, 2021
I've done a bit of investigating and I believe this problem is due to the fix for docker/for-win#10008 . On Mac and Windows
dockerdruns inside the VM, in a network namespace. When a random port is needed to expose a container (e.g.docker run -p 80) there was a bug wheredockerdwould choose a port which was free inside it's network namespace, but which wasn't free on the host. This was fixed by choosing a random port on both the host and on127.0.0.1in Linux, connecting them together, and rewriting thedocker psresults over/var/run/docker.sockto show the correct value for the host:There was a bug in this bugfix, in the test for "is the user requesting a random port". The code checks for the port being "", but not the port being "0". So the reason that
-p 0:...works at the moment is because it's avoiding this bugfix, by accident.Considering the possibility of making
-p 80and-p 0:80behave the same i.e. fixing the bug in the bugfix, I can see 2 further problems:127.0.0.1inside the Linux network namespace, so the port isn't available on the gateway IPdocker pson the host.I could probably fix (1) by binding internally on
0.0.0.0rather than127.0.0.1; hopefully this would be safe because the Linux network namespace withdockerdis almost empty. However this still leaves the problem of discovering the internal port number.It is possible to bypass the Docker API proxy which is rewriting the port number:
docker run -v /var/run/docker.sock:/var/run/docker.sock docker ps: would show the host portdocker run -v /var/run/docker.sock.raw:/var/run/docker.sock docker ps: would show the internal portI don't know how convenient this would be to change. It's a bit of an internal detail of Docker Desktop.
So in summary:
-p 0:...seems to work by accident rather than design 😱gesellix commentedon Oct 5, 2021
❤️
rnorth commentedon Oct 6, 2021
@djs55 thank you so much for the detailed analysis and explanation! I really appreciate you spending time looking into this.
We'll have some internal talks to discuss what we do at our end, but my immediate thoughts are (in no particular order):
I think I'd strongly echo @gesellix's ❤️ reaction: this ought to help us implement the 'accidental workaround' with much more confidence. Having the behaviour be covered by a test is a great idea - thanks.
I learned something new today 😄. Slightly longer term, this wouldn't be a trivial change for us, but might be something we could use from the Testcontainers side if essential. We already have to cater for other differences when we're running inside a container, so extending this might be practical if we can do it safely. My only concern would be that we'd be even deeper into coupling to Docker Desktop internals, so we have to consider that.
We'll keep this ticket updated with plans from the Testcontainers side.
Thanks again.
docker-robott commentedon Jan 4, 2022
Issues go stale after 90 days of inactivity.
Mark the issue as fresh with
/remove-lifecycle stalecomment.Stale issues will be closed after an additional 30 days of inactivity.
Prevent issues from auto-closing with an
/lifecycle frozencomment.If this issue is safe to close now please do so.
Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale
docker-robott commentedon Mar 5, 2022
Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.
If you have found a problem that seems similar to this, please open a new issue.
Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked