Repository navigation
Connection refused right after container start #71
Description
There is a problem with migration from Docker Desktop to Colima docker environment.
When the container is just started, it's mapped port is not available immediately.
I'm not quite sure about all what happens underneath (is it an issue of Lima/Colima/jna/docker-java), but let me give my assumption, the demo is provided as well.
We use testcontainers, a popular framework that bootstraps docker containers for test purposes, in our case it's Postgres. We use image postgres:11.6-alpine.
When the container starts, it allows to retrieve mapped port and right after that the connection is set up.
With Docker desktop it works just fine, but with colima the retry-with-interval workaround is required. Default logic was not expecting this, it was just working fine for years.
My assumption is that the logic that retrieves mapped port is not blocked by startup await, while in Docker desktop there is a strong guarantee about it.
I've created a demo project that shows the issue.
If you start with Docker Desktop, you will see log:
...testcontainers initialization without issues...
...
jdbc:postgresql://localhost:55001/test?loggerLevel=OFF
>>>> Successfully first time
If you start the same test with Colima, you will see log:
...testcontainers initialization with many retry failures (bootstraps service Ryuk container first)
...
jdbc:postgresql://localhost:49154/test?loggerLevel=OFF
>>>> First exception: org.postgresql.util.PSQLException: Connection to localhost:49154 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
...sleep here...
>>>> Successfully second time
The mapped port is obtained in internal logic in this line:
String jdbcUrl = container.getJdbcUrl();
Worth to mention, that on Apple M1 this failure is more explicit (probably because it's faster), with Intel I see only testcontainers Ryuk failures (most probably they added this retry policy to cover the described issue), but still there is an obvious problem.
Let me know if I can provide more details or more convenient example (not sure that Java example is good enough for you).
Activity
seregamorph commentedon Nov 25, 2021
The full log for Colima execution:
jandubois commentedon Nov 25, 2021
This is expected, and due to the way port forwarding in Lima works: the guest agent running inside the VM checks every 5 seconds for open ports. It sends any changes since the previous check to the host agent (running on the host), which will in turn create a tunnel for each new port, and tear down the tunnel for each port that has been closed.
So you should expect to wait on average about 2½s before the port becomes available on the host, and potentially up to 5s (or more if the machine is busy).
To avoid the delay you would need to use an additional vmnet port, via
vde_vmnet. I don't know if Colima exposes support for this, but Lima can manage the additional daemons automatically. It does requirerootto set up initially.abiosoft commentedon Nov 30, 2021
This is something to consider for future releases
abiosoft commentedon Jan 13, 2022
The plan is to assign reachable IP addresses to the VM so they can be accessed directly in scenarios like this.
It is currently doable with lima override and network config but it is not user friendly.
workaround while abiosoft/colima#71 is open
a-inacio commentedon Feb 23, 2022
On my case (MacOS), due to dropping docker desktop I realised that my
/var/run/docker.sockwas still targeting the old file (this would then be making other stuff to fail).Solved it by removing the old symlink and then
sudo ln -s /Users/$USER/.colima/docker.sock /var/run/docker.sock.abiosoft commentedon Mar 19, 2022
Kindly install the current development version with
brew install --HEAD colima.You can retrieve the IP address via
colima ls, the VM should always be reachable via the IP address.cristianrgreco commentedon Apr 6, 2023
I am running into this on a Linux machine, where although the port is bound, I'm unable to connect right away.
divStar commentedon May 8, 2023
I'm afraid I am also experiencing this issue - with a postgres image (11-alpine). It seems flaky though and I am not sure where to even address this, but the issues only started occurring after installing
colima(instead of Docker Desktop).Ryuk starts and the database seems to start, too, but then sometimes it'll go through with the tests and sometimes I receive:
I am not sure this is a
colimaissue and I tried it with various configurations. Sometimes deleting the image and pulling it anew helps, sometimes it does not.Does anyone here perhaps have an idea?
seregamorph commentedon May 9, 2023
@divStar you can add a retry get connection logic before using the
DataSourcewrapping the postgres container. That's my code:abiosoft commentedon May 9, 2023
It is due to the way port forwarding works. There is a chance of up to 3s delay before a port is forwarded from the container to your localhost.
An option for a scenario like this would be to enable IP address for the VM and you can simply use the VM IP to access it. However, it is currently erratic on macOS ventura and would not be a recommended alternative.
Zordid commentedon Mar 8, 2024
I am currently also experiencing the issue that colima is running fine, I can start containers using docker-compose (A Kafka cluster) - but from the outside, I am unable to reach the forwarded ports. It's just connection refused. Then I stumbled on this comment and wanted to see my colima VM address, but there is none:
What does that mean?
My real problem I try to get solved though is this:
Any help is appreciated!
rfay commentedon Mar 8, 2024
We see this intermittently in about half of DDEV's automated test runs on Colima. It's a persistent problem.
jithinlal commentedon Nov 21, 2024
why is this closed?
DmitriiKholkin commentedon Dec 12, 2024
The problem is still here in .NET with TestContainers. Half of the starts fail.
abiosoft commentedon Dec 16, 2024
There needs to be a proper monitoring of Docker containers and dynamic port forwarding handled by Colima itself.
Rather than falling back to Lima's which can have a delay of up to 3 seconds.
divStar commentedon Jan 10, 2025
Is it possible to help with this somehow? The issue happens so often with zonky and testcontainers, that I am considering giving Rancher Desktop a try even though colima is perfect in terms of control and its minimalism...
abiosoft commentedon Jan 10, 2025
This has been prioritised.
rfay commentedon Jan 10, 2025
I think Rancher, Lima, and Colima share this basic problem but could be wrong. I think it's a Lima problem. It may also be related to the port-bind management discussed in
abiosoft commentedon Jan 10, 2025
Yeah, they are all affected by this issue.
abiosoft commentedon Feb 12, 2026
This has been long fixed.