TestContainers not working with Rancher Desktop #4791

Activity

1. kiview commented on Dec 20, 2021

   

   kiview

   on Dec 20, 2021

   Member

   More actions

   Hi @mcekovic, thanks for trying out Testcontainers with Rancher Desktop. In case Rancher Desktop tries to be fully compatible with the Docker API, I would suggest reporting this issue with Rancher Desktop since Testconainters works fine with Docker itself.

   Therefore, I will close this issue, but feel free to keep us in the loop.
2. kiview

   closed this as completedon Dec 20, 2021
3. kiview

   added

   resolution/wontfix

   on Dec 20, 2021
4. mcekovic commented on Dec 20, 2021

   

   mcekovic

   on Dec 20, 2021

   Author

   More actions

   OK, thanks for the response, will notify if something changes on this topic as Rancher Desktops approaches the first regular release.

   👍React with 👍1Reacted by Kevin Wittek
5. mcekovic commented on Jan 27, 2022

   

   mcekovic

   on Jan 27, 2022

   Author

   More actions

   Unfortunately no luck either with final 1.0.0 version of Rancher Desktop and Test Containers 1.16.3.
6. kiview commented on Jan 27, 2022

   

   kiview

   on Jan 27, 2022

   Member

   More actions

   Thanks for trying out. Does Rancher Desktop now expose the Docker API in some way? Maybe there is some configuration workaround that can make this work.
7. mcekovic commented on Jan 27, 2022

   

   mcekovic

   on Jan 27, 2022

   Last edited by mcekovic

   Author

   More actions

   RancherDesktop exposes DockerAPI via Moby. Docker API in Rancher Desktop on Windows is by default exposed at standard location: npipe:////./pipe/docker_engine. TestContainers manages to start communication to the Docker API, makes some calls, but then hang during the check for disk space:

   	at com.github.dockerjava.api.async.ResultCallbackTemplate.awaitCompletion(ResultCallbackTemplate.java:91)
   	at org.testcontainers.DockerClientFactory.checkDiskSpace(DockerClientFactory.java:294)
   

   OK, fine, disk space check does not work in RancherDesktop.
   However, when I disable TestContainers checks (checks.disable=true) then it loads the image, it can be seen with Docker command line that container is created and started:

   $ docker container ls
   CONTAINER ID   IMAGE                              COMMAND                  CREATED          STATUS          PORTS
       NAMES
   105312cf6289   redis:latest                       "docker-entrypoint.s…"   7 seconds ago    Up 7 seconds    0.0.0.0:49166->6379/tcp, :::49166->6379/tcp   distracted_rhodes
   

   TestContainers seems to verify exposed Redis port successfully:

   [testcontainers-wait-1] DEBUG org.testcontainers.containers.wait.strategy.HostPortWaitStrategy - External port check passed for [6379] mapped as [49166] in PT0.3359626S

   but after that TestContainers waits at (stacktrace):

   	at org.testcontainers.containers.wait.strategy.HostPortWaitStrategy.waitUntilReady(HostPortWaitStrategy.java:51)
   	at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:51)
   	at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:929)
   	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:468)
   	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:331)
   	at org.testcontainers.containers.GenericContainer$$Lambda$447/0x0000000800dfc660.call(Unknown Source)
   	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
   	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:329)
   	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:317)
   

   and finally timeouts with:

   [main] DEBUG 🐳 [redis:latest] - Wait strategy threw an exception
   [testcontainers-wait-0] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.classic.InternalHttpClient - ep-0000000F: cancel
   org.testcontainers.containers.ContainerLaunchException: Timed out waiting for container port to open (localhost ports: [49166] should be listening)
   	at org.testcontainers.containers.wait.strategy.HostPortWaitStrategy.waitUntilReady(HostPortWaitStrategy.java:90)
   	at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:51)
   	at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:929)
   	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:468)
   	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:331)
   	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
   	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:329)
   	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:317)
   
   ...
   
   [main] ERROR 🐳 [redis:latest] - Log output from the failed container:
   1:C 27 Jan 2022 20:02:50.144 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
   1:C 27 Jan 2022 20:02:50.144 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
   1:C 27 Jan 2022 20:02:50.144 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
   1:M 27 Jan 2022 20:02:50.145 * monotonic clock: POSIX clock_gettime
   1:M 27 Jan 2022 20:02:50.145 * Running mode=standalone, port=6379.
   1:M 27 Jan 2022 20:02:50.145 # Server initialized
   1:M 27 Jan 2022 20:02:50.145 * Ready to accept connections
   
   
   org.testcontainers.containers.ContainerLaunchException: Container startup failed
   	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:336)
   	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:317)
   
   ...
   
   Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
   	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
   	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:329)
   	... 69 more
   Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
   	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:525)
   	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:331)
   	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
   	... 70 more
   Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for container port to open (localhost ports: [49166] should be listening)
   	at org.testcontainers.containers.wait.strategy.HostPortWaitStrategy.waitUntilReady(HostPortWaitStrategy.java:90)
   	at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:51)
   	at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:929)
   	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:468)
   	... 72 more
   

   However, during the wait, I can see with netstat on the Windows host that random port 49166 is indeed listening:

   TCP 127.0.0.1:49166 0.0.0.0:0 LISTENING 10276

   Are there some hints about incompatibility so I can pass the info to the Rancher Desktop team?
8. mcekovic commented on Jan 27, 2022

   

   mcekovic

   on Jan 27, 2022

   Last edited by mcekovic

   Author

   More actions

   Did some debugging. Seems that org.testcontainers.containers.wait.internal.InternalCommandPortListeningCheck hangs when executing in container via Java Docker client:

      public ExecStartCmd execStartCmd(String execId) {
         return new ExecStartCmdImpl(this.getDockerCmdExecFactory().createExecStartCmdExec(), execId);
      }
   

   This is root cause for both cases, when checking available disk space and when checking internal container listening ports.
9. mcekovic commented on Jan 27, 2022

   

   mcekovic

   on Jan 27, 2022

   Last edited by mcekovic

   Author

   More actions

   Seems that et the end of exec command (Docker API call) Docker returns:

   [docker-java-stream--729518257] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.wire - http-outgoing-1 << "end of stream"

   While Moby in Rancher Desktop:

   [docker-java-stream-1614402714] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.wire - http-outgoing-1 << "[\r][\n]"

   Would it be something to notify Rancher Desktop or Moby project members?
10. jrmcdonald commented on Jan 28, 2022

    

    jrmcdonald

    on Jan 28, 2022

    More actions

    I think this is related to this issue in Rancher-Desktop:

    rancher-sandbox/rancher-desktop#1171

    👍React with 👍1Reacted by Mileta Cekovic
11. kiview commented on Feb 1, 2022

    

    kiview

    on Feb 1, 2022

    Member

    More actions

    Seems they have fixed the issue in Rancher-Desktop:
    rancher-sandbox/rancher-desktop#1374
12. mcekovic commented on Feb 1, 2022

    

    mcekovic

    on Feb 1, 2022

    Author

    More actions

    Thanks for notification, will try when they release it.
13. mcekovic commented on Feb 4, 2022

    

    mcekovic

    on Feb 4, 2022

    Last edited by mcekovic

    Author

    More actions

    Just note that TestContainers works nicely with Rancher Desktop 1.0.1 with the fix for rancher-sandbox/rancher-desktop#1374 for me and for several of my colleagues.
    Thank you for the support.
14. kiview commented on Feb 4, 2022

    

    kiview

    on Feb 4, 2022

    Member

    More actions

    Thanks for letting us know @mcekovic, this is great news indeed! 🙌
    Anything out of the ordinary to make it work together, or is the setup straightforward?
15. mcekovic commented on Feb 4, 2022

    

    mcekovic

    on Feb 4, 2022

    Last edited by mcekovic

    Author

    More actions

    Just installing Rancher Desktop and switching to use dockerd (moby) container runtime (default is containerd) on the 'Kubernetes Settings' screen (there is a screenshot here) and everything works out-of-the box :).

    No need for setting DOCKER_HOST env. variable or anything else (like when manually using Docker installed inside WSL).

    👍React with 👍3Reacted by Kevin Wittek, Sergei Egorov and Rafael Coelho

    🎉React with 🎉1Reacted by Sergei Egorov
16. edeandrea commented on Feb 9, 2022

    

    edeandrea

    on Feb 9, 2022

    Last edited by edeandrea

    Contributor

    More actions

    @mcekovic Did you have to do anything else? It doesn't work for me...I'm seeing things like this:

    10:34:15 INFO  [or.te.do.DockerClientProviderStrategy] (build-32) Loaded org.testcontainers.dockerclient.UnixSocketClientProviderStrategy from ~/.testcontainers.properties, will try it first
    10:34:16 INFO  [or.te.do.DockerClientProviderStrategy] (build-32) Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
    10:34:16 INFO  [or.te.DockerClientFactory] (build-32) Docker host IP address is localhost
    10:34:16 INFO  [or.te.DockerClientFactory] (build-32) Connected to docker: 
      Server Version: 20.10.11
      API Version: 1.41
      Operating System: Alpine Linux v3.14
      Total Memory: 3942 MB
    10:34:16 INFO  [or.te.ut.ImageNameSubstitutor] (build-32) Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
    10:34:16 WARN  [io.qu.de.st.ClassTransformingBuildStep] (build-7) Cannot transform io.smallrye.context.Jdk12CompletionStageWrapper as its containing application archive could not be found.
    10:34:16 WARN  [io.qu.de.st.ClassTransformingBuildStep] (build-7) Cannot transform io.smallrye.context.Jdk12CompletableFutureWrapper as its containing application archive could not be found.
    10:34:17 WARN  [or.te.ut.ResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at localhost:49157: java.net.ConnectException: Connection refused (Connection refused)
    	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
    	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
    	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
    	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
    	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
    	at java.base/java.net.Socket.connect(Socket.java:591)
    	at org.testcontainers.utility.ResourceReaper.lambda$null$3(ResourceReaper.java:194)
    	at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
    	at org.testcontainers.utility.ResourceReaper.lambda$start$4(ResourceReaper.java:190)
    	at java.base/java.lang.Thread.run(Thread.java:834)
    
    10:34:17 WARN  [or.te.ut.ResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at localhost:49157: java.net.ConnectException: Connection refused (Connection refused)
    	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
    	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
    	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
    	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
    	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
    	at java.base/java.net.Socket.connect(Socket.java:591)
    	at org.testcontainers.utility.ResourceReaper.lambda$null$3(ResourceReaper.java:194)
    	at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
    	at org.testcontainers.utility.ResourceReaper.lambda$start$4(ResourceReaper.java:190)
    	at java.base/java.lang.Thread.run(Thread.java:834)
    
    10:34:17 WARN  [or.te.ut.ResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at localhost:49157: java.net.ConnectException: Connection refused (Connection refused)
    	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
    	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
    	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
    	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
    	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
    	at java.base/java.net.Socket.connect(Socket.java:591)
    	at org.testcontainers.utility.ResourceReaper.lambda$null$3(ResourceReaper.java:194)
    	at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
    	at org.testcontainers.utility.ResourceReaper.lambda$start$4(ResourceReaper.java:190)
    	at java.base/java.lang.Thread.run(Thread.java:834)
    
    10:34:18 WARN  [or.te.ut.ResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at localhost:49157: java.net.ConnectException: Connection refused (Connection refused)
    	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
    	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
    	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
    	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
    	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
    	at java.base/java.net.Socket.connect(Socket.java:591)
    	at org.testcontainers.utility.ResourceReaper.lambda$null$3(ResourceReaper.java:194)
    	at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
    	at org.testcontainers.utility.ResourceReaper.lambda$start$4(ResourceReaper.java:190)
    	at java.base/java.lang.Thread.run(Thread.java:834)
    
    10:34:18 WARN  [or.te.ut.ResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at localhost:49157: java.net.ConnectException: Connection refused (Connection refused)
    	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
    	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
    	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
    	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
    	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
    	at java.base/java.net.Socket.connect(Socket.java:591)
    	at org.testcontainers.utility.ResourceReaper.lambda$null$3(ResourceReaper.java:194)
    	at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
    	at org.testcontainers.utility.ResourceReaper.lambda$start$4(ResourceReaper.java:190)
    	at java.base/java.lang.Thread.run(Thread.java:834)
    
    10:34:18 INFO  [or.te.DockerClientFactory] (build-32) Ryuk started - will monitor and terminate Testcontainers containers on JVM exit
17. mcekovic commented on Feb 9, 2022

    

    mcekovic

    on Feb 9, 2022

    Author

    More actions

    Are you running Rancher Desktop on Mac OS?
    For me it worked out-of-box on Windows 10 (just switching to dockerd).
    Seems the stack traces you get are only warnings, it manages to connect to Ryuk image finally. Do you have some other errors later?
18. edeandrea commented on Feb 9, 2022

    

    edeandrea

    on Feb 9, 2022

    Contributor

    More actions

    I am on macos. I guess it went on and seemed to work, but I had never seen these warnings before
19. kiview commented on Feb 10, 2022

    

    kiview

    on Feb 10, 2022

    Member

    More actions

    I think it took just a bit until the port was mapped and reachable for the host.

    What TC does internally is launch Ryuk, and then have a thread communicating with Ryuk. Initially, we wait up to ryuk.container.timeout seconds (which defaults to 30s) to be able to communicate with Ryuk (and retry), so that's why you see these logs.
20. edeandrea commented on Feb 10, 2022

    

    edeandrea

    on Feb 10, 2022

    Contributor

    More actions

    Thanks @kiview . Part of it must be that rancher desktop is spinning stuff up on a k8s distro rather than just launching a container on the host, which takes time.
21. Sandmania commented on Feb 14, 2022

    

    Sandmania

    on Feb 14, 2022

    Last edited by Sandmania

    More actions

    Sorry, just to clarify, @edeandrea did yout get Testcontainers + Rancher Desktop + macOS combination to work?

    Edit: it does indeed work. My legacy project just had some dependency issues.
22. kiview commented on Feb 14, 2022

    

    kiview

    on Feb 14, 2022

    Member

    More actions

    Yes, it worked for him on macOS.

    👍React with 👍2Reacted by Jouni Latvatalo and Al
23. pidzama commented on Apr 8, 2022

    

    pidzama

    on Apr 8, 2022

    Last edited by pidzama

    More actions

    The same issue can be reproduced on Rancher Desktop 1.2.1 + Windows 10, with this sample project https://github.com/lreimer/hands-on-testcontainers.git
24. Krijger commented on Apr 26, 2022

    

    Krijger

    on Apr 26, 2022

    More actions

    @kiview was this on M1 arm64 architecture? I must admit I'm having issues
25. kiview commented on Apr 26, 2022

    

    kiview

    on Apr 26, 2022

    Member

    More actions

    @edeandrea would need to answer this.
26. edeandrea commented on Apr 26, 2022

    

    edeandrea

    on Apr 26, 2022

    Contributor

    More actions

    No it was intel
27. Krijger commented on Apr 28, 2022

    

    Krijger

    on Apr 28, 2022

    Last edited by Krijger

    More actions

    @edeandrea thanks. I had a different issue, and it was fixed by fixing my JNA dependency according to #3834 . Now I need to see how to run the MS SQL tests :)
28. kiview commented on Sep 12, 2022

    

    kiview

    on Sep 12, 2022

    Member

    More actions

    For future reference, I found this related upstream issue at Rancher:
    rancher-sandbox/rancher-desktop#2609
29. eddumelendez commented on Sep 28, 2023

    

    eddumelendez

    on Sep 28, 2023

    Member

    More actions

    There is Rancher Desktop section that has been added about how to set up Testcontainers.