testcontainers wait indefinitely without effects when running in Gitlab-CI

I'm running a test based on testcontainers-python in a container running in Gitlab CI.

Basically the CI spawns a docker-in-docker container next to the container running the tests. I then set the environment variable DOCKER_HOST so that it can be used by testcontainers.

But in practice the test just get stuck, so I ran it with pytest -s to see the logs of testcontainers and got:

tests/test.py
Pulling image postgres:10
Container started:  4a1115dc98
Waiting to be ready...

And then it gets stuck indefinitely (I waited 20 minutes).

For the record, testcontainers-java works perfectly with this same situation, so it should be possible to make this work.

I'm not sure how I can get more debug information to understand what is happening, but shouldn't it timeout normally at least?

See also #25 for a different but maybe related problem and https://www.testcontainers.org/supported_docker_environment/continuous_integration/gitlab_ci/ for the how to use testcontainers-java in gitlab-ci

I currently get a timeout

Pulling image mysql:5.7.28
⠏
Container started:  0c46b2b744
Waiting to be ready...

I'm using the mysql container and the exact log output is:

======================================================================= ERRORS =======================================================================
_________________________________________________________ ERROR at setup of test_with_mysql __________________________________________________________

    @pytest.fixture(scope="session")
    def mysql():
>       mysql = MySqlContainer("mysql:5.7.28").start()

tests/service/test_functional.py:9:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
venv/lib/python3.7/site-packages/testcontainers/core/generic.py:49: in start
    self._connect()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

wrapped = <bound method DbContainer._connect of <testcontainers.mysql.MySqlContainer object at 0x112823110>>
instance = <testcontainers.mysql.MySqlContainer object at 0x112823110>, args = (), kwargs = {}

    @wrapt.decorator
    def wrapper(wrapped, instance, args, kwargs):
        exception = None
        print(crayons.yellow("Waiting to be ready..."))
        with blindspin.spinner():
            for _ in range(0, config.MAX_TRIES):
                try:
                    return wrapped(*args, **kwargs)
                except Exception as e:
                    sleep(config.SLEEP_TIME)
                    exception = e
            raise TimeoutException(
                """Wait time exceeded {0} sec.
                    Method {1}, args {2} , kwargs {3}.
                     Exception {4}""".format(config.MAX_TRIES,
                                             wrapped.__name__,
>                                            args, kwargs, exception))
E           testcontainers.core.exceptions.TimeoutException: Wait time exceeded 120 sec.
E                               Method _connect, args () , kwargs {}.
E                                Exception No module named 'pymysql'

venv/lib/python3.7/site-packages/testcontainers/core/waiting_utils.py:50: TimeoutException

I can see the container running when I check with $ docker ps. Also, I can make a new issue if that's helpful

Turns out I can't read and apologize for posting in the first place.

$ pip3 install pymysql fixed everything ha

having same issue with gitlab cicd

 $ python test.py
Pulling image mysql:5.7.17
Container started:  f2ea27afde
Waiting to be ready...
Traceback (most recent call last):
  File "test.py", line 12, in <module>
    config = MySqlContainer('mysql:5.7.17').start()
  File "/root/.local/lib/python3.6/site-packages/testcontainers/core/generic.py", line 49, in start
    self._connect()
  File "/root/.local/lib/python3.6/site-packages/wrapt/wrappers.py", line 603, in __call__
    args, kwargs)
  File "/root/.local/lib/python3.6/site-packages/testcontainers/core/waiting_utils.py", line 50, in wrapper
    args, kwargs, exception))
testcontainers.core.exceptions.TimeoutException: Wait time exceeded 120 sec.
                    Method _connect, args () , kwargs {}.
                     Exception (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '172.18.0.2' (timed out)")
(Background on this error at: http://sqlalche.me/e/e3q8)

im using docker:dind with gitlab cicd pipeline.
adding wait for mysql to boot and changing mysql host-name to 'docker' solved my problem

from testcontainers.compose import DockerCompose
from testcontainers.core.utils import inside_container
from testcontainers.core.waiting_utils import wait_container_is_ready

class MyDockerCompose():

    def __init__(self):

        if inside_container(): # if inside docker = gitlab ci/cd set host to docker
            mydb_hostname ='docker'

        self.compose: DockerCompose = DockerCompose(".")
        self.compose.start()
        self.wait_for()

    @wait_container_is_ready()
    def wait_for(self):
        # wait for docker mysql to accept connections
        os.environ['TC_POOLING_INTERVAL'] = '6'
        str_stdout = Utils.run_command(command="mysql --user=%s --password=%s --host=%s --execute 'SHOW DATABASES;'" % ('test', 'test', mydb_hostname))
        return 'MyDatabaseName' in str_stdout

Fixed in #85. Feel free to reopen if problems persist.

Excellent, I can confirm this works :)

Version 3.0 gets stuck for me at the same place but 2.6 works fine when using docker-in-docker with mounted socket.

@karmux

@karmux I will double check. Meanwhile, can you provide your .gitlab-ci.yml here?

@tuvshuud

@tuvshuud it's not Gitlab-CI but Docker images based on official Python images where this freezing happens (using -v /var/run/docker.sock:/var/run/docker.sock argument to docker run). Didn't create a new bug as it seemed related.

@karmux

@karmux, could you post a minimal reproducible example please so we can have a look?

@tillahoffmann

@tillahoffmann

Dockerfile:

FROM python:3.8-alpine
WORKDIR /app
RUN apk add --no-cache \
    gcc \
    git \
    libpq \
    make \
    musl-dev \
    postgresql-dev \
    && pip install --no-cache-dir \
    git+https://github.com/tuvshuud/testcontainers-python.git \
    && apk del \
    gcc \
    git \
    make \
    musl-dev \
    postgresql-dev
COPY . ./

tests/test.py:

from testcontainers.postgres import PostgresContainer

postgres_container = PostgresContainer()
postgres = postgres_container.start()
# Gets stuck here
CONN_URL = postgres.get_connection_url()

Docker build:

docker build -t localhost/python:3.8 .

Docker run:

docker run \
    -it \
    --rm \
    -v /var/run/docker.sock:/var/run/docker.sock \
    localhost/python:3.8 \
    python -m unittest discover tests

@tuvshuud , sorry PR #96 didn't fix it.

If I install testcontainers[postgresql]==2.6 then it works.

@tillahoffmann

As @tillahoffmann mentioned here, you have to install required additional dependencies explicitly.

So your Dockerfile should have SQLAlchemy, psycopg2 packages installed.

FROM python:3.8-alpine
WORKDIR /app
RUN apk add --no-cache \
    gcc \
    git \
    libpq \
    make \
    musl-dev \
    postgresql-dev \
    && pip install --no-cache-dir \
    git+https://github.com/tuvshuud/testcontainers-python.git \
    SQLAlchemy \
    psycopg2 \
    && apk del \
    gcc \
    git \
    make \
    musl-dev \
    postgresql-dev
COPY . ./

I've checked and #96 fixed it. @karmux

@tillahoffmann Do you think we need more verbosity in _connect() , so that others can troubleshoot easily? Because of missing packages, wrong container ip address or port that is being used, testcontainer process may hang until timeout.

There may be an opportunity to update the exceptions we catch in wait_container_is_ready to only include TimeoutErrors, for example. In other words, any exception that is not a TimeoutError, such as an ImportError, would just be raised as normal. Although we'd have to make sure we don't break anything by accident. What do you think?

testcontainers-python/testcontainers/core/waiting_utils.py

Lines 40 to 45 in 87fb2c1

    
           for _ in range(0, config.MAX_TRIES): 
        
               try: 
        
                   return wrapped(*args, **kwargs) 
        
               except Exception as e: 
        
                   time.sleep(config.SLEEP_TIME) 
        
                   exception = e

Still waiting till it gets timeout with 3.0.2.

Pulling image postgres:latest
⠏
Container started:  2010c8ddfd
Waiting to be ready...
⠏E
======================================================================
ERROR: test (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/app/tests/test.py", line 13, in <module>
    postgres = postgres_container.start()
  File "/usr/local/lib/python3.8/site-packages/testcontainers/core/generic.py", line 42, in start
    self._connect()
  File "/usr/local/lib/python3.8/site-packages/testcontainers/core/waiting_utils.py", line 46, in wrapper
    raise TimeoutException(
testcontainers.core.exceptions.TimeoutException: Wait time exceeded 120 sec.
                    Method _connect, args () , kwargs {}.
                     Exception (psycopg2.OperationalError) could not connect to server: Connection refused
        Is the server running on host "localhost" (127.0.0.1) and accepting
        TCP/IP connections on port 32788?
could not connect to server: Address not available
        Is the server running on host "localhost" (::1) and accepting
        TCP/IP connections on port 32788?

(Background on this error at: http://sqlalche.me/e/e3q8)


----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (errors=1)

$ docker run -it --rm localhost/python:3.8 pip freeze
asgiref==3.2.7
blindspin==2.0.1
certifi==2020.4.5.1
chardet==3.0.4
colorama==0.4.3
coverage==5.1
crayons==0.3.0
deprecation==2.1.0
docker==4.2.1
flake8==3.8.2
idna==2.9
mccabe==0.6.1
mypy==0.780
mypy-extensions==0.4.3
packaging==20.4
psycopg2-binary==2.8.5
pycodestyle==2.6.0
pyflakes==2.2.0
pyparsing==2.4.7
python-dateutil==2.8.1
requests==2.23.0
six==1.15.0
SQLAlchemy==1.3.17
sqlalchemy-stubs==0.3
testcontainers==3.0.2
typed-ast==1.4.1
typing-extensions==3.7.4.2
urllib3==1.25.9
websocket-client==0.57.0
wrapt==1.12.1

@karmux

@karmux #96 merged into master but hasn’t released into patch version yet. Meanwhile you may try with pip install git+https://github.com/testcontainers/testcontainers-python.git

@tuvshuud

Works now! Thank you @tuvshuud and @tillahoffmann!

@tuvshuud

3.0.3 is now on pypi so you should be able to revert to pip install testcontainers without the need to install from master. Thanks for the fix, @tuvshuud.

@tillahoffmann

@tillahoffmann Mine is still hanging. I updated testcontainer to 3.0.3 from above link. Not sure what am i doing wrong.
rootdir: /Users/ypatel/postgres
collecting ... testing
collected 1 item

test_db_containers.py
Pulling image localhost/opensource/postgres/postgresql10:10.12
⠼
Container started: 75a7bd9cd9
Waiting to be ready...
F

============================================================================== FAILURES ==============================================================================
_____________________________________________________________________ test_docker_run_postgress ______________________________________________________________________

def test_docker_run_postgress():
    postgres_container = PostgresContainer("localhost/opensource/postgres/postgresql10:10.12")

  with postgres_container as postgres:

test_db_containers.py:20:

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/testcontainers/core/container.py:64: in enter
return self.start()
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/testcontainers/core/generic.py:42: in start
self._connect()

wrapped = <bound method DbContainer._connect of <testcontainers.postgres.PostgresContainer object at 0x7feb9af0aeb0>>
instance = <testcontainers.postgres.PostgresContainer object at 0x7feb9af0aeb0>, args = (), kwargs = {}

@wrapt.decorator
def wrapper(wrapped, instance, args, kwargs):
    exception = None
    print(crayons.yellow("Waiting to be ready..."))
    with blindspin.spinner():
        for _ in range(0, config.MAX_TRIES):
            try:
                return wrapped(*args, **kwargs)
            except Exception as e:
                time.sleep(config.SLEEP_TIME)
                exception = e

      raise TimeoutException(

            """Wait time exceeded {0} sec.
                Method {1}, args {2} , kwargs {3}.
                 Exception {4}""".format(config.MAX_TRIES,
                                         wrapped.__name__,
                                         args, kwargs, exception))

E testcontainers.core.exceptions.TimeoutException: Wait time exceeded 120 sec.
E Method _connect, args () , kwargs {}.
E Exception (psycopg2.OperationalError) server closed the connection unexpectedly
E This probably means the server terminated abnormally
E before or while processing the request.
E
E (Background on this error at: http://sqlalche.me/e/e3q8)

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/testcontainers/core/waiting_utils.py:46: TimeoutException
====================================================================== short test summary info =======================================================================
FAILED test_db_containers.py::test_docker_run_postgress - testcontainers.core.exceptions.TimeoutException: Wait time exceeded 120 sec.

In my case I am running MySQL Test container within docker, and when I do MysqlContainer.start(), it tries by default to connect to docker_host ip, which is causing the issue.
Do I have to add some flag in docker run command?

@aashish-agarwal-de

@aashish-agarwal-de you can use TC_HOST environment variable to set correct host address if host detection in the library is not working correctly.

It is detecting the gateway host, byt sql alchemy is not able to connect to that ip.
It is only able to connect to host.docker.internal

Is there a way to pass the hostname to MySQL test container, so that it does try to connect to gatwway ip in docker?

In CI this was resolved my passing --network=host in docker run command, but I am still facing issues with running it in my local system. I am working on mac

I am also working on a Mac and I am facing the same issue

@tharwan

@tharwan @aashish-agarwal-de I was able reproduce this issue and made changes. Can you confirm if this works for you?

pip install git+https://github.com/tuvshuud/testcontainers-python.git@improve-host-detection

There is a bug that TC_HOST environment variable is not respected when set in docker in docker environment.

@tuvshuud

@tuvshuud I spent several hours trying to figure this issue out and once I pulled in your improve-host-detection branch it all started working as expected. I would love to see this fix merged. Any thoughts what we can do to help make that happen?

In my case I had a pytest fixture that works perfectly fine when run locally. However, when I tried to run it via docker-compose I see the db container come up, but SQLAlchemy fromthe original container is never able to reach it:
Exception (psycopg2.OperationalError) connection to server at "172.17.0.1", port 55582 failed: Connection refused

I am happy to know there is a solution. Would be great if this solution were merged.

@willgme

@willgme happy to hear that it resolves your issue. To be merged, we need to refactor some codes to pass the all test cases and test on windows and other DinD environments. I just don't have time these days

Navigation Menu

Repository navigation

testcontainers wait indefinitely without effects when running in Gitlab-CI #43

1 remaining item

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	for _ in range(0, config.MAX_TRIES):
	try:
	return wrapped(args, *kwargs)
	except Exception as e:
	time.sleep(config.SLEEP_TIME)
	exception = e

testcontainers wait indefinitely without effects when running in Gitlab-CI #43

Description

Activity

victornoel commented on Nov 28, 2019

GregHilston commented on Jan 9, 2020

GregHilston commented on Jan 9, 2020

ismailsimsek commented on Feb 10, 2020

ismailsimsek commented on Feb 21, 2020

tillahoffmann commented on Apr 28, 2020

victornoel commented on Apr 28, 2020

karmux commented on May 21, 2020

tuvshuud commented on May 23, 2020

karmux commented on May 28, 2020

tillahoffmann commented on May 29, 2020

karmux commented on May 29, 2020

tuvshuud commented on Jun 2, 2020

tillahoffmann commented on Jun 3, 2020

1 remaining item

karmux commented on Jun 4, 2020

tuvshuud commented on Jun 5, 2020

karmux commented on Jun 5, 2020

tillahoffmann commented on Jun 8, 2020

ypatel26 commented on Jun 8, 2020

aashish-agarwal-de commented on Jun 5, 2021

tuvshuud commented on Jun 5, 2021

aashish-agarwal-de commented on Jun 5, 2021

aashish-agarwal-de commented on Jun 5, 2021

aashish-agarwal-de commented on Jun 5, 2021

tharwan commented on Oct 13, 2021

tuvshuud commented on Oct 24, 2021

tuvshuud commented on Oct 24, 2021

willgme commented on Feb 10, 2022

tuvshuud commented on Feb 10, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions