PostgresContainer start stuck at waiting to be ready in Gitlab CI using docker:dind

I am trying to setup test environment in Gitlab CI/CD. But starting PostgresContainer stuck forever.

Version:

Python3.8
SQLAlchemy==1.3.12
testcontainers==2.6.0

Samples from my .gitlab-ci.yml

variables:
  DOCKER_HOST: tcp://docker:2375/
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""

test:
    stage: test
    image: python:3.8-slim
    services:
        - docker:19.03.8-dind
    script:
        - pytest tests -s -v --log-cli-level=debug

Logs from gitlab-runner

�$ pytest tests -s -v --log-cli-level=debug�
============================= test session starts ==============================
platform linux -- Python 3.8.1, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- /builds/platform/account/venv/bin/python
cachedir: .pytest_cache
rootdir: /builds/platform/account
plugins: flask-1.0.0
collecting ... collected 2 items

tests/functional/test_login.py::test_simple_login 
-------------------------------- live log setup --------------------------------
DEBUG    docker.utils.config:config.py:21 Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
DEBUG    docker.utils.config:config.py:28 No config file found
DEBUG    docker.utils.config:config.py:21 Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
DEBUG    docker.utils.config:config.py:28 No config file found

-----Start postgres db container-----


Pulling image postgres:11.4-alpine
DEBUG    urllib3.connectionpool:connectionpool.py:221 Starting new HTTP connection (1): docker:2375
DEBUG    urllib3.connectionpool:connectionpool.py:428 http://docker:2375 "POST /v1.35/containers/create HTTP/1.1" 404 50
DEBUG    docker.auth:auth.py:41 Looking for auth config
DEBUG    docker.auth:auth.py:43 No auth config in memory - loading from filesystem
DEBUG    docker.utils.config:config.py:21 Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
DEBUG    docker.utils.config:config.py:28 No config file found
DEBUG    docker.auth:auth.py:242 Looking for auth entry for 'docker.io'
DEBUG    docker.auth:auth.py:253 No entry found
DEBUG    docker.auth:auth.py:58 No auth config found
DEBUG    urllib3.connectionpool:connectionpool.py:428 http://docker:2375 "POST /v1.35/images/create?tag=11.4-alpine&fromImage=postgres HTTP/1.1" 200 None
DEBUG    urllib3.connectionpool:connectionpool.py:428 http://docker:2375 "GET /v1.35/images/postgres:11.4-alpine/json HTTP/1.1" 200 None
DEBUG    urllib3.connectionpool:connectionpool.py:428 http://docker:2375 "POST /v1.35/containers/create HTTP/1.1" 201 88
DEBUG    urllib3.connectionpool:connectionpool.py:428 http://docker:2375 "GET /v1.35/containers/f5813a1784187260cb46ee3f71edde72d189f00cc35308666a4df624a4bc680e/json HTTP/1.1" 200 None
DEBUG    urllib3.connectionpool:connectionpool.py:428 http://docker:2375 "POST /v1.35/containers/f5813a1784187260cb46ee3f71edde72d189f00cc35308666a4df624a4bc680e/start HTTP/1.1" 204 0

Container started:  f5813a1784
Waiting to be ready...
DEBUG    urllib3.connectionpool:connectionpool.py:428 http://docker:2375 "GET /v1.35/containers/json?limit=-1&all=0&size=0&trunc_cmd=0&filters=%7B%22id%22%3A+%5B%22f5813a1784187260cb46ee3f71edde72d189f00cc35308666a4df624a4bc680e%22%5D%7D HTTP/1.1" 200 1101
DEBUG    sqlalchemy.pool.impl.QueuePool:base.py:643 Error on connect(): could not connect to server: Connection timed out
	Is the server running on host "172.18.0.2" and accepting
	TCP/IP connections on port 5432?

DEBUG    urllib3.connectionpool:connectionpool.py:428 http://docker:2375 "GET /v1.35/containers/json?limit=-1&all=0&size=0&trunc_cmd=0&filters=%7B%22id%22%3A+%5B%22f5813a1784187260cb46ee3f71edde72d189f00cc35308666a4df624a4bc680e%22%5D%7D HTTP/1.1" 200 1092
DEBUG    sqlalchemy.pool.impl.QueuePool:base.py:643 Error on connect(): could not connect to server: Connection timed out
	Is the server running on host "172.18.0.2" and accepting
	TCP/IP connections on port 5432?

Informations from docker:dind

[root@gitlab-runner ~]# docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                     NAMES
f5813a178418        postgres:11.4-alpine   "docker-entrypoint.s…"   9 minutes ago       Up 9 minutes        0.0.0.0:32768->5432/tcp   musing_khorana

[root@gitlab-runner-build-1 ~]# docker inspect f5813a178418
[
    {
        "Id": "f5813a1784187260cb46ee3f71edde72d189f00cc35308666a4df624a4bc680e",
        "Created": "2020-04-15T05:22:14.043451232Z",
        "Path": "docker-entrypoint.sh",
        "Args": [
            "postgres"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 271,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2020-04-15T05:22:14.631050518Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:5239fade3a90b73a10592a252289d6d916d050f39dafca0650ae14e878c23b0a",
        "ResolvConfPath": "/var/lib/docker/containers/f5813a1784187260cb46ee3f71edde72d189f00cc35308666a4df624a4bc680e/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/f5813a1784187260cb46ee3f71edde72d189f00cc35308666a4df624a4bc680e/hostname",
        "HostsPath": "/var/lib/docker/containers/f5813a1784187260cb46ee3f71edde72d189f00cc35308666a4df624a4bc680e/hosts",
        "LogPath": "/var/lib/docker/containers/f5813a1784187260cb46ee3f71edde72d189f00cc35308666a4df624a4bc680e/f5813a1784187260cb46ee3f71edde72d189f00cc35308666a4df624a4bc680e-json.log",
        "Name": "/musing_khorana",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {
                "5432/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": ""
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Capabilities": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/8c0f74be6b6273dfa92f746332d4c7b5a02183efa5bde3c40abf25f3329dbc59-init/diff:/var/lib/docker/overlay2/29efd662a010319d509a75483ff5650fa0ae46b41c98e374e4b3cc5e2c3e0167/diff:/var/lib/docker/overlay2/8723a0ffff99f7e5af3d94f9ebaaf155b9883836203690a908c64819718f8f66/diff:/var/lib/docker/overlay2/603fd62f94e002cdf27665e27b2ce6d0016da652eaf13838fcd66c0027c1171f/diff:/var/lib/docker/overlay2/fa06a053297fdcca4423d783ed6a8586bea4dd5aa9b9cb2631ad773f1cb1218d/diff:/var/lib/docker/overlay2/c72b4d9e0c149ec0062d2f8da3b23ff758ab58d0a55410e2c6697efe196ef9d8/diff:/var/lib/docker/overlay2/2a7c676edf9d77134aa92aa35b567935486e1efa37b13019f6ad6ce937824828/diff:/var/lib/docker/overlay2/bf218232ea043ee53d881f802b971e48327fd574505ed13b0be86cebbbd7f7a7/diff:/var/lib/docker/overlay2/fbd9de45ae6e8b7efc153206057e355beaf88899a462723c775db0b520fe882e/diff:/var/lib/docker/overlay2/903dc6646e74d53678e8d395df9bb080cceeb23bb945e628006d0650404dbdce/diff",
                "MergedDir": "/var/lib/docker/overlay2/8c0f74be6b6273dfa92f746332d4c7b5a02183efa5bde3c40abf25f3329dbc59/merged",
                "UpperDir": "/var/lib/docker/overlay2/8c0f74be6b6273dfa92f746332d4c7b5a02183efa5bde3c40abf25f3329dbc59/diff",
                "WorkDir": "/var/lib/docker/overlay2/8c0f74be6b6273dfa92f746332d4c7b5a02183efa5bde3c40abf25f3329dbc59/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "60cdc4a7b03157ab2f14b66e4620480f24b2562912912ac9ed5305b439eb13a5",
                "Source": "/var/lib/docker/volumes/60cdc4a7b03157ab2f14b66e4620480f24b2562912912ac9ed5305b439eb13a5/_data",
                "Destination": "/var/lib/postgresql/data",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "f5813a178418",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "5432/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "POSTGRES_USER=test",
                "POSTGRES_PASSWORD=test",
                "POSTGRES_DB=test",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "LANG=en_US.utf8",
                "PG_MAJOR=11",
                "PG_VERSION=11.4",
                "PG_SHA256=02802ddffd1590805beddd1e464dd28a46a41a5f1e1df04bab4f46663195cc8b",
                "PGDATA=/var/lib/postgresql/data"
            ],
            "Cmd": [
                "postgres"
            ],
            "Image": "postgres:11.4-alpine",
            "Volumes": {
                "/var/lib/postgresql/data": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "a1dbb4c5bd2422ce0cffa574e1b44408c2aa7fbdef720d4ebd18bce550d9fe50",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "5432/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "32768"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/a1dbb4c5bd24",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "6e8581cc476a339fb31a62df1a6acd6ae98e2bee101c713caffe1bb02a6318b8",
            "Gateway": "172.18.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.18.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:12:00:02",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "8fe294d72eef7fd7fc47f6bad666ad0c70b395c3314b3a73545eac718c232b2e",
                    "EndpointID": "6e8581cc476a339fb31a62df1a6acd6ae98e2bee101c713caffe1bb02a6318b8",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:12:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

It seems to me that after starting the container, testcontainer tries to connect with container_ip:5432 rather than randomly generated docker_dind_host:32768. Without docker:dind, it works perfectly.

Should get_container_host_ip() return docker:dind host ip? Also get_exposed_port() return mapped host port on docker:dind?

Thanks for the report. I'll close this for now as a duplicate of #43, but let me know if this is a distinct issue.

Thanks for the report. I'll close this for now as a duplicate of #43, but let me know if this is a distinct issue.

It was a duplicate. Sorry about that. I had to implement my own get_container_host_ip() and get_exposed_port() to solve this.

Would you be happy to share your implementation? Unfortunately, we can't test PRs from forks on GitLab CI because of this issue, but I can try and integrate your changes within the main repository.

I'm on 3.4.2 and I'm still seeing the GitLab build hang (forever). I've followed the instructions at https://www.testcontainers.org/supported_docker_environment/continuous_integration/gitlab_ci/

Is this actually fixed?

-- RUNNING INTEGRATION TESTS --
============================= test session starts ==============================
platform linux -- Python 3.9.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /builds/albedospace/archive-service
plugins: cov-2.12.1
collected 3 items
test/int/test_app.py 
Pulling image postgis/postgis:13-master
Container started: dfedf1f0f0
Waiting to be ready...

Navigation Menu

Repository navigation

PostgresContainer start stuck at waiting to be ready in Gitlab CI using docker:dind #81

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

PostgresContainer start stuck at waiting to be ready in Gitlab CI using docker:dind #81

Description

Activity

tillahoffmann commented on Apr 15, 2020

tuvshuud commented on Apr 16, 2020

tillahoffmann commented on Apr 16, 2020

mkochco commented on Sep 7, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions