We've received several reports that jobs running in GitLab CI and using the Docker-in-Docker approach to build images started to hang during execution. Some examples can be found at:
Please notice, that the problem is not on pulling the gitlab/gitlab-runner-helper image, which is done properly and is part of a post build step after the hanging job was terminated. The failure happens when a Docker image is being build, using the Docker-in-Docker service from docker:dind image.
Reviewing all of the cases we've found that they all are caused by using docker:dind or doker:19.03-dind, so the floating versions pointing to docker:19.03.6-dind which was released several hours ago. The time of the image release aligns with all of the reports that we've received.
The only change in the image, since the last docker:19.03.5-dind push (which works!) was done, is the automatic update from Docker 19.03.5 to Docker 19.03.6 (3dc42d4). Whatever is causing the problems, it's in Docker itself.
Leaving this as a note and a workaround description for all of you who may look here to find out problems with GitLab CI and Docker-in-Docker, which was happening in the past :)
So, if you think you're affected by this issue, set the used image explicitly to docker:19.03.5-dind.
Also... maybe it's worth to revert the release of docker:19.03.6-dind until it's fixed?
We've received several reports that jobs running in GitLab CI and using the Docker-in-Docker approach to build images started to hang during execution. Some examples can be found at:
Please notice, that the problem is not on pulling the
gitlab/gitlab-runner-helperimage, which is done properly and is part of apost buildstep after the hanging job was terminated. The failure happens when a Docker image is being build, using the Docker-in-Docker service fromdocker:dindimage.Reviewing all of the cases we've found that they all are caused by using
docker:dindordoker:19.03-dind, so the floating versions pointing todocker:19.03.6-dindwhich was released several hours ago. The time of the image release aligns with all of the reports that we've received.The only change in the image, since the last
docker:19.03.5-dindpush (which works!) was done, is the automatic update from Docker19.03.5to Docker19.03.6(3dc42d4). Whatever is causing the problems, it's in Docker itself.Leaving this as a note and a workaround description for all of you who may look here to find out problems with GitLab CI and Docker-in-Docker, which was happening in the past :)
So, if you think you're affected by this issue, set the used image explicitly to
docker:19.03.5-dind.Also... maybe it's worth to revert the release of
docker:19.03.6-dinduntil it's fixed?