I've run into and tried to fix this exact same race several times, and it's complex because if this code fails, I think Docker-in-Docker will fail too, so it has to succeed and simply ignoring failure isn't correct. 😅

Every time I run into this, I fix it by checking whether dockerd is ready yet from a separate container. I bet you could emulate something similar by doing something like unshare --pid dind dockerd instead so it gets started in a totally new PID namespace. 🤔

Just to give you a bit of background I am running minikube with dind in a gitlab pipeline.

I think Docker-in-Docker will fail too, so it has to succeed and simply ignoring failure isn't correct.

Since I started ignoring sed failures by patching the dind script, docker in docker spins up successfully and I've never encountered the problem again and my pipelines run just fine.

Every time I run into this, I fix it by checking whether dockerd is ready yet from a separate container.

I can do this, but I don't understand how this would fix my problem. As dind terminates when sed returns with a non zero exit code. Which means dockerd never actually starts / becomes ready. If this is a race and sed needs to run successfully, should dind then maybe retry the sed line until it succeeds?

setup_cgroup_controllers(){
    sed -e 's/ / +/g' -e 's/^/+/' < /sys/fs/cgroup/cgroup.controllers \
        > /sys/fs/cgroup/cgroup.subtree_control \
        &>/dev/null
}
# busy wait for setup_cgroup_controllers to complete
while ! setup_cgroup_controllers; do sleep 0; done

What do you think @tianon ?

Hmm, that's not a bad idea, although it would need to include the xargs line above it too, I think (since that's the line that moves all the processes such that the following line can succeed). 🤔

I don't think we need a function, and something like this could probably work (although needs some testing):

	mkdir -p /sys/fs/cgroup/init
	# this happens in a loop because things like "docker exec" on our dind
	# container will create new processes, which creates a race between our
	# moving everything to "init" and enabling subtree_control
	while ! {
		# move the processes from the root group to the /init group,
		# otherwise writing subtree_control fails with EBUSY.
		# An error during moving non-existent process (i.e., "cat") is ignored.
		xargs -rn1 < /sys/fs/cgroup/cgroup.procs > /sys/fs/cgroup/init/cgroup.procs || :
		# enable controllers
		sed -e 's/ / +/g' -e 's/^/+/' < /sys/fs/cgroup/cgroup.controllers \
			> /sys/fs/cgroup/cgroup.subtree_control
	}; do true; done

(I don't think we need to worry too much about the error messages here -- this basically just puts the existing code into a tight little loop, as you've proposed.)

I forgot what I'd done previously to reproduce this reliably (I recall having a ~100% failure rate in the past), but with the following I managed to get something that does reproduce the race/failure some percentage of the time, and after applying the above change I was no longer able to reproduce:

#!/usr/bin/env bash
set -Eeuo pipefail -x

docker run -di --rm --name dind --mount "type=bind,src=$PWD/hack/dind,dst=/dind,ro" --entrypoint /dind --privileged docker:dind dockerd --log-level error
trap 'docker rm -vf dind' EXIT
docker logs -f dind &

while state="$(docker container inspect --format '{{ .State.Status }}' dind)" && [ "$state" = 'running' ] && ! docker exec dind docker version; do true; done

@tianon : Awesome !

A few questions:

• Are you aware then set -e is not inherited into the { ... }? This will print Should never get here:

#!/usr/bin/env bash
set -e
while ! {
    false
    true
}; do sleep 0; done
echo "Should never get here"

• Did you keep the errorignore (... || :) in the xargs line on purpose?
• If sed fails the xargs line will run again: Does this not cause side-effects?

• Are you aware then set -e is not inherited into the { ... }? This will print Should never get here:

Yes, that's not a big deal in our case because the xargs line includes || : (my original version used && isntead of { ...; } for this very reason, but I realized the delta/diff would be smaller and easier to maintain if it was "just" indentation changes because it doesn't matter for us)

• Did you keep the errorignore (... || :) in the xargs line on purpose?
• If sed fails the xargs line will run again: Does this not cause side-effects?

Yes, in fact we need the xargs line to run again if the sed fails -- the xargs line is the one that "moves" processes from the root cgroup into the (newly created) init cgroup, and new processes showing up is why the sed fails, so while it doesn't matter as much for your specific case, if we want to solve for the general problem we have to assume the new processes that have shown up are not necessarily short-lived (and thus need to also be moved before we try again).

ok, @tianon . Glad you considered all these things. 🚀
Then I'd say this is looking good. What does now need to happen for this to be merged?

@tianon : changes were added by @JSchltggr 🚀 I guess this can be merged now?

@thaJeztah we got the squash 🎉 . Thank you @JSchltggr . Can this be merged now?