Skip to content

Commit aefabe5

Browse files
AkihiroSudaAkihiroSuda
committed
Dockerfile.test: add "cri-in-userns" (aka rootless) test stage
The `cri-in-userns` stage is for testing "CRI-in-UserNS", which should be used in conjunction with "Kubelet-in-UserNS": https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2033-kubelet-in-userns-aka-rootless This feature is mostly expected to be used for `kind` and `minikube`. Requires Rootless Docker/Podman/nerdctl with cgroup v2 delegation: https://rootlesscontaine.rs/getting-started/common/cgroup2/ (Rootless Docker/Podman/nerdctl prepares the UserNS, so we do not need to create UserNS by ourselves) Usage: ``` podman build --target cri-in-userns -t cri-in-userns -f contrib/Dockerfile.test . podman run -it --rm --privileged cri-in-userns ``` The stage is tested on CI with Rootless Podman on Fedora 34 on Vagrant. Signed-off-by: Akihiro Suda <[email protected]>
1 parent 0573e22 commit aefabe5

5 files changed

Lines changed: 139 additions & 1 deletion

File tree

‎.github/workflows/ci.yml‎

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -526,7 +526,7 @@ jobs:
526526
*-junit.xml
527527
528528
cgroup2:
529-
name: CGroupsV2 and SELinux Integration
529+
name: CGroupsV2 - SELinux enforced
530530
# nested virtualization is only available on macOS hosts
531531
runs-on: macos-10.15
532532
timeout-minutes: 45
@@ -580,3 +580,31 @@ jobs:
580580
path: |
581581
${{github.workspace}}/*-junit.xml
582582
${{github.workspace}}/critestreport/*
583+
584+
cgroup2-misc:
585+
name: CGroupsV2 - rootless CRI test
586+
# nested virtualization is only available on macOS hosts
587+
runs-on: macos-10.15
588+
timeout-minutes: 45
589+
needs: [project, linters, protos, man]
590+
steps:
591+
- uses: actions/checkout@v2
592+
593+
- name: "Cache ~/.vagrant.d/boxes"
594+
uses: actions/cache@v2
595+
with:
596+
path: ~/.vagrant.d/boxes
597+
key: vagrant-${{ hashFiles('Vagrantfile*') }}
598+
599+
- name: Vagrant start
600+
run: |
601+
# Retry if it fails (download.fedoraproject.org returns 404 sometimes)
602+
vagrant up || vagrant up
603+
604+
# slow, so separated from the regular cgroup2 task
605+
- name: CRI-in-UserNS test with Rootless Podman
606+
run: |
607+
vagrant up --provision-with=install-rootless-podman
608+
# Execute rootless podman to create the UserNS env
609+
vagrant ssh -- podman build --target cri-in-userns -t cri-in-userns -f /vagrant/contrib/Dockerfile.test /vagrant
610+
vagrant ssh -- podman run --rm --privileged cri-in-userns

‎Vagrantfile‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -257,4 +257,31 @@ EOF
257257
SHELL
258258
end
259259

260+
# Rootless Podman is used for testing CRI-in-UserNS
261+
# (We could use rootless nerdctl, but we are using Podman here because it is available in dnf)
262+
config.vm.provision "install-rootless-podman", type: "shell", run: "never" do |sh|
263+
sh.upload_path = "/tmp/vagrant-install-rootless-podman"
264+
sh.inline = <<~SHELL
265+
#!/usr/bin/env bash
266+
set -eux -o pipefail
267+
# Delegate cgroup v2 controllers to rootless
268+
mkdir -p /etc/systemd/system/[email protected]
269+
cat > /etc/systemd/system/[email protected]/delegate.conf << EOF
270+
[Service]
271+
Delegate=yes
272+
EOF
273+
systemctl daemon-reload
274+
# Install Podman
275+
dnf install -y podman
276+
# Configure Podman to resolve `golang` to `docker.io/library/golang`
277+
mkdir -p /etc/containers
278+
cat > /etc/containers/registries.conf <<EOF
279+
[registries.search]
280+
registries = ['docker.io']
281+
EOF
282+
# Disable SELinux to allow overlayfs
283+
setenforce 0
284+
SHELL
285+
end
286+
260287
end

‎contrib/Dockerfile.test‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,3 +58,25 @@ COPY --from=proto3 /usr/local/include/google /usr/local/include/google
5858
COPY --from=runc /usr/local/sbin/runc /usr/local/go/bin/runc
5959

6060
COPY . .
61+
62+
# cri-in-userns stage is for testing "CRI-in-UserNS", which should be used in conjunction with
63+
# "Kubelet-in-UserNS": https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2033-kubelet-in-userns-aka-rootless
64+
# This feature is mostly expected to be used for `kind` and `minikube`.
65+
#
66+
# Requires Rootless Docker/Podman/nerdctl with cgroup v2 delegation: https://rootlesscontaine.rs/getting-started/common/cgroup2/
67+
# (Rootless Docker/Podman/nerdctl prepares the UserNS, so we do not need to create UserNS by ourselves)
68+
FROM dev AS cri-in-userns
69+
RUN make BUILDTAGS="no_btrfs no_devmapper" binaries install
70+
RUN apt-get update && apt-get install -y iptables
71+
RUN ./script/setup/install-cni
72+
RUN ./script/setup/install-critools
73+
COPY contrib/Dockerfile.test.d/cri-in-userns/etc_containerd_config.toml /etc/containerd/config.toml
74+
COPY contrib/Dockerfile.test.d/cri-in-userns/docker-entrypoint.sh /docker-entrypoint.sh
75+
VOLUME /var/lib/containerd
76+
ENTRYPOINT ["/docker-entrypoint.sh"]
77+
# Skip "runtime should support unsafe sysctls": `container init caused: write sysctl key fs.mqueue.msg_max: open /proc/sys/fs/mqueue/msg_max: permission denied`
78+
# Skip "runtime should support safe sysctls": `container init caused: write sysctl key kernel.shm_rmid_forced: open /proc/sys/kernel/shm_rmid_forced: permission denied`
79+
# Skip "should allow privilege escalation when (NoNewPrivis is) false": expected log "Effective uid: 0\n" (stream="stdout") not found in logs [{timestamp:{wall:974487519 ext:63761339984 loc:<nil>} stream:stdout log:Effective uid: 1000) }]
80+
CMD ["critest", "--ginkgo.skip=should support unsafe sysctls|should support safe sysctls|should allow privilege escalation when false"]
81+
82+
FROM dev AS default

‎contrib/Dockerfile.test.d/cri-in-userns/docker-entrypoint.sh‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
#!/bin/bash
2+
3+
# Copyright The containerd Authors.
4+
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
set -eu -o pipefail
18+
19+
# Check 4294967295 to detect UserNS (https://github.com/opencontainers/runc/blob/v1.0.0/libcontainer/userns/userns_linux.go#L29-L32)
20+
if grep -Eq "0[[:space:]]+0[[:space:]]+4294967295" /proc/self/uid_map; then
21+
echo >&2 "ERROR: Needs to be executed in UserNS (i.e., rootless Docker/Podman/nerdctl)"
22+
exit 1
23+
fi
24+
25+
if [ ! -f "/sys/fs/cgroup/cgroup.controllers" ]; then
26+
echo >&2 "ERROR: Needs cgroup v2"
27+
exit 1
28+
fi
29+
30+
for f in cpu memory pids; do
31+
if ! grep -qw "$f" "/sys/fs/cgroup/cgroup.controllers"; then
32+
echo >&2 "ERROR: Needs cgroup v2 controller ${f} to be delegated"
33+
exit 1
34+
fi
35+
done
36+
37+
echo >&2 "Enabling cgroup v2 nesting"
38+
# https://github.com/moby/moby/blob/v20.10.7/hack/dind#L28-L38
39+
mkdir -p /sys/fs/cgroup/init
40+
xargs -rn1 < /sys/fs/cgroup/cgroup.procs > /sys/fs/cgroup/init/cgroup.procs || :
41+
sed -e 's/ / +/g' -e 's/^/+/' < /sys/fs/cgroup/cgroup.controllers \
42+
> /sys/fs/cgroup/cgroup.subtree_control
43+
44+
set -x
45+
echo >&2 "Running containerd in background"
46+
containerd &
47+
48+
echo >&2 "Waiting for containerd"
49+
until ctr plugins list; do sleep 3; done
50+
51+
exec "$@"

‎contrib/Dockerfile.test.d/cri-in-userns/etc_containerd_config.toml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
version = 2
2+
3+
[plugins]
4+
[plugins."io.containerd.grpc.v1.cri"]
5+
disable_apparmor = true
6+
restrict_oom_score_adj = true
7+
disable_hugetlb_controller = true
8+
[plugins."io.containerd.grpc.v1.cri".containerd]
9+
# Rootless overlayfs requires kernel >= 5.11 && !selinux
10+
snapshotter = "overlayfs"

0 commit comments

Comments
 (0)