Draft: LXC support for self-hosted runners by earl-warren · Pull Request #1682 · nektos/act

earl-warren · 2023-03-16T11:37:44Z

Description

The LXC support for self-hosted runners is used to run tests that do not fit the constraints imposed by the docker backend such as having a systemd capable environment.

It creates a container from scratch on every run. If the tests accidentally damage essential services such as the ssh server, it will not have any impact on the host running the LXC container. If the same accident happens without the isolation provided by the LXC container, the host itself will be damaged. The LXC support provides a robust isolation for each job in the workflows, which the self-hosted platform does not.

Implementation details

It is roughly the equivalent of doing the following:

Creating a LXC container (with lxc-create)
Running the commands in the container (with lxc-attach)
Destroying the LXC container (with lxc-destroy)

This is inherently insecure, in the same way the self-hosted platform is. Hardening LXC containers is possible but it makes them no more useful than docker containers.

FAQ

Why not run LXC inside a Docker container? Because it does not work. Although it could probably be hacked, that would not be more than that.
Why not run the LXC logic from the sef-hosted platform? Because it would require significant work and LXC knowledge from each ACT user. Instead they only have to care for their own tests, not to setup and teardown well isolated LXC containers.
Why not run the LXC containers as unprivileged to improve security? Because it does not work with most templates. Given the restrictions it imposes, the user is better of using Docker.
Why is apparmor disabled? Because it cannot be configured to allow essential operations such as mounting file systems

earl-warren · 2023-03-16T11:39:52Z

This is an early draft go gauge interest and I'm ready to do the needful if there is. It is developed and used in the context of the https://forgejo.org project.

panekj · 2023-03-16T11:40:20Z

pkg/runner/run_context.go

+var startTemplate = template.Must(template.New("start").Parse(`#!/bin/sh -xe
+lxc-create --name="{{.Name}}" --template={{.Template}} -- --release {{.Release}} $packages
+tee -a /var/lib/lxc/{{.Name}}/config <<'EOF'
+security.nesting = true
+lxc.cap.drop =
+lxc.apparmor.profile = unconfined
+#
+# /dev/net (docker won't work without /dev/net/tun)
+#
+lxc.cgroup2.devices.allow = c 10:200 rwm
+lxc.mount.entry = /dev/net dev/net none bind,create=dir 0 0
+#
+# /dev/kvm (libvirt / kvm won't work without /dev/kvm)
+#
+lxc.cgroup2.devices.allow = c 10:232 rwm
+lxc.mount.entry = /dev/kvm dev/kvm none bind,create=file 0 0
+#
+# /dev/loop
+#
+lxc.cgroup2.devices.allow = c 10:237 rwm
+lxc.cgroup2.devices.allow = b 7:* rwm
+lxc.mount.entry = /dev/loop-control dev/loop-control none bind,create=file 0 0
+#
+# /dev/mapper
+#
+lxc.cgroup2.devices.allow = c 10:236 rwm
+lxc.mount.entry = /dev/mapper dev/mapper none bind,create=dir 0 0
+#
+# /dev/fuse
+#
+lxc.cgroup2.devices.allow = b 10:229 rwm
+lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file 0 0
+EOF
+
+mkdir -p /var/lib/lxc/{{.Name}}/rootfs/{{ .Root }}
+mount --bind {{ .Root }} /var/lib/lxc/{{.Name}}/rootfs/{{ .Root }}
+
+mkdir /var/lib/lxc/{{.Name}}/rootfs/tmpdir
+mount --bind {{.TmpDir}} /var/lib/lxc/{{.Name}}/rootfs/tmpdir
+
+lxc-start {{.Name}}
+lxc-wait --name {{.Name}} --state RUNNING
+
+#
+# Wait for the network to come up
+#
+cat > /var/lib/lxc/{{.Name}}/rootfs/tmpdir/networking.sh <<'EOF'
+#!/bin/sh -xe
+for d in $(seq 60); do
+  getent hosts wikipedia.org > /dev/null && break
+  sleep 1
+done
+getent hosts wikipedia.org
+EOF
+chmod +x /var/lib/lxc/{{.Name}}/rootfs/tmpdir/networking.sh
+
+lxc-attach --name {{.Name}} -- /tmpdir/networking.sh
+
+cat > /var/lib/lxc/{{.Name}}/rootfs/tmpdir/node.sh <<'EOF'
+#!/bin/sh -xe
+# https://github.com/nodesource/distributions#debinstall
+apt-get install -y curl git
+curl -fsSL https://deb.nodesource.com/setup_16.x | bash -
+apt-get install -y nodejs
+EOF
+chmod +x /var/lib/lxc/{{.Name}}/rootfs/tmpdir/node.sh
+
+lxc-attach --name {{.Name}} -- /tmpdir/node.sh
+
+`))


This should be separate file an embedded into go

panekj · 2023-03-16T11:40:27Z

pkg/runner/run_context.go

+var stopTemplate = template.Must(template.New("stop").Parse(`#!/bin/sh -x
+lxc-ls -1 --filter="^{{.Name}}" | while read container ; do
+   lxc-stop --kill --name="$container"
+   umount "/var/lib/lxc/$container/rootfs/{{ .Root }}"
+   umount "/var/lib/lxc/$container/rootfs/tmpdir"
+   lxc-destroy --force --name="$container"
+done
+`))


panekj · 2023-03-16T11:40:56Z

pkg/container/host_environment.go

+	command := make([]string, len(commandparam))
+	copy(command, commandparam)
+	if user == "root" {
+		command = append([]string{"/usr/bin/sudo"}, command...)


hardcoded sudo

panekj · 2023-03-16T11:41:02Z

pkg/container/host_environment.go

+		command = append([]string{"/usr/bin/sudo"}, command...)
+	} else {
+		common.Logger(ctx).Debugf("lxc-attach --name %v %v", e.Name, command)
+		command = append([]string{"/usr/bin/sudo", "--preserve-env", "--preserve-env=PATH", "/usr/bin/lxc-attach", "--keep-env", "--name", e.Name, "--"}, command...)


panekj · 2023-03-16T11:47:42Z

This should be separate backend from host

KnisterPeter · 2023-03-20T08:23:57Z

@earl-warren This sounds very interesting. I haven't read through the code but I would like to see it integrated into act. I guess we need to discuss about the structure and interfaces a bit more. It's not always clearly separated currently.

@ChristopherHX might have some ideas and expectations here as well.

earl-warren · 2023-03-23T09:40:23Z

Great to hear 🎉

It could also be an optional feature of the self-hosted platform. If LXC is available and the option is set, wrap it into a LXC container for better isolation. Otherwise not. It would be entirely transparent to the workflows being run.

@ChristopherHX is is something you think to be a valuable addition to the self-hosted platform you implemented? Or would you prefer to go in another direction?

ChristopherHX · 2023-03-23T10:15:13Z

@ChristopherHX is is something you think to be a valuable addition to the self-hosted platform you implemented? Or would you prefer to go in another direction?

I primary made my self-hosted for platforms like freebsd, openbsd, plan9 etc. where docker is not a thing (and GitHub Actions doesn't support), so this should be a different implemention of the ExecutionsEnvironment interface.

I prefer that you create a new struct, which overrides (in golang it is probably a different wording, using interfaces it should work) some HostEnvironment functions. Then instantiate it via a custom identifier from the runcontext, maybe -lxc:<baseimage> instead of -self-hosted ( needs design consent, it took long for my PR to land ).

I don't think we need to add Root() and Name() to the interface. You can override the Close() executor inside the struct with access to custom fields. The start method also have access to the fields, if you create a local var with the specfic type and later assign it to jobcontainer.

earl-warren · 2023-03-23T11:53:20Z

Thanks for the feedback @ChristopherHX

I prefer that you create a new struct, which overrides (in golang it is probably a different wording, using interfaces it should work) some HostEnvironment functions. Then instantiate it via a custom identifier from the runcontext, maybe -lxc: instead of -self-hosted ( needs design consent, it took long for my PR to land ).

I'm happy to go in this direction. The current implementation is more of a proof of concept / hack than anything else and reworking it entirely is what I was expecting anyways.

If anyone think it should go differently please speak up. Otherwise I'll start working as soon as time permits, which could be as soon as in two weeks from now.

mergify · 2023-07-17T19:56:25Z

@earl-warren this pull request is now in conflict 😩

earl-warren · 2023-07-17T20:48:07Z

I'm still motivated, it just takes a little longer than expected.

ChristopherHX · 2023-07-17T21:01:02Z

The HostEnvironment took me ca. 1,5 years (1 year in my fork, 1/2 years in review) to get in here. You can still be faster than me 😅.

github-actions · 2024-01-14T00:11:26Z

PR is stale and will be closed in 14 days unless there is new activity

github-actions · 2024-07-14T00:13:08Z

PR is stale and will be closed in 14 days unless there is new activity

ChristopherHX · 2024-08-29T19:14:51Z

......https://github.com/nektos/act/tree/act-lxc

Maybe I continue here someday, some stuff from here has been ported. (This is based on my macOS tart addon)

However the sudo and script gen stuff doesn't make it straight forward.

Tbh, I need more disk space / do that on a raspberry pi to actually make that run like the macOS tart.

YOU54F · 2024-09-05T10:37:58Z

I was watching this, as it looks really interesting.

I was using lxd for ephemeral gh runners

Leveraging macpine which is a macos tool in go that creates alpine vms with qemu (similar to lima)

https://beringresearch.github.io/macpine/lxd_macpine/

Tbh, I need more disk space / do that on a raspberry pi to actually make that run like the macOS tart.

☝🏾 this! I ended up buying a 4tb nvme and external tb4 dock to satisfy chonki vm's and my desire to create many of them 😅

ChristopherHX · 2024-09-07T16:08:08Z

I consider to not use the lxc-* commands and instead use lxd api.

Found this https://pkg.go.dev/github.com/lxc/lxd/client

Additionally I plan to not use bind mounts for act, instead make use of sftp to copy data

hmm lxd has been forked because of canonical changing the license..

act PR nektos/act#1682 * shell script to start the LXC container * create and destroy a LXC container * run commands with lxc-attach * expose additional devices for docker & libvirt to work * install node 16 & git for checkout to work [FORGEJO] start/stop lxc working directory is /tmp [FORGEJO] use lxc-helpers to create/destroy containers [FORGEJO] do not setup LXC (cherry picked from commit 5b94ff3226848791b93e72d2e0f0ee4bba29a989) Conflicts: pkg/container/host_environment.go Conflicts: pkg/container/host_environment.go [FORGJEO] upgrade to node20

earl-warren added 6 commits March 16, 2023 12:27

add stopHostEnvironment to tear down the LXC container

c7f0532

shell script to start the LXC container

9c95665

create and destroy a LXC container

3ee8456

run commands with lxc-attach

fbad698

expose additional devices for docker & libvirt to work

643a462

install node 16 & git for checkout to work

2ccb312

earl-warren requested a review from a team as a code owner March 16, 2023 11:37

pull-request-size bot added the size/L label Mar 16, 2023

panekj requested changes Mar 16, 2023

View reviewed changes

mergify bot added the conflict PR has conflicts label Jul 17, 2023

github-actions bot added the stale label Jan 14, 2024

catthehacker removed the stale label Jan 15, 2024

github-actions bot added the stale label Jul 14, 2024

github-actions bot closed this Jul 29, 2024

mergify bot removed the conflict PR has conflicts label Jul 29, 2024

Uh oh!

Conversation

earl-warren commented Mar 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Implementation details

FAQ

Uh oh!

earl-warren commented Mar 16, 2023

Uh oh!

panekj Mar 16, 2023

Choose a reason for hiding this comment

Uh oh!

panekj Mar 16, 2023

Choose a reason for hiding this comment

Uh oh!

panekj Mar 16, 2023

Choose a reason for hiding this comment

Uh oh!

panekj Mar 16, 2023

Choose a reason for hiding this comment

Uh oh!

panekj commented Mar 16, 2023

Uh oh!

KnisterPeter commented Mar 20, 2023

Uh oh!

earl-warren commented Mar 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ChristopherHX commented Mar 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

earl-warren commented Mar 23, 2023

Uh oh!

mergify bot commented Jul 17, 2023

Uh oh!

earl-warren commented Jul 17, 2023

Uh oh!

ChristopherHX commented Jul 17, 2023

Uh oh!

github-actions bot commented Jan 14, 2024

Uh oh!

github-actions bot commented Jul 14, 2024

Uh oh!

ChristopherHX commented Aug 29, 2024

Uh oh!

YOU54F commented Sep 5, 2024

Uh oh!

ChristopherHX commented Sep 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

earl-warren commented Mar 16, 2023 •

edited

Loading

earl-warren commented Mar 23, 2023 •

edited

Loading

ChristopherHX commented Mar 23, 2023 •

edited

Loading

ChristopherHX commented Sep 7, 2024 •

edited

Loading