test: run integration tests with rootless Podman #1348

Merged

mfenniak merged 1 commit from aahlenst/runner:podman-ci into main 2026-02-16 01:43:03 +00:00

Conversation 10 Commits 1 Files changed 5 +102 -13

aahlenst commented 2026-02-01 16:03:58 +00:00

Member

Copy link

• other
  • PR: test: run integration tests with rootless Podman

<!--start release-notes-assistant--> <!--URL:https://code.forgejo.org/forgejo/runner--> - other - [PR](https://code.forgejo.org/forgejo/runner/pulls/1348): <!--number 1348 --><!--line 0 --><!--description dGVzdDogcnVuIGludGVncmF0aW9uIHRlc3RzIHdpdGggcm9vdGxlc3MgUG9kbWFu-->test: run integration tests with rootless Podman<!--description--> <!--end release-notes-assistant-->

aahlenst force-pushed podman-ci from 0a3a10205f to a7bc204ae1 2026-02-01 16:10:23 +00:00 Compare

aahlenst force-pushed podman-ci from a7bc204ae1 to 756ded3f21 2026-02-01 16:21:36 +00:00 Compare

aahlenst force-pushed podman-ci from 756ded3f21 to deb2d6f923 2026-02-01 16:31:22 +00:00 Compare

aahlenst force-pushed podman-ci from deb2d6f923 to 6ff0b2948e 2026-02-01 16:55:13 +00:00 Compare

aahlenst force-pushed podman-ci from 6ff0b2948e to 9b3243020f 2026-02-01 21:44:45 +00:00 Compare

aahlenst force-pushed podman-ci from 9b3243020f to 3fee69e65c 2026-02-01 21:48:04 +00:00 Compare

aahlenst force-pushed podman-ci from 3fee69e65c to 0c002cda67 2026-02-01 21:59:51 +00:00 Compare

aahlenst force-pushed podman-ci from 0c002cda67 to 8b7270e360 2026-02-01 22:05:51 +00:00 Compare

aahlenst force-pushed podman-ci from 8b7270e360 to af3ff54f56 2026-02-01 22:09:12 +00:00 Compare

aahlenst force-pushed podman-ci from af3ff54f56 to 2773e7a7ec 2026-02-01 22:19:53 +00:00 Compare

aahlenst force-pushed podman-ci from 2773e7a7ec to 0e30242dea 2026-02-01 22:26:14 +00:00 Compare

aahlenst commented 2026-02-01 22:41:23 +00:00

Author

Member

Copy link

@viceice I'm struggling with setting up Podman. The problem seems to be systemd in conjunction with a non-privileged user. It looks like parts of systemd aren't active and other parts like systemd-logind are inactive or missing. Do you have any advice? Or can I somewhere see how lxc-trixie is created so that I can reproduce it locally?

@viceice I'm struggling with setting up Podman. The problem seems to be systemd in conjunction with a non-privileged user. It looks like [parts of systemd aren't active](https://code.forgejo.org/forgejo/runner/actions/runs/16135/jobs/5/attempt/1) and other parts like systemd-logind are inactive or missing. Do you have any advice? Or can I somewhere see how `lxc-trixie` is created so that I can reproduce it locally?

viceice commented 2026-02-02 07:47:33 +00:00

Owner

Copy link

@aahlenst wrote in #1348 (comment):

@viceice I'm struggling with setting up Podman. The problem seems to be systemd in conjunction with a non-privileged user. It looks like parts of systemd aren't active and other parts like systemd-logind are inactive or missing. Do you have any advice? Or can I somewhere see how lxc-trixie is created so that I can reproduce it locally?

https://code.forgejo.org/forgejo/runner/src/branch/main/act/runner/lxc-helpers-lib.sh

this script is used. I don't much more about it

@aahlenst wrote in https://code.forgejo.org/forgejo/runner/pulls/1348#issuecomment-76682: > @viceice I'm struggling with setting up Podman. The problem seems to be systemd in conjunction with a non-privileged user. It looks like [parts of systemd aren't active](https://code.forgejo.org/forgejo/runner/actions/runs/16135/jobs/5/attempt/1) and other parts like systemd-logind are inactive or missing. Do you have any advice? Or can I somewhere see how `lxc-trixie` is created so that I can reproduce it locally? https://code.forgejo.org/forgejo/runner/src/branch/main/act/runner/lxc-helpers-lib.sh this script is used. I don't much more about it

aahlenst commented 2026-02-02 16:52:20 +00:00

Author

Member

Copy link

Thanks. That's helpful.

It looks like the lxc-helpers are creating some problem. Without them, everything's fine:

$ sudo lxc-create --name test-container --template download -- --dist debian --release trixie --arch amd64
$ sudo lxc-start --name test-container
$ sudo lxc-attach --name test-container
root@test-container:/# systemctl status
● test-container
    State: running
    Units: 196 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
    Since: Mon 2026-02-02 15:38:17 UTC; 9s ago
  systemd: 257.9-1~deb13u1
  Tainted: unmerged-bin
   CGroup: /
           ├─.lxc
           │ ├─177 /bin/bash
           │ ├─178 systemctl status
           │ └─179 "(pager)"
           ├─init.scope
           │ └─1 /sbin/init
           └─system.slice
             ├─console-getty.service
             │ └─164 /sbin/agetty -o "-- \\u" --noreset --noclear --keep-baud 115200,57600,38400,9600 - vt220
             ├─dbus.service
             │ └─156 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
             ├─polkit.service
             │ └─168 /usr/lib/polkit-1/polkitd --no-debug --log-level=notice
             ├─systemd-hostnamed.service
             │ └─167 /usr/lib/systemd/systemd-hostnamed
             ├─systemd-journald.service
             │ └─116 /usr/lib/systemd/systemd-journald
             ├─systemd-logind.service
             │ └─157 /usr/lib/systemd/systemd-logind
             ├─systemd-networkd.service
             │ └─139 /usr/lib/systemd/systemd-networkd
             └─systemd-resolved.service
               └─135 /usr/lib/systemd/systemd-resolved
root@test-container:/# useradd --create-home --shell /bin/bash forgejo
root@test-container:/# sudo -i -u forgejo systemctl --user enable --now podman.socket
Created symlink '/home/forgejo/.config/systemd/user/sockets.target.wants/podman.socket' → '/usr/lib/systemd/user/podman.socket'.
root@test-container:/# sudo -i -u forgejo systemctl --user status podman.socket
● podman.socket - Podman API Socket
     Loaded: loaded (/usr/lib/systemd/user/podman.socket; enabled; preset: enabled)
     Active: active (listening) since Mon 2026-02-02 16:07:01 UTC; 18ms ago
 Invocation: 46457e8c6dfd488c98c7b6dd9d4fed95
   Triggers: ● podman.service
       Docs: man:podman-system-service(1)
     Listen: /run/user/1000/podman/podman.sock (Stream)
     CGroup: /user.slice/user-1000.slice/[email protected]/app.slice/podman.socket

Feb 02 16:07:01 test-container systemd[5704]: Listening on podman.socket - Podman API Socket.
root@test-container:/# sudo -i -u forgejo podman info --format '{{.Host.RemoteSocket.Exists}}'
true
root@test-container:/# sudo -i -u forgejo podman info --format '{{.Host.RemoteSocket.Path}}'
/run/user/1000/podman/podman.sock

The default configuration of the lxc-helpers is causing the degradation of systemd. That itself is concerning. A degraded systemd isn't the best foundation for running tests. One reason for the degradation is that /proc isn't mounted. That causes some services to freak out. I haven't yet figured out which part of the default configuration is responsible.

With an empty configuration, the results are slightly better. But systemd-logind is still inactive.

$ lxc-helpers.sh --os trixie --config "" lxc_container_create podman-test
$ lxc-helpers.sh lxc_container_start podman-test
$ lxc-helpers.sh lxc_container_run podman-test
root@podman-test:/# systemctl status
● podman-test
    State: running
    Units: 192 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
    Since: Mon 2026-02-02 15:40:49 UTC; 20s ago
  systemd: 257.9-1~deb13u1
  Tainted: unmerged-bin
   CGroup: /
           ├─.lxc
           │ ├─148 /bin/bash
           │ └─149 systemctl status
           ├─init.scope
           │ └─1 /sbin/init
           └─system.slice
             ├─console-getty.service
             │ └─141 /sbin/agetty -o "-- \\u" --noreset --noclear --keep-baud 115200,57600,38400,9600 - vt220
             ├─networking.service
             │ └─83 dhclient -4 -v -i -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -I -df /var/lib/dhcp/dhclient6.eth0.leases eth0
             ├─ssh.service
             │ └─143 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"
             └─systemd-journald.service
               └─43 /usr/lib/systemd/systemd-journald
root@podman-test:/# systemctl status systemd-logind
○ systemd-logind.service - User Login Management
     Loaded: loaded (/usr/lib/systemd/system/systemd-logind.service; static)
    Drop-In: /usr/lib/systemd/system/systemd-logind.service.d
             └─dbus.conf
     Active: inactive (dead)
  Condition: start condition unmet at Mon 2026-02-02 15:40:50 UTC; 28min ago
             ├─ ConditionPathExists=|/usr/bin/dbus-daemon was not met
             └─ ConditionPathExists=|/usr/bin/dbus-broker was not met
       Docs: man:sd-login(3)
             man:systemd-logind.service(8)
             man:logind.conf(5)
             man:org.freedesktop.login1(5)
   FD Store: 0 (limit: 768)

Feb 02 15:40:50 podman-test systemd[1]: systemd-logind.service - User Login Management was skipped because no trigger condition checks were met.

The problem is that neither dbus nor libpam-systemd are present, which are both required. I haven't yet figured out why they are missing. Their names do not appear anywhere in Forgejo Runner's sources.

I've seen forgejo/lxc-helpers#34. But that's not an approach I want to pursue because I expect users to use systemd.

Thanks. That's helpful. It looks like the lxc-helpers are creating some problem. Without them, everything's fine: ``` $ sudo lxc-create --name test-container --template download -- --dist debian --release trixie --arch amd64 $ sudo lxc-start --name test-container $ sudo lxc-attach --name test-container root@test-container:/# systemctl status ● test-container State: running Units: 196 loaded (incl. loaded aliases) Jobs: 0 queued Failed: 0 units Since: Mon 2026-02-02 15:38:17 UTC; 9s ago systemd: 257.9-1~deb13u1 Tainted: unmerged-bin CGroup: / ├─.lxc │ ├─177 /bin/bash │ ├─178 systemctl status │ └─179 "(pager)" ├─init.scope │ └─1 /sbin/init └─system.slice ├─console-getty.service │ └─164 /sbin/agetty -o "-- \\u" --noreset --noclear --keep-baud 115200,57600,38400,9600 - vt220 ├─dbus.service │ └─156 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only ├─polkit.service │ └─168 /usr/lib/polkit-1/polkitd --no-debug --log-level=notice ├─systemd-hostnamed.service │ └─167 /usr/lib/systemd/systemd-hostnamed ├─systemd-journald.service │ └─116 /usr/lib/systemd/systemd-journald ├─systemd-logind.service │ └─157 /usr/lib/systemd/systemd-logind ├─systemd-networkd.service │ └─139 /usr/lib/systemd/systemd-networkd └─systemd-resolved.service └─135 /usr/lib/systemd/systemd-resolved root@test-container:/# useradd --create-home --shell /bin/bash forgejo root@test-container:/# sudo -i -u forgejo systemctl --user enable --now podman.socket Created symlink '/home/forgejo/.config/systemd/user/sockets.target.wants/podman.socket' → '/usr/lib/systemd/user/podman.socket'. root@test-container:/# sudo -i -u forgejo systemctl --user status podman.socket ● podman.socket - Podman API Socket Loaded: loaded (/usr/lib/systemd/user/podman.socket; enabled; preset: enabled) Active: active (listening) since Mon 2026-02-02 16:07:01 UTC; 18ms ago Invocation: 46457e8c6dfd488c98c7b6dd9d4fed95 Triggers: ● podman.service Docs: man:podman-system-service(1) Listen: /run/user/1000/podman/podman.sock (Stream) CGroup: /user.slice/user-1000.slice/[email protected]/app.slice/podman.socket Feb 02 16:07:01 test-container systemd[5704]: Listening on podman.socket - Podman API Socket. root@test-container:/# sudo -i -u forgejo podman info --format '{{.Host.RemoteSocket.Exists}}' true root@test-container:/# sudo -i -u forgejo podman info --format '{{.Host.RemoteSocket.Path}}' /run/user/1000/podman/podman.sock ``` The default configuration of the lxc-helpers is causing the degradation of systemd. That itself is concerning. A degraded systemd isn't the best foundation for running tests. One reason for the degradation is that `/proc` isn't mounted. That causes some services to freak out. I haven't yet figured out which part of the default configuration is responsible. With an empty configuration, the results are slightly better. But systemd-logind is still inactive. ``` $ lxc-helpers.sh --os trixie --config "" lxc_container_create podman-test $ lxc-helpers.sh lxc_container_start podman-test $ lxc-helpers.sh lxc_container_run podman-test root@podman-test:/# systemctl status ● podman-test State: running Units: 192 loaded (incl. loaded aliases) Jobs: 0 queued Failed: 0 units Since: Mon 2026-02-02 15:40:49 UTC; 20s ago systemd: 257.9-1~deb13u1 Tainted: unmerged-bin CGroup: / ├─.lxc │ ├─148 /bin/bash │ └─149 systemctl status ├─init.scope │ └─1 /sbin/init └─system.slice ├─console-getty.service │ └─141 /sbin/agetty -o "-- \\u" --noreset --noclear --keep-baud 115200,57600,38400,9600 - vt220 ├─networking.service │ └─83 dhclient -4 -v -i -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -I -df /var/lib/dhcp/dhclient6.eth0.leases eth0 ├─ssh.service │ └─143 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups" └─systemd-journald.service └─43 /usr/lib/systemd/systemd-journald root@podman-test:/# systemctl status systemd-logind ○ systemd-logind.service - User Login Management Loaded: loaded (/usr/lib/systemd/system/systemd-logind.service; static) Drop-In: /usr/lib/systemd/system/systemd-logind.service.d └─dbus.conf Active: inactive (dead) Condition: start condition unmet at Mon 2026-02-02 15:40:50 UTC; 28min ago ├─ ConditionPathExists=|/usr/bin/dbus-daemon was not met └─ ConditionPathExists=|/usr/bin/dbus-broker was not met Docs: man:sd-login(3) man:systemd-logind.service(8) man:logind.conf(5) man:org.freedesktop.login1(5) FD Store: 0 (limit: 768) Feb 02 15:40:50 podman-test systemd[1]: systemd-logind.service - User Login Management was skipped because no trigger condition checks were met. ``` The problem is that neither dbus nor libpam-systemd are present, which are both required. I haven't yet figured out why they are missing. Their names do not appear anywhere in Forgejo Runner's sources. I've seen https://code.forgejo.org/forgejo/lxc-helpers/issues/34. But that's not an approach I want to pursue because I expect users to use systemd.

aahlenst force-pushed podman-ci from 0e30242dea to c15dca1863 2026-02-03 14:16:32 +00:00 Compare

aahlenst force-pushed podman-ci from c15dca1863 to e191e62e06 2026-02-03 14:26:39 +00:00 Compare

aahlenst force-pushed podman-ci from e191e62e06 to 56472b28ac 2026-02-03 15:02:24 +00:00 Compare

aahlenst force-pushed podman-ci from 56472b28ac to 1cdf2fafe3 2026-02-03 15:37:56 +00:00 Compare

aahlenst force-pushed podman-ci from 1cdf2fafe3 to fb85469ccc 2026-02-03 15:42:10 +00:00 Compare

aahlenst force-pushed podman-ci from fb85469ccc to 146e72accd 2026-02-03 15:47:42 +00:00 Compare

aahlenst force-pushed podman-ci from 146e72accd to 0b440da167 2026-02-03 16:34:48 +00:00 Compare

aahlenst force-pushed podman-ci from 0b440da167 to c7d6af9f4e 2026-02-03 16:41:27 +00:00 Compare

aahlenst force-pushed podman-ci from c7d6af9f4e to 9eb056115a 2026-02-03 22:10:26 +00:00 Compare

aahlenst force-pushed podman-ci from 9eb056115a to c07ccd0319 2026-02-03 22:13:20 +00:00 Compare

aahlenst force-pushed podman-ci from c07ccd0319 to 6379d75fb7 2026-02-03 22:17:25 +00:00 Compare

aahlenst force-pushed podman-ci from 6379d75fb7 to aa4f37069a 2026-02-03 22:20:18 +00:00 Compare

aahlenst force-pushed podman-ci from aa4f37069a to b1a33ac3a3 2026-02-03 22:23:47 +00:00 Compare

aahlenst force-pushed podman-ci from b1a33ac3a3 to 555dc771d5 2026-02-03 22:28:34 +00:00 Compare

aahlenst force-pushed podman-ci from 555dc771d5 to 154f9657d5 2026-02-03 22:30:46 +00:00 Compare

aahlenst force-pushed podman-ci from 154f9657d5 to 4b82421c6d 2026-02-03 22:33:29 +00:00 Compare

aahlenst force-pushed podman-ci from 4b82421c6d to 4f0000f6d5 2026-02-03 22:35:15 +00:00 Compare

aahlenst force-pushed podman-ci from 4f0000f6d5 to 790b9894db 2026-02-04 16:56:34 +00:00 Compare

aahlenst force-pushed podman-ci from 790b9894db to 23180253c9 2026-02-04 17:00:26 +00:00 Compare

aahlenst force-pushed podman-ci from 23180253c9 to d03584f55f 2026-02-04 18:43:20 +00:00 Compare

aahlenst force-pushed podman-ci from d03584f55f to 91a8f1ac26 2026-02-04 18:45:51 +00:00 Compare

aahlenst force-pushed podman-ci from 91a8f1ac26 to ea7e72a76a 2026-02-04 19:08:04 +00:00 Compare

aahlenst force-pushed podman-ci from ea7e72a76a to d3f18e2605 2026-02-04 21:50:06 +00:00 Compare

aahlenst force-pushed podman-ci from d3f18e2605 to 11115886a6 2026-02-05 15:09:35 +00:00 Compare

aahlenst force-pushed podman-ci from 11115886a6 to 58f54787f4 2026-02-05 15:47:05 +00:00 Compare

aahlenst force-pushed podman-ci from 58f54787f4 to e9cbd00e4e 2026-02-08 16:35:34 +00:00 Compare

aahlenst force-pushed podman-ci from e9cbd00e4e to 07f1ed59cf 2026-02-08 18:49:00 +00:00 Compare

aahlenst force-pushed podman-ci from 07f1ed59cf to 155040ec3e 2026-02-08 21:11:04 +00:00 Compare

aahlenst commented 2026-02-08 21:27:12 +00:00

Author

Member

Copy link

Okay, this is getting ridiculous. Even if I ultimately get it green, a runtime of over 2 hours is not sustainable. With Docker, the very same tests run in roughly 15 minutes. Locally, without LXC underneath, it takes roughly 1 hour 45 min. So not much better.

Podman seems to have problems stopping containers and --init isn't helping much if at all. Output from my workstation with Fedora 43 on AMD64:

$ podman run --rm -d -t docker.io/library/ubuntu:24.04
12afa1f270366a01aee53a0039773f7134c12032f089f2ff8ab17be00587fda7
$ podman stop 12afa1f270366a01aee53a0039773f7134c12032f089f2ff8ab17be00587fda7
WARN[0010] StopSignal SIGTERM failed to stop container quizzical_sammet in 10 seconds, resorting to SIGKILL 
12afa1f270366a01aee53a0039773f7134c12032f089f2ff8ab17be00587fda7
$ podman --version
podman version 5.7.1

@mfenniak Do you have any ideas? If not, should I abandon this effort or try to reach out to the Podman project for assistance?

Okay, this is getting ridiculous. Even if I ultimately get it green, a runtime of over 2 hours is not sustainable. With Docker, the very same tests run in roughly 15 minutes. Locally, without LXC underneath, it takes roughly 1 hour 45 min. So not much better. Podman seems to have problems stopping containers and `--init` isn't helping much if at all. Output from my workstation with Fedora 43 on AMD64: ``` $ podman run --rm -d -t docker.io/library/ubuntu:24.04 12afa1f270366a01aee53a0039773f7134c12032f089f2ff8ab17be00587fda7 $ podman stop 12afa1f270366a01aee53a0039773f7134c12032f089f2ff8ab17be00587fda7 WARN[0010] StopSignal SIGTERM failed to stop container quizzical_sammet in 10 seconds, resorting to SIGKILL 12afa1f270366a01aee53a0039773f7134c12032f089f2ff8ab17be00587fda7 $ podman --version podman version 5.7.1 ``` @mfenniak Do you have any ideas? If not, should I abandon this effort or try to reach out to the Podman project for assistance?

mfenniak commented 2026-02-09 01:58:43 +00:00

Owner

Copy link

I do agree that podman has some weird shutdown behaviour, as I find that my podman-based runner tends to keep sending UpdateTask service calls long after a workflow is complete. Whatever that weird behaviour is, it's a reasonable guess that it's the cause of the slow test.

But I think that your pared down reproduction isn't a good example of it -- docker behaves the same way in this test. (note here I've timed the stop because it doesn't give a useful warning)

$ docker run --rm -d -t docker.io/library/ubuntu:24.04
03651f10c9c6089b9958f6ec5d26cbafd0a2a70b8783981d75241a642fc669a9

$ time docker stop -t 30 03651f10c9c6089b9958f6ec5d26cbafd0a2a70b8783981d75241a642fc669a9
03651f10c9c6089b9958f6ec5d26cbafd0a2a70b8783981d75241a642fc669a9
docker stop -t 30   0.01s user 0.02s system 0% cpu 30.163 total

I like the idea of stripping this problem down to something that can be reproduced with the CLI, and then asking for advice from the podman project if there's not an obvious answer remaining. It makes sense to me to shelve this effort until this shutdown timing issue can be improved.

I do agree that podman has some weird shutdown behaviour, as I find that my podman-based runner tends to keep sending `UpdateTask` service calls long after a workflow is complete. Whatever that weird behaviour is, it's a reasonable guess that it's the cause of the slow test. But I think that your pared down reproduction isn't a good example of it -- docker behaves the same way in this test. (note here I've timed the `stop` because it doesn't give a useful warning) ``` $ docker run --rm -d -t docker.io/library/ubuntu:24.04 03651f10c9c6089b9958f6ec5d26cbafd0a2a70b8783981d75241a642fc669a9 $ time docker stop -t 30 03651f10c9c6089b9958f6ec5d26cbafd0a2a70b8783981d75241a642fc669a9 03651f10c9c6089b9958f6ec5d26cbafd0a2a70b8783981d75241a642fc669a9 docker stop -t 30 0.01s user 0.02s system 0% cpu 30.163 total ``` I like the idea of stripping this problem down to something that can be reproduced with the CLI, and then asking for advice from the podman project if there's not an obvious answer remaining. It makes sense to me to shelve this effort until this shutdown timing issue can be improved.

aahlenst commented 2026-02-09 16:14:25 +00:00

Author

Member

Copy link

I created a Bash script that resembles what go test -count 1 -run 'TestRunner_RunEvent/shells/bash$' -v ./... does:

#! /usr/bin/env bash

set -euo pipefail

container=$(docker run --detach --entrypoint="/usr/bin/tail" docker.io/library/node:20-bookworm "-f" "/dev/null")
docker exec "$container" bash -c 'if [[ -n "$BASH" ]]; then echo "I am $BASH"; else exit 1; fi'
docker rm -f -v "$container"

container=$(docker run --detach --entrypoint="/usr/bin/tail" docker.io/library/node:22-bookworm "-f" "/dev/null")
docker exec "$container" bash -c 'if [[ -n "$BASH" ]]; then echo "I am $BASH"; else exit 1; fi'
docker rm -f -v "$container"

container=$(docker run --detach --entrypoint="/usr/bin/tail" docker.io/library/node:20-bookworm "-f" "/dev/null")
docker exec "$container" bash -c 'if [[ -n "$BASH" ]]; then echo "I am $BASH"; else exit 1; fi'
docker rm -f -v "$container"

Podman took way longer than Docker to complete it, warning that SIGTERM did not work after every rm. After reading this Podman issue where someone complained about the warning, I concluded that it is expected behaviour. Which means that it needs to be fixed in Forgejo Runner.

Inserting podman stop -t 0 "$container" before every rm makes Podman as fast as Docker.

I measured how long it takes Forgejo Runner to remove a container with Podman:

act/container/docker_run.go

Lines 452 to 455 in 0f65ceb

	err := cr.cli.ContainerRemove(ctx, cr.id, container.RemoveOptions{
	RemoveVolumes: true,
	Force: true,
	})

A little more than 10 seconds.

Adding cr.cli.ContainerStop(ctx, cr.id, container.StopOptions{Timeout: &zero}); before ContainerRemove() brings the total time down to less than 1 second.

For some reason, the tests are still slow. 🤔

I created a Bash script that resembles what `go test -count 1 -run 'TestRunner_RunEvent/shells/bash$' -v ./...` does: ```bash #! /usr/bin/env bash set -euo pipefail container=$(docker run --detach --entrypoint="/usr/bin/tail" docker.io/library/node:20-bookworm "-f" "/dev/null") docker exec "$container" bash -c 'if [[ -n "$BASH" ]]; then echo "I am $BASH"; else exit 1; fi' docker rm -f -v "$container" container=$(docker run --detach --entrypoint="/usr/bin/tail" docker.io/library/node:22-bookworm "-f" "/dev/null") docker exec "$container" bash -c 'if [[ -n "$BASH" ]]; then echo "I am $BASH"; else exit 1; fi' docker rm -f -v "$container" container=$(docker run --detach --entrypoint="/usr/bin/tail" docker.io/library/node:20-bookworm "-f" "/dev/null") docker exec "$container" bash -c 'if [[ -n "$BASH" ]]; then echo "I am $BASH"; else exit 1; fi' docker rm -f -v "$container" ``` Podman took way longer than Docker to complete it, warning that `SIGTERM` did not work after every `rm`. After reading [this Podman issue](https://github.com/containers/podman/issues/14656) where someone complained about the warning, I concluded that it is expected behaviour. Which means that it needs to be fixed in Forgejo Runner. Inserting `podman stop -t 0 "$container"` before every `rm` makes Podman as fast as Docker. I measured how long it takes Forgejo Runner to remove a container with Podman: https://code.forgejo.org/forgejo/runner/src/commit/0f65ceb2bea5d0a9c35e8230be5ce33fcb266770/act/container/docker_run.go#L452-L455 A little more than 10 seconds. Adding `cr.cli.ContainerStop(ctx, cr.id, container.StopOptions{Timeout: &zero});` before `ContainerRemove()` brings the total time down to less than 1 second. For some reason, the tests are still slow. 🤔

aahlenst force-pushed podman-ci from 155040ec3e to 339b5908bd 2026-02-09 16:51:11 +00:00 Compare

mfenniak commented 2026-02-09 20:09:52 +00:00

Owner

Copy link

@aahlenst wrote in #1348 (comment):

Inserting podman stop -t 0 "$container" before every rm makes Podman as fast as Docker.

🤔 OK, that makes sense to me. Always SIGTERM our containers (job, service, & step) when we're done with them, since a graceful cleanup of the entrypoint process (or tail 🤣) isn't important.

Theoretically, if someone was using a service container and mapping a volume that they reuse, there could be some risk to going immediately to a SIGKILL and prevent a clean shutdown of a service... but that isn't a reasonable usage of the runner.

@aahlenst wrote in https://code.forgejo.org/forgejo/runner/pulls/1348#issuecomment-77503: > Inserting `podman stop -t 0 "$container"` before every `rm` makes Podman as fast as Docker. 🤔 OK, that makes sense to me. Always SIGTERM our containers (job, service, & step) when we're done with them, since a graceful cleanup of the entrypoint process (or tail 🤣) isn't important. Theoretically, if someone was using a service container and mapping a volume that they reuse, there could be some risk to going immediately to a SIGKILL and prevent a clean shutdown of a service... but that isn't a reasonable usage of the runner.

👍 1

aahlenst commented 2026-02-09 22:04:48 +00:00

Author

Member

Copy link

@mfenniak wrote in #1348 (comment):

Theoretically, if someone was using a service container and mapping a volume that they reuse, there could be some risk to going immediately to a SIGKILL and prevent a clean shutdown of a service... but that isn't a reasonable usage of the runner.

The way the Docker API is being used in Forgejo Runner suggests it was the author's intent to outright kill all containers. And Docker complies. Podman is more cautious.

I tried my luck with Go's profiling tools. Found another problem:

act/container/docker_run.go

Lines 352 to 366 in 0f65ceb

	func GetHostInfo(ctx context.Context) (info system.Info, err error) {
	var cli client.APIClient
	cli, err = GetDockerClient(ctx)
	if err != nil {
	return info, err
	}
	defer cli.Close()
	info, err = cli.Info(ctx)
	if err != nil {
	return info, err
	}
	return info, nil
	}

Removing the Info call cuts the runtime of a single test in half. I have no idea yet what could be done about it.

There's another one:

act/container/docker_pull.go

Line 36 in 0f65ceb

info, err := cli.Info(ctx)

@mfenniak wrote in https://code.forgejo.org/forgejo/runner/pulls/1348#issuecomment-77518: > Theoretically, if someone was using a service container and mapping a volume that they reuse, there could be some risk to going immediately to a SIGKILL and prevent a clean shutdown of a service... but that isn't a reasonable usage of the runner. The way the Docker API is being used in Forgejo Runner suggests it was the author's intent to outright kill all containers. And Docker complies. Podman is more cautious. I tried my luck with Go's profiling tools. Found another problem: https://code.forgejo.org/forgejo/runner/src/commit/0f65ceb2bea5d0a9c35e8230be5ce33fcb266770/act/container/docker_run.go#L352-L366 Removing the `Info` call cuts the runtime of a single test in half. I have no idea yet what could be done about it. There's another one: https://code.forgejo.org/forgejo/runner/src/commit/0f65ceb2bea5d0a9c35e8230be5ce33fcb266770/act/container/docker_pull.go#L36

aahlenst force-pushed podman-ci from 339b5908bd to 05524beacf 2026-02-13 20:18:35 +00:00 Compare

aahlenst force-pushed podman-ci from 05524beacf to 226bbed730 2026-02-15 21:44:41 +00:00 Compare

aahlenst force-pushed podman-ci from 226bbed730 to e6d5c86e0b 2026-02-15 22:06:05 +00:00 Compare

aahlenst force-pushed podman-ci from e6d5c86e0b to b797c54d90 2026-02-15 22:08:50 +00:00 Compare

aahlenst changed title from WIP: chore: run integration tests with rootless Podman to test: run integration tests with rootless Podman 2026-02-15 22:09:57 +00:00

aahlenst commented 2026-02-15 22:13:20 +00:00

Author

Member

Copy link

With the recent test fixes and other improvements, I got it green and a full test run with Podman completes in less than 20 minutes.

Ready for review. 😅

With the recent test fixes and other improvements, I got it green and a [full test run with Podman completes in less than 20 minutes](https://code.forgejo.org/forgejo/runner/actions/runs/16716/jobs/5/attempt/1). Ready for review. 😅

mfenniak approved these changes 2026-02-16 01:42:58 +00:00

mfenniak left a comment

Owner

Copy link

Wonderful! 🙂 Great work.

Wonderful! 🙂 Great work.

mfenniak merged commit feff6fcb63 into main 2026-02-16 01:43:03 +00:00

mfenniak referenced this pull request from a commit 2026-02-16 01:43:05 +00:00

test: run integration tests with rootless Podman (#1348)

viceice-bot referenced this pull request from actions/setup-forgejo 2026-02-19 01:36:13 +00:00

Update dependency forgejo/runner to v12.7.0 #896

mfenniak referenced this pull request from a commit 2026-02-19 03:00:44 +00:00

Update dependency forgejo/runner to v12.7.0 (#896)

forgejo-mirror referenced this pull request from a commit 2026-02-19 04:01:08 +00:00

Update module code.forgejo.org/forgejo/runner/v12 to v12.7.0 (forgejo) (#11363)

viceice-bot referenced this pull request 2026-02-20 20:19:16 +00:00

Update forgejo-runner to v12.7.0 #1394

Sign in to join this conversation.

Reviewers

No reviewers

mfenniak

Labels

Clear labels   

FreeBSD

  

Kind/Breaking

Breaking change that won't be backward compatible

  

Kind/Bug

Something is not working

  

Kind/Chore

It is no fun but it needs to be done, this is a chore.

  

Kind/DependencyUpdate

for pull requests that are created by renovate when updating a dependency

  

Kind/Documentation

Documentation changes

  

Kind/Enhancement

Improve existing functionality

  

Kind/Feature

New functionality

  

Kind/Security

This is security issue

  

Kind/Testing

Issue or pull request related to testing

  

Priority

Critical

The priority is critical

  

Priority

High

The priority is high

  

Priority

Low

The priority is low

  

Priority

Medium

The priority is medium

  

Reviewed

Confirmed

Issue has been confirmed

  

Reviewed

Duplicate

This issue or pull request already exists

  

Reviewed

Invalid

Invalid issue

  

Reviewed

Won't Fix

This issue won't be fixed

  

Status

Abandoned

Somebody has started to work on this but abandoned work

  

Status

Blocked

Something is blocking this issue or pull request

  

Status

Need More Info

Feedback is required to reproduce issue or to continue work

  

Windows

Windows ports is an independent project, see https://github.com/Crown0815/Forgejo-runner-windows-builder

  

linux-powerpc64le

powerpc64le

  

linux-riscv64

  

linux-s390x

  

run-end-to-end-tests

Run https://code.forgejo.org/forgejo/end-to-end with a runner compiled from this pull request

  

run-forgejo-tests

Run https://codeberg.org/forgejo/forgejo tests importing the runner from this pull request

No labels

FreeBSD

Kind/Breaking

Kind/Bug

Kind/Chore

Kind/DependencyUpdate

Kind/Documentation

Kind/Enhancement

Kind/Feature

Kind/Security

Kind/Testing

Priority

Critical

Priority

High

Priority

Low

Priority

Medium

Reviewed

Confirmed

Reviewed

Duplicate

Reviewed

Invalid

Reviewed

Won't Fix

Status

Abandoned

Status

Blocked

Status

Need More Info

Windows

linux-powerpc64le

linux-riscv64

linux-s390x

run-end-to-end-tests

run-forgejo-tests

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

3 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

forgejo/runner!1348

No description provided.