kernel regression in the ostree.hotfix test #2087
Description
A little bit ago the ostree.hotfix test started failing in rawhide kernel bodhi updates testing. I initially ignored the failure because there were a bunch of other tests from openQA failing on the kernel too and I figured since it was an early rc for 6.19 I'd let them fix some bugs upstream first.
Fast-forward to yesterday and I see this regression is trying to make its way into Fedora 43 with kernel-6.17.13-300.fc43 .
--- PASS: ostree.hotfix/unlock (6.50s)
--- PASS: ostree.hotfix/install (0.55s)
--- PASS: ostree.hotfix/uninstall (0.33s)
--- FAIL: ostree.hotfix/persist (604.72s)
unlock.go:226: Failed to reboot machine: machine "370f9636-d3eb-4f06-b76f-bb04a52cd64b" failed to start: ssh journalctl failed: time limit exceeded
cluster.go:173: "logger --tag kola '=== DONE: ostree.hotfix/persist ==='" failed: output , status ssh: handshake failed: read tcp 127.0.0.1:57696->127.0.0.1:40471: read: connection reset by peer
The actual failure in the logs looks more like this:
[ 5.924341] systemd[1]: Mounted sysroot.mount - /sysroot.^M
[^[[0;32m OK ^[[0m] Mounted ^[[0;1;39msysroot.mount^[[0m - /sysroot.^M^M
Starting ^[[0;1;39mostree-prepare-root.service^[[0m - OSTree Prepare OS/...^M^M
[ 5.929801] systemd[1]: Starting ostree-prepare-root.service - OSTree Prepare OS/...^M
[ 5.937052] ostree-prepare-root[563]: Resolved OSTree target to: /sysroot/ostree/deploy/fedora-coreos/deploy/4c729e29a50e7d35d24bdc475b4f4aff6f24db28795217c672b5599b8f215333.0^M
[ 5.939383] ostree-prepare-root[563]: sysroot.readonly configuration value: 1 (fs writable: 1)^M
[ 5.961621] netfs: FS-Cache loaded^M
[ 5.974573] erofs (device erofs): mounted with root inode @ nid 36.^M
[ 5.993525] systemd[1]: run-ostree-.private-cfsroot\x2dlower.mount: Deactivated successfully.^M
[ 5.995227] ostree-prepare-root[563]: composefs: mounted successfully (verity=false)^M
[ 5.998094] overlayfs: maximum fs stacking depth exceeded^M
[ 5.999891] ostree-prepare-root[563]: ostree-prepare-root: failed to mount /usr overlayfs: Invalid argument^M
[^[[0;1;31mFAILED^[[0m] Failed to start ^[[0;1;39mostree-prepare-root.service^[[0m - OSTree Prepare OS/.^M^M
Luckily this is very easy to reproduce. Just build a Fedora CoreOS testing-devel with the kernel rpms in overrides/rpm.
Once you have a build then cosa run -c and:
sudo ostree admin unlock --hotfix
reboot