fix: eliminate log data loss caused by LXC & PTY buffer #1287

Merged

mfenniak merged 2 commits from mfenniak/forgejo-runner:experiment-nsenter into main 2026-01-20 21:39:59 +00:00

Conversation 12 Commits 2 Files changed 2 +42 -38

mfenniak commented 2026-01-13 21:33:30 +00:00

Owner

Copy link

This PR is based upon #1286 which adds an environmental sanity check -- that test should pass as it's own PR first to ensure that this PR is not causing any behavioural regressions that could be detected by it.

Alternative fix for #1258. This PR replaces the use of lxc-attach with nsenter to get into the LXC container and execute workflow commands. nsenter does not have the short-write behaviour that was observed with lxc-attach where a failure to write stdout data to the pty would be ignored -- nsenter simply execvp or execl the child process and allows it to inherit its FDs.

The in-memory buffer which was intended to pull data out of the PTY buffer as fast as possible is removed in this PR, restoring normal backpressure on stdout.

• bug fixes
  • PR: fix: eliminate log data loss caused by LXC & PTY buffer

This PR is based upon #1286 which adds an environmental sanity check -- that test should pass as it's own PR first to ensure that this PR is not causing any behavioural regressions that could be detected by it. Alternative fix for #1258. This PR replaces the use of `lxc-attach` with `nsenter` to get into the LXC container and execute workflow commands. `nsenter` does not have the short-write behaviour that was observed with `lxc-attach` where a failure to write stdout data to the pty would be ignored -- `nsenter` simply [`execvp`](https://github.com/util-linux/util-linux/blob/6766bbf17ad618a6a7d35d4cffe5134da2ffe077/sys-utils/nsenter.c#L853) or [`execl`](https://github.com/util-linux/util-linux/blob/6766bbf17ad618a6a7d35d4cffe5134da2ffe077/lib/exec_shell.c#L42C2-L42C7) the child process and allows it to inherit its FDs. The in-memory buffer which was intended to pull data out of the PTY buffer as fast as possible is removed in this PR, restoring normal backpressure on stdout. <!--start release-notes-assistant--> <!--URL:https://code.forgejo.org/forgejo/runner--> - bug fixes - [PR](https://code.forgejo.org/forgejo/runner/pulls/1287): <!--number 1287 --><!--line 0 --><!--description Zml4OiBlbGltaW5hdGUgbG9nIGRhdGEgbG9zcyBjYXVzZWQgYnkgTFhDICYgUFRZIGJ1ZmZlcg==-->fix: eliminate log data loss caused by LXC & PTY buffer<!--description--> <!--end release-notes-assistant-->

👍 1 ❤️ 1

mfenniak referenced this pull request 2026-01-13 21:35:55 +00:00

fix: minimize log data loss caused by LXC & PTY buffer #1270

mfenniak commented 2026-01-13 21:39:07 +00:00

Author

Owner

Copy link

I'm confident in this change with respect to the simpler behaviour of stdout, and therefore the integrity of the logs.

However, my confidence is less strong on "is nsenter doing the same things as lxc-attach to isolate the process", as that question seems to require a lot of knowledge to fully answer that I don't have. I've researched and built the #1286 test case around what I could discover, but hopefully a reviewer will have some confidence to say that it covers the right test subjects. 🙂

I'm confident in this change with respect to the simpler behaviour of stdout, and therefore the integrity of the logs. However, my confidence is less strong on "is `nsenter` doing the same things as `lxc-attach` to isolate the process", as that question seems to require a lot of knowledge to fully answer that I don't have. I've researched and built the #1286 test case around what I could discover, but hopefully a reviewer will have some confidence to say that it covers the right test subjects. 🙂

aahlenst commented 2026-01-13 21:47:14 +00:00

Member

Copy link

I don't know anything about LXC. So I refrain from reviewing this PR. Nonetheless, thanks a lot for figuring this out. It's maddening when I try to fix CI problems and the logs are missing.

I don't know anything about LXC. So I refrain from reviewing this PR. Nonetheless, thanks a lot for figuring this out. It's maddening when I try to fix CI problems and the logs are missing.

👍 1

mfenniak force-pushed experiment-nsenter from 1fc6171c62 to 05f32612d8 2026-01-13 21:48:15 +00:00 Compare

mfenniak force-pushed experiment-nsenter from 05f32612d8 to db49823df7 2026-01-13 22:00:46 +00:00 Compare

mfenniak force-pushed experiment-nsenter from db49823df7 to 20e559cccd 2026-01-13 22:40:55 +00:00 Compare

mfenniak force-pushed experiment-nsenter from 20e559cccd to 771f35dc32 2026-01-14 02:43:56 +00:00 Compare

mfenniak added the

run-end-to-end-tests

label 2026-01-14 20:12:05 +00:00

cascading-pr referenced this pull request from actions/setup-forgejo 2026-01-14 20:13:40 +00:00

cascading-pr from https://code.forgejo.org/forgejo/runner refs/pull/1287/head to forgejo/runner-1287 #829

cascading-pr commented 2026-01-14 20:13:41 +00:00

Contributor

Copy link

cascading-pr updated at actions/setup-forgejo#829

cascading-pr updated at https://code.forgejo.org/actions/setup-forgejo/pulls/829

mfenniak added the

Kind/Bug

label 2026-01-15 22:39:32 +00:00

aahlenst commented 2026-01-16 15:18:11 +00:00

Member

Copy link

As nobody with LXC expertise has shown up so far: the implementation looks good, the previously added test looks good, too. So I'm willing to hit that "Approve" button. If it turns out that there's a problem, we can still revert.

In any case, it might make sense to report our problem to the LXC project. They might know a better workaround, greenlight our workaround, and perhaps even enhance lxc-attach.

As nobody with LXC expertise has shown up so far: the implementation looks good, the previously added test looks good, too. So I'm willing to hit that "Approve" button. If it turns out that there's a problem, we can still revert. In any case, it might make sense to report our problem to the LXC project. They might know a better workaround, greenlight our workaround, and perhaps even enhance `lxc-attach`.

mfenniak commented 2026-01-16 17:28:50 +00:00

Author

Owner

Copy link

I've expanded my scope looking for help from the Forgejo Dev chat, to Forgejo Chat, and now to the LXC forum (https://discuss.linuxcontainers.org/t/fixing-forgejo-runners-lxc-logging/25918). I'll let this sit for another day to see if these efforts provide any feedback.

I've expanded my scope looking for help from the Forgejo Dev chat, to Forgejo Chat, and now to the LXC forum (https://discuss.linuxcontainers.org/t/fixing-forgejo-runners-lxc-logging/25918). I'll let this sit for another day to see if these efforts provide any feedback.

mfenniak referenced this pull request from forgejo/end-to-end 2026-01-20 02:23:13 +00:00

feat: add OIDC workload identity federation tests #1364

mfenniak commented 2026-01-20 04:08:49 +00:00

Author

Owner

Copy link

@aahlenst wrote in #1287 (comment):

As nobody with LXC expertise has shown up so far: the implementation looks good, the previously added test looks good, too. So I'm willing to hit that "Approve" button. If it turns out that there's a problem, we can still revert.

I'll take you up on that offer if you're still good with it. Upstream discussion with LXC (link in last comment) is promising in terms of collaborating on a fix, but in the near term I think this change will help close the logging gap.

@aahlenst wrote in https://code.forgejo.org/forgejo/runner/pulls/1287#issuecomment-73767: > As nobody with LXC expertise has shown up so far: the implementation looks good, the previously added test looks good, too. So I'm willing to hit that "Approve" button. If it turns out that there's a problem, we can still revert. I'll take you up on that offer if you're still good with it. Upstream discussion with LXC (link in last comment) is promising in terms of collaborating on a fix, but in the near term I think this change will help close the logging gap.

aahlenst approved these changes 2026-01-20 11:31:16 +00:00

aahlenst commented 2026-01-20 11:33:51 +00:00

Member

Copy link

@mfenniak wrote in #1287 (comment):

Upstream discussion with LXC (link in last comment) is promising in terms of collaborating on a fix.

I've been following it and it's great to see because both projects will be better off afterwards 🤞

@mfenniak wrote in https://code.forgejo.org/forgejo/runner/pulls/1287#issuecomment-74426: > Upstream discussion with LXC (link in last comment) is promising in terms of collaborating on a fix. I've been following it and it's great to see because both projects will be better off afterwards 🤞

mfenniak merged commit 6721a406e5 into main 2026-01-20 21:39:59 +00:00

mfenniak deleted branch experiment-nsenter 2026-01-20 21:40:00 +00:00

mfenniak referenced this pull request from a commit 2026-01-20 21:40:00 +00:00

fix: eliminate log data loss caused by LXC & PTY buffer (#1287)

mihalicyn commented 2026-01-22 15:46:49 +00:00

First-time contributor

Copy link

Dear friends,

I'm developer from LXC project :)

Mathieu asked me to look at this PR and leave some comment.

This particular PR looks good if it solves the problem, but please, don't keep this as a permanent solution (because it is really easy to mess up with the containers and the way you enter them from a security standpoint and not only) and make sure that you switch back to lxc-attach and help us to validate next LXC release to ensure that the problem went away.

PR with the fix https://github.com/lxc/lxc/pull/4633

Kind regards,
Alex

Dear friends, I'm developer from LXC project :) Mathieu [asked me](https://discuss.linuxcontainers.org/t/fixing-forgejo-runners-lxc-logging/25918/4?u=amikhalitsyn) to look at this PR and leave some comment. This particular PR looks good if it solves the problem, but please, don't keep this as a permanent solution (because it is really easy to mess up with the containers and the way you enter them from a security standpoint and not only) and make sure that you switch back to `lxc-attach` and help us to validate next LXC release to ensure that the problem went away. PR with the fix https://github.com/lxc/lxc/pull/4633 Kind regards, Alex

mfenniak commented 2026-01-22 15:58:22 +00:00

Author

Owner

Copy link

@mihalicyn Thank you for your time taking a look at this! 🙂 It is definitely intended as a temporary solution, and I'll be happy to help validate the upstream fix.

@mihalicyn Thank you for your time taking a look at this! 🙂 It is definitely intended as a temporary solution, and I'll be happy to help validate the upstream fix.

mfenniak referenced this pull request 2026-01-22 17:27:24 +00:00

fix: revert switch from lxc-attach to nsenter #1317

viceice commented 2026-01-22 20:29:15 +00:00

Owner

Copy link

@mihalicyn What would be the best option to get the fixed lxc version into our lxc templates? it shouldn't need too much additional time

@mihalicyn What would be the best option to get the fixed lxc version into our lxc templates? it shouldn't need too much additional time

viceice commented 2026-01-22 20:31:35 +00:00

Owner

Copy link

https://code.forgejo.org/forgejo/lxc-helpers

we use these lxc helper scripts to create the container and templates. we need to support bookworm and trixie, older releases could be dropped if needed

https://code.forgejo.org/forgejo/lxc-helpers we use these lxc helper scripts to create the container and templates. we need to support bookworm and trixie, older releases could be dropped if needed

viceice-bot referenced this pull request 2026-01-23 00:31:10 +00:00

Update forgejo-runner to v12.6.0 #1321

viceice-bot referenced this pull request from actions/setup-forgejo 2026-01-23 00:34:13 +00:00

Update dependency forgejo/runner to v12.6.0 #846

mfenniak referenced this pull request from a commit 2026-01-23 02:07:37 +00:00

Update forgejo-runner to v12.6.0 (#1321)

mfenniak referenced this pull request from a commit 2026-01-23 02:23:44 +00:00

Update dependency forgejo/runner to v12.6.0 (#846)

forgejo-mirror referenced this pull request from a commit 2026-01-23 03:01:19 +00:00

Update module code.forgejo.org/forgejo/runner/v12 to v12.6.0 (forgejo) (#10999)

forgejo-mirror referenced this pull request from a commit 2026-01-23 04:01:23 +00:00

Update module code.forgejo.org/forgejo/runner/v12 to v12.6.0 (v14.0/forgejo) (#11001)

mfenniak referenced this pull request 2026-01-23 18:09:31 +00:00

fix: PATH modifications are lost in LXC executions #1326

mfenniak referenced this pull request from a commit 2026-01-23 19:03:19 +00:00

fix: `PATH` modifications are lost in LXC executions (#1326)

viceice-bot referenced this pull request from actions/setup-forgejo 2026-01-23 20:32:28 +00:00

Update dependency forgejo/runner to v12.6.1 #852

mfenniak referenced this pull request from a commit 2026-01-23 20:51:47 +00:00

Update dependency forgejo/runner to v12.6.1 (#852)

mfenniak referenced this pull request 2026-01-25 19:29:30 +00:00

fix: inputs containing '-' don't function in LXC executor #1336

mfenniak referenced this pull request from a commit 2026-01-26 01:30:51 +00:00

fix: inputs containing '-' don't function in LXC executor (#1336)

Sign in to join this conversation.

Reviewers

No reviewers

aahlenst

Labels

Clear labels   

FreeBSD

  

Kind/Breaking

Breaking change that won't be backward compatible

  

Kind/Bug

Something is not working

  

Kind/Chore

It is no fun but it needs to be done, this is a chore.

  

Kind/DependencyUpdate

for pull requests that are created by renovate when updating a dependency

  

Kind/Documentation

Documentation changes

  

Kind/Enhancement

Improve existing functionality

  

Kind/Feature

New functionality

  

Kind/Security

This is security issue

  

Kind/Testing

Issue or pull request related to testing

  

Priority

Critical

The priority is critical

  

Priority

High

The priority is high

  

Priority

Low

The priority is low

  

Priority

Medium

The priority is medium

  

Reviewed

Confirmed

Issue has been confirmed

  

Reviewed

Duplicate

This issue or pull request already exists

  

Reviewed

Invalid

Invalid issue

  

Reviewed

Won't Fix

This issue won't be fixed

  

Status

Abandoned

Somebody has started to work on this but abandoned work

  

Status

Blocked

Something is blocking this issue or pull request

  

Status

Need More Info

Feedback is required to reproduce issue or to continue work

  

Windows

Windows ports is an independent project, see https://github.com/Crown0815/Forgejo-runner-windows-builder

  

linux-powerpc64le

powerpc64le

  

linux-riscv64

  

linux-s390x

  

run-end-to-end-tests

Run https://code.forgejo.org/forgejo/end-to-end with a runner compiled from this pull request

  

run-forgejo-tests

Run https://codeberg.org/forgejo/forgejo tests importing the runner from this pull request

No labels

FreeBSD

Kind/Breaking

Kind/Bug

Kind/Chore

Kind/DependencyUpdate

Kind/Documentation

Kind/Enhancement

Kind/Feature

Kind/Security

Kind/Testing

Priority

Critical

Priority

High

Priority

Low

Priority

Medium

Reviewed

Confirmed

Reviewed

Duplicate

Reviewed

Invalid

Reviewed

Won't Fix

Status

Abandoned

Status

Blocked

Status

Need More Info

Windows

linux-powerpc64le

linux-riscv64

linux-s390x

run-end-to-end-tests

run-forgejo-tests

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

5 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

forgejo/runner!1287

No description provided.