Skip to content

Revert "systemd: 256.6 -> 256.7"#355284

Closed
waltmck wants to merge 1 commit intoNixOS:masterfrom
waltmck:revert-systemd
Closed

Revert "systemd: 256.6 -> 256.7"#355284
waltmck wants to merge 1 commit intoNixOS:masterfrom
waltmck:revert-systemd

Conversation

@waltmck
Copy link

@waltmck waltmck commented Nov 11, 2024

This reverts commit 66f78a9.

systemd 256.7 introduces a breaking change to systemd-boot causing the system to fail to boot on some architectures (including generation rollback) and necessitating manual recovery. From the discussion, it looks like there are some pretty serious fundamentally mistaken assumptions in their code which may take some time to rectify. In order to avoid bricking our users by default, we should revert to the last working version until this issue is worked out upstream.

Pinging maintainers:
@flokli
@kloenk

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@github-actions github-actions bot added the 6.topic: systemd Software suite that provides an array of system components for Linux operating systems. label Nov 11, 2024
@NixOSInfra NixOSInfra added the 12.first-time contribution This PR is the author's first one; please be gentle! label Nov 11, 2024
@emilazy
Copy link
Member

emilazy commented Nov 11, 2024

I think at most we should revert the specific patch that’s at fault. It’s far too close to release to do an entire systemd rollback; we actually probably can’t ship a different systemd with the final release at all since we’re already far into the last staging-next cycle before release (though we might be able to get a patch in within the first few weeks after release). As far as I know the issue has only been observed on platforms with 16K pages, which we don’t really officially support (since none of our shipped kernels do), though if the incorrectness is observable on any 4K platforms then that would certainly make it more serious.

@ElvishJerricco
Copy link
Contributor

Yea, we absolutely are not going to rollback the entire systemd version when it's only been observed as an issue on one platform. I'd be fine with shipping a revert of the offending patch (we already do that for one other issue), but rolling back systemd entirely is way too extreme of a solution.

@waltmck
Copy link
Author

waltmck commented Nov 11, 2024

Besides my Macbook (which cannot run stock NixOS) I only have a common x86 machine available to test, where the release works fine.

Besides the page size problem, from the discussion in the issue the release should be broken on any platform without physical memory available below the 4G physical address--this is uncommon, but is allowed by the x86-64 spec. @mzyngier mentioned that his Opteron A1100 from 2014 would be broken, but I do not have one on hand to verify (I replied asking him to confirm whether he can reproduce the issue).

The specific problematic commit would be this one; reverting it has been verified by @andre4ik3 to fix the problem.

alyssais added a commit to alyssais/nixpkgs that referenced this pull request Nov 11, 2024
@emilazy
Copy link
Member

emilazy commented Nov 11, 2024

Ultimately whatever we do is going to have to land within the first few weeks post‐release, unless we decide to significantly delay the release for this, which seems unlikely when the main issue is on unsupported platforms and supported hardware that may be susceptible to the issue is very rare. systemd changes have to go through staging and the current staging-next cycle is already a few days late relative to the release schedule, so our only option to address this is throwing a revert of the patch into staging and waiting for the first post‐24.11 cycle to run.

@alyssais
Copy link
Member

#355290

@RossComputerGuy
Copy link
Member

RossComputerGuy commented Nov 11, 2024

I think at most we should revert the specific patch that’s at fault. It’s far too close to release to do an entire systemd rollback; we actually probably can’t ship a different systemd with the final release at all

Agreed, if this was earlier then I wouldn't reject this but at this point, we probably should only revert the specific commit in systemd. There's already a revert patch from the issue thread (systemd/systemd#35026 (comment)). We probably could get that into staging-next and revert it after 24.11 branch off.

Edit: didn't see Alyssa's comment appear until after I made this comment.

@emilazy
Copy link
Member

emilazy commented Nov 11, 2024

We cannot put it into staging-next, it’s way too far into the cycle. It can go into the first backport cycle after release.

@RossComputerGuy
Copy link
Member

Oh ok, I didn't know that. Also didn't see that comment you posted after Alyssa made the PR until I posted my comment.

@ElvishJerricco
Copy link
Contributor

Closing in favor of #355290

github-actions bot pushed a commit that referenced this pull request Nov 15, 2024
Fixes boot for me.

Closes: #355284
(cherry picked from commit 63ce466)
vcunat pushed a commit that referenced this pull request Nov 30, 2024
Fixes boot for me.

Closes: #355284
(cherry picked from commit 63ce466)
(cherry picked from commit 91ea63f)
hsjobeki pushed a commit to hsjobeki/nixpkgs that referenced this pull request Dec 3, 2024
Fixes boot for me.

Closes: NixOS#355284
(cherry picked from commit 63ce466)
(cherry picked from commit 91ea63f)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: systemd Software suite that provides an array of system components for Linux operating systems. 12.first-time contribution This PR is the author's first one; please be gentle!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants