systemd: make systemd-ssh-generator work by NyCodeGHG · Pull Request #372979 · NixOS/nixpkgs

NyCodeGHG · 2025-01-11T16:39:10Z

Makes systemd-ssh-generator and systemd-ssh-proxy work on NixOS.

I successfully built this and ran nixosTests.systemd-ssh-proxy and nixosTests.openssh.
I also built the minimal iso and manually tested the vsock feature with a qemu vm.

closes #371191

Things done

Add a 👍 reaction to pull requests you find important.

phaer

Very nice to see this coming to upstream NixOS, thank you very much!

phaer · 2025-01-29T11:37:59Z

nixos/doc/manual/release-notes/rl-2505.section.md

I don't think there's a proper issue about this yet, but systemd itself would also support passing credentials via kernel params which could be very useful for VMs. But that doesn't work with our current systemd-in-stage-1 implementation. @ElvishJerricco wrote a reproducer for that in https://gist.github.com/ElvishJerricco/dca95eb4ea9fc410bd525c3b15b68fdd

nixos/tests/systemd-ssh-proxy.nix

r-vdp

Overall this looks good, I will try to test it out over the next days.

nixos/modules/services/networking/ssh/sshd.nix

pkgs/os-specific/linux/systemd/default.nix

NyCodeGHG · 2025-02-05T18:52:02Z

rebased onto latest staging to get around bluez and linux build failure when building nixosTests.systemd-ssh-proxy.driverInteractive

We need to patch systemd to make systemd-ssh-generator(8) work: - systemd doesn't follow symlinks when checking for a packaged [email protected] unit - systemd tries to link the ssh config dropins by default with tmpfiles to /usr, that is not possible, so we include the snippet manually.

NyCodeGHG · 2025-02-18T18:56:55Z

We should probably rename programs.ssh.enableSystemdSshProxy to programs.ssh.systemd-ssh-proxy.enable, that looks a lot better

philiptaron

I have a few questions about the systemd service files I'd like to understand before ✅, but I'm inclined to merge after that.

The one piece that looks most scary to me is the new tmpfiles.d configuration, which is doing more than just SSH stuff.

nixos/modules/programs/ssh.nix

nixos/modules/services/networking/ssh/sshd.nix

philiptaron · 2025-03-14T15:51:54Z

nixos/modules/services/networking/ssh/sshd.nix

https://linux.die.net/man/8/sshd:

-i Specifies that sshd is being run from inetd(8). sshd is normally not run from inetd because it needs to generate the server key before it can respond to the client, and this may take tens of seconds. Clients would have to wait too long if the key was regenerated every time. However, with small key sizes (e.g. 512) using sshd from inetd may be feasible.

This documentation makes me confused. I'm assuming socket activation is inetd-like, and this is the flag that's different. However the rest (about waiting too long, generating keys) is odd.

I'm also confused by this, not sure if host keys are meant?
Since those are generated a single time

nixos/modules/services/networking/ssh/sshd.nix

philiptaron · 2025-03-14T16:14:41Z

nixos/tests/systemd-ssh-proxy.nix

🐍🪔

Wonderful.

philiptaron · 2025-03-14T16:15:33Z

nixos/tests/systemd-ssh-proxy.nix

I love this worked example of how to do this. One of the huge benefits of tests is just showing how to do things.

It would have been nice to avoid using an ISO for this. That's a very large build to be pushing to the cache that no one will ever use (yes, Hydra pushes these intermediate builds).

Also, this is technically nested virtualisation (double nested if the builder itself is a VM), and I'm not sure if we can take that as a given on the hardware we use for Hydra builders?

I believe we can test this functionality with nixos-containers instead. That would avoid both problems. I'll give it a look.

philiptaron · 2025-03-14T16:17:45Z

pkgs/os-specific/linux/systemd/0019-meson-Don-t-link-ssh-dropins.patch

The commit introducing this patch says:

systemd tries to link the ssh config dropins by default with tmpfiles to /usr, that is not possible, so we include the snippet manually.

philiptaron · 2025-03-14T16:18:17Z

pkgs/os-specific/linux/systemd/0020-install-unit_file_exists_full-follow-symlinks.patch

The commit introducing this patch says:

systemd doesn't follow symlinks when checking for a packaged [email protected] unit

philiptaron · 2025-03-14T16:21:09Z

pkgs/os-specific/linux/systemd/default.nix

Future: sshdconfdir (note the d) is about https://github.com/systemd/systemd/blob/3a15daf4402ba2e35c27f21339e987d89f4f8a97/src/userdb/20-systemd-userdb.conf.in

Make sure SSH authorized keys recorded in user records can be consumed by SSH

This enables systemd-ssh-generator to find the sshd binary.

…r(1)

philiptaron

I'm planning on approving after the patch to do only the SSH provisioning (not the hosts or MOTD stuff) lands.

philiptaron · 2025-03-14T18:09:10Z

nixos/modules/programs/ssh.nix

https://github.com/systemd/systemd/blob/main/src/ssh-generator/20-systemd-ssh-proxy.conf.in

philiptaron · 2025-03-14T18:21:01Z

nixos/modules/system/boot/systemd.nix

We could (and I kind of think maybe we should?) patch this:

https://github.com/systemd/systemd/blob/3a15daf4402ba2e35c27f21339e987d89f4f8a97/src/ssh-generator/ssh-generator.c#L463

philiptaron · 2025-03-14T18:25:10Z

nixos/modules/system/boot/systemd/tmpfiles.nix

Yes, I'd like that.

This enables provisioning of root ssh keys with systemd credentials (e.g. passed in via smbios strings or kernel params)

philiptaron

Test passes, and tested again rebased onto latest staging.

I did notice a few odd things:

There are i686 stdenv things in the closure of the test!
There are musl things in the closure of the test!
The test ends successfully, but in the last messages on screen is

vm-test-run-systemd-ssh-proxy> virthost # [   30.559645] systemd[1]: Failed to start SSH per-connection Daemon (PID 1077/UID 1000).
vm-test-run-systemd-ssh-proxy> virthost # [   30.560918] systemd[1]: session-1.scope: Deactivated successfully.
vm-test-run-systemd-ssh-proxy> virthost # [   30.563206] systemd-logind[728]: Removed session 1.

I didn't debug further.

sternenseemann · 2025-04-15T12:09:37Z

I think this may have broken more NixOS tests. I've found an instance of this in the GitLab test which blocks the channel #398846. It's likely worth checking if this affects other tests as well.

NyCodeGHG · 2025-04-15T15:19:36Z

@sternenseemann thanks for noticing! I've had a quick look with a few ripgreps and the only test which looked like it could have been affected was nixosTests.xxh, but that one passes.

SSH key generation was split out into its own systemd service in #372979, but dependent service definitions weren't updated. The `apply-ec2-data` service needs to run before SSH key generation, as it fetches host keys defined in ec2 user data and these keys should take priority over generating new ones. Currently, the ordering doesn't specify which should run first of `apply-ec2-data` and `sshd-keygen`; in practice it seems that `sshd-keygen` often wins the race, though. Update the dependencies so that `apply-ec2-data` always runs first.

SSH key generation was split out into its own systemd service in #372979, but dependent service definitions weren't updated. The `apply-ec2-data` service needs to run before SSH key generation, as it fetches host keys defined in ec2 user data and these keys should take priority over generating new ones. Currently, the ordering doesn't specify which should run first of `apply-ec2-data` and `sshd-keygen`; in practice it seems that `sshd-keygen` often wins the race, though. Update the dependencies so that `apply-ec2-data` always runs first. (cherry picked from commit d9ac3ba)

SSH key generation was split out into its own systemd service in NixOS#372979, but dependent service definitions weren't updated. The `apply-ec2-data` service needs to run before SSH key generation, as it fetches host keys defined in ec2 user data and these keys should take priority over generating new ones. Currently, the ordering doesn't specify which should run first of `apply-ec2-data` and `sshd-keygen`; in practice it seems that `sshd-keygen` often wins the race, though. Update the dependencies so that `apply-ec2-data` always runs first. (cherry picked from commit d9ac3ba)

nix-owners bot requested review from flokli and kloenk January 11, 2025 16:48

NyCodeGHG force-pushed the systemd-ssh branch from 29ab9ba to 23b36d5 Compare January 11, 2025 21:14

NyCodeGHG marked this pull request as ready for review January 11, 2025 21:30

nix-owners bot requested review from ElvishJerricco, aanderse and arianvp January 11, 2025 21:32

NyCodeGHG force-pushed the systemd-ssh branch from 23b36d5 to da9dd95 Compare January 27, 2025 08:35

github-actions bot added the 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. label Jan 27, 2025

phaer approved these changes Jan 29, 2025

View reviewed changes

wegank added the 12.approvals: 1 This PR was reviewed and approved by one person. label Feb 1, 2025

Shawn8901 mentioned this pull request Feb 2, 2025

systemd-ssh-generator[43858]: Disabling SSH generator logic, since sshd is not installed. #371191

Open

NyCodeGHG force-pushed the systemd-ssh branch from da9dd95 to cf51558 Compare February 4, 2025 17:10

wegank removed the 12.approvals: 1 This PR was reviewed and approved by one person. label Feb 4, 2025

r-vdp reviewed Feb 4, 2025

View reviewed changes

nixos/modules/services/networking/ssh/sshd.nix Outdated Show resolved Hide resolved

pkgs/os-specific/linux/systemd/default.nix Outdated Show resolved Hide resolved

NyCodeGHG force-pushed the systemd-ssh branch 2 times, most recently from 63585df to 3ae5dfe Compare February 5, 2025 18:51

NyCodeGHG force-pushed the systemd-ssh branch from 3ae5dfe to 28c2232 Compare February 13, 2025 13:50

NyCodeGHG force-pushed the systemd-ssh branch from 28c2232 to ea8c28a Compare February 18, 2025 19:29

philiptaron self-requested a review March 6, 2025 20:30

philiptaron reviewed Mar 14, 2025

View reviewed changes

NyCodeGHG added 4 commits March 14, 2025 18:26

nixos/ssh: include systemd-ssh-proxy config

5e43294

nixos/systemd: include sshd in PATH if openssh is enabled

aab69d7

This enables systemd-ssh-generator to find the sshd binary.

nixos/sshd: move key generation into its own systemd unit

63842ed

nixos/sshd: always ship a [email protected] unit for systemd-ssh-generato…

afeb76d

…r(1)

NyCodeGHG force-pushed the systemd-ssh branch from ea8c28a to 4757d06 Compare March 14, 2025 17:37

philiptaron reviewed Mar 14, 2025

View reviewed changes

NyCodeGHG force-pushed the systemd-ssh branch from 4757d06 to 22f2ddc Compare March 15, 2025 19:14

NyCodeGHG added 3 commits March 15, 2025 20:15

nixos/sshd: enable root ssh authorized_keys provisioning

0ea1aed

This enables provisioning of root ssh keys with systemd credentials (e.g. passed in via smbios strings or kernel params)

nixosTests.systemd-ssh-proxy: init

8a641dd

doc/rl-2505: document systemd-ssh-generator changes

a0962df

NyCodeGHG force-pushed the systemd-ssh branch from 22f2ddc to a0962df Compare March 15, 2025 19:15

philiptaron approved these changes Mar 16, 2025

View reviewed changes

philiptaron merged commit f4dd3ba into NixOS:staging Mar 16, 2025
26 of 27 checks passed

NyCodeGHG deleted the systemd-ssh branch March 16, 2025 11:01

ElvishJerricco mentioned this pull request Mar 17, 2025

Fix systemd-ssh changes #390565

Merged

13 tasks

This was referenced Mar 22, 2025

nixos/test-driver: add backdoor based on systemd-ssh-proxy & AF_VSOCK #392030

Merged

nixos/test-driver: use breakpointHook to debug tests with pdb & ssh; add dump() function #392117

Closed

This was referenced Apr 15, 2025

gitlab: 17.10.1 -> 17.10.4 #398548

Merged

nixosTests.gitlab: fix #398837

Merged

philiptaron mentioned this pull request Jul 24, 2025

services.openssh: use notify-reload for sshd #427620

Merged

13 tasks

talyz mentioned this pull request Sep 18, 2025

nixos/ec2-data: sshd.service -> sshd-keygen.service #444040

Merged

13 tasks

Uh oh!

Conversation

NyCodeGHG commented Jan 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Things done

Uh oh!

phaer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

r-vdp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

NyCodeGHG commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NyCodeGHG commented Feb 18, 2025

Uh oh!

philiptaron left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

🐍🪔

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

philiptaron left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

philiptaron left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sternenseemann commented Apr 15, 2025

Uh oh!

NyCodeGHG commented Apr 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

NyCodeGHG commented Jan 11, 2025 •

edited

Loading

NyCodeGHG commented Feb 5, 2025 •

edited

Loading