Skip to content

Systemd optional deps#217249

Merged
flokli merged 7 commits intoNixOS:stagingfrom
software-artificer:systemd-optional-deps
Mar 9, 2023
Merged

Systemd optional deps#217249
flokli merged 7 commits intoNixOS:stagingfrom
software-artificer:systemd-optional-deps

Conversation

@software-artificer
Copy link
Contributor

@software-artificer software-artificer commented Feb 20, 2023

Description of changes

Allow to slim down the systemd build by:

  • Making ACL (access control list) support via libacl optional;
  • Making Linux Audit Framework integration optional;
  • Making kmod/modprobe integration optional;
  • Making IDN (Internationalized Domain Names) support via libidn2 optional;
  • Making PAM (Pluggable Authentication Module) stack integration optional;
  • Fix the systemd-initrd-simple test

Fixes #216461

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual) Testing on NixOS.
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.05 Release Notes (or backporting 22.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.
Test suites

Tests were executed from the current master branch with my commits cherry-picked on top in order to avoid some broken state in the current staging.

Test logs are uploaded to pastebin due to their massive size and GH limits

@github-actions github-actions bot added the 6.topic: systemd Software suite that provides an array of system components for Linux operating systems. label Feb 20, 2023
@ofborg ofborg bot requested review from Mic92, flokli and kloenk February 20, 2023 01:56
@ofborg ofborg bot added 10.rebuild-darwin: 101-500 This PR causes between 101 and 500 packages to rebuild on Darwin. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. labels Feb 20, 2023
Copy link
Member

@flokli flokli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. CI error might be unrelated, and fix itself after another rebase to staging.

Comment on lines 135 to +137
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a question on this one: what would be the preferred way to do something like that? Would it be what I have done in the change or something like this instead:

Suggested change
assert withHomed -> withCryptsetup;
assert withHomed -> withPam;
assert withHomed -> withCryptsetup && withPam;

Copy link
Contributor Author

@software-artificer software-artificer Feb 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if this is necessary, but it seems that way, because if we go from kmod enable to kmod disabled or vice-versa we might have issues reloading some of the systemd units on nixos-rebuild switch. Please, correct me if I misunderstand this.

@software-artificer software-artificer force-pushed the systemd-optional-deps branch 2 times, most recently from e6c8a7b to 3aec4cd Compare February 21, 2023 08:09
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I understand we only care about the libudev provided by the systemdMinimal, so I have disabled the kmod integration here, because AFAIK it is only used by the systemd-modules-load, but not in the Udev implementation (this is used instead).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mobile-nixos at least uses the udevd from systemdMinimal, so this change breaks mobile-nixos.

I think it's fair enough to say that systemdStage1 or a full systemd should be used instead, but in that case we should probably also prevent udevd and rules from being installed at all in the minimal (libudev-only) variant of the package -- because I imagine cases where a kmod-free udevd is indeed desired should be extremely rare and if someone needs that they'll probably know it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lheckemann thanks for the information. I will drop a PR later to but it back on. Do we have any tests covering mobile-nixos use case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened #224421, apologies for the inconvenience. A question I have on this: would it make sense to have systemdMobile with precisely tailored set of options enabled in order to avoid running into similar problems in the future?

Comment on lines 26730 to 26734
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume we care about ACL and kmod in stage1, but I am not sure about the PAM stack. Can it be dropped?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure. It's probably needed to ssh in (?), as well as to enter the root password in case there was an error? @ElvishJerricco, any thoughts?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest I'm not entirely sure. We shouldn't need PAM, as we don't include anything else related to PAM (for instance, the ssh module in #169116 has UsePAM no). I would wager that kmod is needed for systemd-modules-load, in which case yes that is a requirement for systemd stage 1. But I'm not sure about withAcl.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ElvishJerricco!

If we aren't using PAM this early, I am happy to drop it and re-test relevant things.

ACL I think is needed. I am not deeply familiar with NixOS yet (I have just recently started playing around with it), but a lot of modern system use ACL to allow access to /dev/dri/ device nodes, and that's something Plymouth for example takes advantage of IIRC.

@software-artificer
Copy link
Contributor Author

Hey @flokli, thanks a lot for your guidance so far. I think I have trimmed everything to the bare minimum at this point, but I have a bunch of questions that I would need your guidance on, whenever you have some time 🙏🏻 Thanks in advance.

What would be the best way to test all of this? Do I need to write some tests, and if so what do we want to actually test here? I really haven't had an opportunity to figure out testing just yet, so appreciate any help on this.

@ofborg ofborg bot requested a review from flokli February 21, 2023 08:46
@flokli
Copy link
Member

flokli commented Feb 21, 2023

What would be the best way to test all of this? Do I need to write some tests, and if so what do we want to actually test here? I really haven't had an opportunity to figure out testing just yet, so appreciate any help on this.

We have a bunch of tests in nixos/tests, which you can simply run by nix-build'ing them. Maybe a good way to start would be the systemd-initrd-* ones (make sure to confirm they did succeed before). That should give you an idea if there's any major regressions.

@software-artificer
Copy link
Contributor Author

Just a small progress update: I am continuing to work on this, just having some trouble running tests and verifying everything (due to my LXC set-up), so needed to build a VM and a bit of infra to be able to test this properly and will come back with the results at some point soon-ish, time permitting.

@software-artificer
Copy link
Contributor Author

So I've got a system running NixOS now, so I can run the tests, but I am having a problem building the test dependencies to be able to run the tests:

=================================== FAILURES ===================================
_________________________ TestLexers.testIPythonLexer __________________________

self = <IPython.lib.tests.test_lexers.TestLexers testMethod=testIPythonLexer>

    def testIPythonLexer(self):
        fragment = '!echo $HOME\n'
        tokens = [
            (Token.Operator, '!'),
        ]
        tokens.extend(self.bash_lexer.get_tokens(fragment[1:]))
>       self.assertEqual(tokens, list(self.lexer.get_tokens(fragment)))
E       AssertionError: Lists differ: [(Tok[78 chars] (Token.Name.Variable, '$HOME'), (Token.Text.Whitespace, '\n')] != [(Tok[78 chars] (Token.Name.Variable, '$HOME'), (Token.Text, '\n')]
E       
E       First differing element 4:
E       (Token.Text.Whitespace, '\n')
E       (Token.Text, '\n')
E       
E         [(Token.Operator, '!'),
E          (Token.Name.Builtin, 'echo'),
E          (Token.Text.Whitespace, ' '),
E          (Token.Name.Variable, '$HOME'),
E       -  (Token.Text.Whitespace, '\n')]
E       ?             -----------
E       
E       +  (Token.Text, '\n')]

IPython/lib/tests/test_lexers.py:25: AssertionError

Which seems to be a problem between pygments 2.14.0 and ipython <8.8.

How would one normally deal with this? Just disable the offending tests for now as they're completely unrelated to the change? Or do I just cherry-pick my change on top of the current stable channel (so I can use prebuilt dependencies) and test thing that way? Any ideas @flokli?

@flokli
Copy link
Member

flokli commented Mar 1, 2023

Sorry for the delay in replying. I think you've been bitten by another unrelated regression in staging, which might be very unstable sometimes (that's what we use the staging-next stabilization cycles for).

In these cases, it's probably easiest to test your changes by cherry-picking these commits on top of master (pushed to a separate branch, but still keep the PR open as-is), and once you're happy with the results, merge this PR.

@software-artificer software-artificer force-pushed the systemd-optional-deps branch 2 times, most recently from c640c08 to 08585dd Compare March 5, 2023 03:29
Comment on lines -664 to +676
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missed this one initially, which was failing the build. Now fixed.

@github-actions github-actions bot added the 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS label Mar 6, 2023
@software-artificer
Copy link
Contributor Author

Hey @flokli, I think I have reached the point where we can start getting ready to merge this. Do you want me to test or verify anything else before? Or do I make this PR ready for review and we work from there?

@flokli
Copy link
Member

flokli commented Mar 6, 2023

@filakhtov thanks! Yes, this looks pretty good now. I think we should be running some more tests (installer tests, systemd-initrd* tests), to make sure there's no new test failures (some of these might already be broken currently).

If there's no new regressions, fine for me to merge this :-)

Expose a new `withLibidn2` flag (defauts to true for backwards compatibility) to be able to conditionally enable and disable integration with `libidn2`, which is used by the `systemd-network` and `systemd-resolved` to support internationalized domain names.
Expose a new `withAcl` flag (defaults to true for backwards compatibility) to be able to conditionally enable and disable an integration with `libacl` library, which is used by variety of systemd tools and daemon, e.g. `journald` will check ACLs in addition to regular permissions when accessing journal files and `systemd-nspawn` will update ACL entries when used with the `--private-users-chown` flag.
Expose a new `withAudit` flag (defaults to `true` for backwards compatibility) to be able to conditionally enable and disable an integration with the `libaudit` library, which is used to integrate with Linux Audit Framework for logging various security-relevant events.
Expose a new `withPam` option to allow enabling and disabling
integration with PAM stack, including the `systemd-user-sessions` daemon
and the associated `.service` file, as well as `pam_systemd.so` PAM
module for integration with `systemd-logind` and user session
registration with the systemd cgroup hierarchy.
Expose a new `withKmod` option to be able to enable and disable kmod
integration, including the `systemd-modules-load` tool for automatic
modules loading during the system boot sequence.
The `systemdMinimal` build is used purely for Udev and therefore does
not require all the extra dependencies that are needed for normal
operation. As more flags were exposed to allow disabling additional
opional dependencies the `systemdMinimal` will now take advantage of
those.
@github-actions github-actions bot removed the 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS label Mar 8, 2023
@flokli flokli marked this pull request as ready for review March 8, 2023 15:14
@flokli flokli requested a review from a team as a code owner March 8, 2023 15:14
Copy link
Member

@flokli flokli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for the work!

@software-artificer
Copy link
Contributor Author

Hey @flokli thank you very much for bearing with me on this, as well as for your support and guidance! 🙇🏻‍♂️

When PAM was made optional initially, we decided to keep it enabled for stage 1, but as was later pointed out during the code review it is unnecessary, because we never use PAM in stage 1, even in network-enabled stage 1 with OpenSSH we have `UsePAM` set to `no`, so disabling it now.
@ofborg ofborg bot requested a review from flokli March 9, 2023 07:01
@flokli
Copy link
Member

flokli commented Mar 9, 2023

Let's get this in, and deal with any further regressions during the next stabilization cycle. This is probably better tested than some other staging changes ;-)

Thanks a lot for the work!

@flokli flokli merged commit a4c558e into NixOS:staging Mar 9, 2023
@software-artificer software-artificer deleted the systemd-optional-deps branch March 9, 2023 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: systemd Software suite that provides an array of system components for Linux operating systems. 10.rebuild-darwin: 101-500 This PR causes between 101 and 500 packages to rebuild on Darwin. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants