I did some more extensive testing and fixed a couple of mistakes.

• Migration of the lease files
• Migration doesn't fail if /var/db/dhcpcd can't be removed (may be a bind mount)
• sudo can be used in runHook (provided NoNewPrivileges is disabled)
• Filepaths can be added to the sandbox using ReadOnlyPaths and such
• NTP daemon still works without the workaround (tested with chronyd)

Why is the move from /var/db to /var/lib done?

Why is the move from /var/db to /var/lib done?

I initially intended to use DynamicUser, which requires to move all the service state under /var/lib/dhcpcd.
Eventually I had to restore to static id/gid because DynamicUser also forces some sandboxing options that can't be relaxed if necessary (see systemd/systemd#20495). Maybe in the future it will be possible.

Anyway, I guess it still makes sense to use /var/lib as it's the path used by the vast majority of services in NixOS.

@uninsane

NetworkManager defaults to its own internal dhcp client, but does expose a config option to instead use dhcpcd. in nixos, that's:

networking.networkmanager.dhcp = "dhcpcd";

I run nixosTests.networking.networkmanager with networking.networkmanager.dhcp = "dhcpc" and it passed.
In any case, I think it's a pretty uncommon setting.

Fixed merge conflicts.

Fixed more merge conflicts.

I'm reasonably confident with my testing. If no one has any objections I'm going to merge it this weekend.

NoNewPrivileges = true has the effect of restricting CapInh to be <= CapPrm in all of the places in which CapInh is actually used. i.e. try making a setcap binary for one of the bits that are in CapInh but not CapPrm. it won't actually grant those caps, so in effect the two capability sets you show are equivalent, given NoNewPrivileges.

i don't think there's much harm in setting CapabilityBoundingSet, but if you trust that the capability system behaves as described, there's no need.

see: man PR_SET_NO_NEW_PRIVS and man capabilities (in the section for Inheritable)

There are errors in the logs of the dhcpcd service:

(e-dhcpcd)[761]: dhcpcd.service: Failed to set up mount namespacing: /run/resolvconf: No such file or directory
(e-dhcpcd)[761]: dhcpcd.service: Failed at step NAMESPACE spawning /nix/store/yr3ywvhmbcpd6w2hm0jbgj09y7saij6g-migrate-dhcpcd: No such file or directory

and

dhcpcd[910]: /nix/store/waiskclr9v15ws56hb33n82wyixgp6pz-openresolv-3.13.2/sbin/.resolvconf-wrapped: line 945: /run/resolvconf/metrics/0001002 enp1s0.dhcp: Permission denied

I can reproduce it with nixosTests.networking.scripted.dhcpSimple. The first error seems to be a race condition: it looks like systemd is trying to set up the sandbox while resolvconf.service hasn't run yet. I'm not sure about the second error, however /etc/resolv.conf file seems to be populated correctly.

Yes, that test is blocking the nixos-unstable channel now.
https://hydra.nixos.org/build/274843085#tabs-buildsteps

Can we revert and revisit later now that a blocking test case has been identified and no clear path forward has been presented?

I have a fix for the race condition, #348305. The second error does not seem critical.

I'm using dhcpcd with dnsmasq and getting:

dhcpcd[2881]: /nix/store/waiskclr9v15ws56hb33n82wyixgp6pz-openresolv-3.13.2/sbin/.resolvconf-wrapped: line 945: /run/resolvconf/metrics/0001002 enp4s0.dhcp : Permission denied
dhcpcd[2916]: /nix/store/waiskclr9v15ws56hb33n82wyixgp6pz-openresolv-3.13.2/libexec/resolvconf/dnsmasq: line 166: /etc/dnsmasq-conf.conf: Read-only file system
dhcpcd[2916]: /nix/store/waiskclr9v15ws56hb33n82wyixgp6pz-openresolv-3.13.2/libexec/resolvconf/dnsmasq: line 174: /etc/dnsmasq-resolv.conf: Read-only file system

This one is actually breaking because it can't write the dnsmasq config.

I'll try to get it working by porting what it does for resolv.conf to these files.

Something like this maybe?

diff --git a/nixos/modules/config/resolvconf.nix b/nixos/modules/config/resolvconf.nix
index 0d56ccf85d4c..6cdd193a2034 100644
--- a/nixos/modules/config/resolvconf.nix
+++ b/nixos/modules/config/resolvconf.nix
@@ -114,6 +114,14 @@ in
         '';
       };
 
+      outputFiles = lib.mkOption {
+        type = lib.types.listOf lib.types.path;
+        default = [ "/etc/resolv.conf" ];
+        description = ''
+          Files written by resolvconf updates
+        '';
+      };
+
     };
 
   };
@@ -150,8 +158,9 @@ in
 
         script = ''
           ${lib.getExe cfg.package} -u
-          chgrp -R resolvconf /etc/resolv.conf /run/resolvconf
-          chmod -R g=u /etc/resolv.conf /run/resolvconf
+          files=(/run/resolvconf ${lib.escapeShellArgs cfg.outputFiles})
+          chgrp -R resolvconf "''${files[@]}"
+          chmod -R g=u "''${files[@]}"
         '';
       };
 
diff --git a/nixos/modules/services/networking/dhcpcd.nix b/nixos/modules/services/networking/dhcpcd.nix
index afc977583594..71a193fe22ce 100644
--- a/nixos/modules/services/networking/dhcpcd.nix
+++ b/nixos/modules/services/networking/dhcpcd.nix
@@ -250,7 +250,7 @@ in
             Restart = "always";
             AmbientCapabilities = [ "CAP_NET_ADMIN" "CAP_NET_RAW" "CAP_NET_BIND_SERVICE" ];
             ReadWritePaths = [ "/proc/sys/net/ipv6" ]
-              ++ lib.optionals useResolvConf [ "/etc/resolv.conf" "/run/resolvconf" ];
+              ++ lib.optionals useResolvConf ([ "/run/resolvconf" ] ++ config.networking.resolvconf.outputFiles);
             DeviceAllow = "";
             LockPersonality = true;
             MemoryDenyWriteExecute = true;
diff --git a/nixos/modules/services/networking/dnsmasq.nix b/nixos/modules/services/networking/dnsmasq.nix
index 633e37ad25e9..f185f566ee63 100644
--- a/nixos/modules/services/networking/dnsmasq.nix
+++ b/nixos/modules/services/networking/dnsmasq.nix
@@ -133,6 +133,11 @@ in
         dnsmasq_conf=/etc/dnsmasq-conf.conf
         dnsmasq_resolv=/etc/dnsmasq-resolv.conf
       '';
+
+      outputFiles = [
+        "/etc/dnsmasq-conf.conf"
+        "/etc/dnsmasq-resolv.conf"
+      ];
     };
 
     systemd.services.dnsmasq = {

I'm using dhcpcd with dnsmasq and getting:

Can you share your config so I can test this setup?

Can you share your config so I can test this setup?

I think services.dnsmasq.enable = true is sufficient. You want to make sure you're hitting this bit in the dnsmasq module:

    networking.resolvconf = lib.mkIf cfg.resolveLocalQueries {
      useLocalResolver = lib.mkDefault true;

      extraConfig = ''
        dnsmasq_conf=/etc/dnsmasq-conf.conf
        dnsmasq_resolv=/etc/dnsmasq-resolv.conf
      '';

Something like this maybe?

Looks good: I would just name the option subscribersFiles, since that's what openresolv calls them, and maybe make it internal.

SUBSCRIBER OPTIONS
openresolv ships with subscribers for the name servers dnsmasq(8), named(8), pdnsd(8), pdns_recursor(1), and unbound(8).
Each subscriber can create configuration files which should be included in the subscribers main configuration file.

@rnhmjoj sounds good. Would you like me to make a PR?

Yes, please.

@rnhmjoj seems like it also needs"/proc/sys/net/ipv4" added to ReadWritePaths? Without that, my IPV4-only setup fails like so:

Oct 25 16:27:32 fracture systemd[1]: Starting DHCP Client...
Oct 25 16:27:32 fracture dhcpcd[2651]: dhcpcd-10.0.6 starting
Oct 25 16:27:32 fracture dhcpcd[2653]: dev: loaded udev
Oct 25 16:27:32 fracture dhcpcd[2653]: enp1s0: if_init: Read-only file system
Oct 25 16:27:32 fracture dhcpcd[2653]: enp1s0: if_init: Read-only file system
Oct 25 16:27:32 fracture dhcpcd[2653]: no valid interfaces found
Oct 25 16:27:32 fracture dhcpcd[2653]: no valid interfaces found
Oct 25 16:27:48 fracture dhcpcd[2653]: received SIGTERM, stopping
Oct 25 16:27:48 fracture dhcpcd[2653]: dhcpcd exited

Also, even with that added:

Oct 25 16:27:54 fracture dhcpcd[2969]: /nix/store/waiskclr9v15ws56hb33n82wyixgp6pz-openresolv-3.13.2/sbin/.resolvconf-wrapped: line 945: /run/resolvconf/metrics/0001002 enp1s0.dhcp: Permission denied

It appears the metrics sub-directory doesn't have the group and permissions needed to enable this:

shados@fracture[~] λ ls -lah /run/resolvconf
total 0
drwxrwxr-x  4 root resolvconf  80 Oct 25 16:27 .
drwxr-xr-x 33 root root       920 Oct 25 16:27 ..
drwxrwxr-x  2 root resolvconf  80 Oct 25 16:27 interfaces
drwxr-xr-x  2 root root        60 Oct 25 16:25 metrics

seems like it also needs"/proc/sys/net/ipv4" added to ReadWritePaths?

I haven't seen this one before, strange. It looks correct: it appers to be writing to /proc/sys/net/ipv4/conf/%s/promote_secondaries.

Oct 25 16:27:54 fracture dhcpcd[2969]: /nix/store/waiskclr9v15ws56hb33n82wyixgp6pz-openresolv-3.13.2/sbin/.resolvconf-wrapped: line 945: /run/resolvconf/metrics/0001002 enp1s0.dhcp: Permission denied

This one is puzzling, because it's resolvconf itself that is creating the file (if you look at line 945) and the directory is writable. I still haven't figured out whats going on exactly. Anyway, if you have a single interface metrics don't matter.

EDIT: just for my curiosity, does you DHCP server provide more than one IP address?

This one is puzzling, because it's resolvconf itself that is creating the file (if you look at line 945) and the directory is writable.

The /run/resolvconf/metrics directory has root as group, and also isn't group- or other-writeable, and dhcpcd doesn't run as root.

But yes, the metrics issue isn't problematic for my use-cases currently, which is good for me.

The /run/resolvconf/metrics directory has root as group, and also isn't group- or other-writeable, and dhcpcd doesn't run as root.

The source of the incorrect permissions was the network-setup script invoking resolvconf as root. I have found a simple solution using POSIX ACLs, see #351225.

Hello, I don't really understand the reason for this change but this breaks dynamic hostname fetching with DHCP.

If you can describe me a bit your setup and how it fails with hardening, I can try to help.
I guess dhcpcd doesn't have the permission to change the hostname.

Anyway, the reason for this change is that dhcpcd is a demon with root privileges, network access, written in a memory unsafe language, which is quite a combination. It has a built-in security mechanism to mitigate this, but it caused some issues, so it's better to use systemd to isolate it. By default the daemon has limited permissions, which is fine for a majority of users.

I understand but this was utterly frustrating to stumble across because I don't believe my use-case is rare at all (cloud VMs or anything that uses network boot) and I would have had expected more thorough testing of the basic features of a DHCP client after such a breaking change.

Anyway, I managed to fix this with the following:

  systemd.services.dhcpcd = {
    serviceConfig = {
      ProtectHostname = lib.mkForce false;
      SystemCallFilter = lib.mkBefore [ "sethostname" ];
      AmbientCapabilities = [ "CAP_SYS_ADMIN" ];
    };
  };

Let me know if this makes sense to you and I can make a PR.

This took even more time to debug because yet another breaking change happened in #359571 which has a side effect of changing the default hostname from localhost to nixos, and dhcpcd only changes the hostname if it is localhost or localhost.localdomain. So now adding env force_hostname=YES to dhcpcd's config is also needed to fetch the hostname with DHCP. An alternative would be to specify the new default hostname of NixOS in dhcpcd's derivation using the --with-default-hostname configure flag but I'm not sure if it's the best solution.

Sorry, but the basic features are configuring addresses, routes and nameservers, which is what is been tested in NixOS. Your use case may not be rare, but I personally never used it, nor anyone else that maintains NixOS networking tests...

I don't think CAP_SYS_ADMIN is acceptable, at least not by default: it's so overloaded that's basically running as root again.

The host name is being set from a shell hook, so it should be easy to fix both issues:

1. use sudo or polkit to allow running hostname
2. add "nixos" as a default hostname

I'm not sure it's that easy. I'm not very familiar with polkit but from what I understand I don't think we can use it here because NixOS doesn't use systemd-hostnamed. I'm guessing sudo is one way to go but that means we have to patch the hook or wrap hostname.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/insecure-systemd-services-like-networkmanager-running-as-root/66873/11