Make .setDefault() a DaemonOpt

Signed-off-by: Sebastiaan van Stijn <[email protected]>

…lts()

Signed-off-by: Sebastiaan van Stijn <[email protected]>

Signed-off-by: Sebastiaan van Stijn <[email protected]>

When starting containerd as a child process of dockerd, we currently generate
a containerd.toml configuration file with the bare minimum of options we need;

- GRPC Socket address (/var/run/docker/containerd/containerd.sock)
- Root directory (/var/lib/docker/containerd/daemon)
- Exec State directory (/var/run/docker/containerd/daemon)
- Debug socket location (/var/run/docker/containerd/containerd-debug.sock)
- GRPC MaxRecvMsgSize, MaxSendMsgSize
- Disabled plugins (cri is disabled by default)
- OOMScore

While this is convenient, containerd provides many configuration options, which
cannot currently be configured through dockerd. Adding daemon options for all of
those options would be hard to maintain, and not something we want to implement.

This patch adds support for a user-provided containerd.toml configuration file
to use instead of the automatically generated configuration file. With this patch
the startup looks like this;

1. if `--containerd` is set, the daemon expects containerd to be a running
   service. No managed instance is started, and the daemon will attempt to
   connect to the given address.
2. The daemon will attempt to _detect_ if containerd is running as a service at
   `/run/containerd/containerd.sock`, or `$XDG_RUNTIME_DIR/containerd/containerd.sock`
   (when running in rootless mode).
3. The daemon checks if a `/etc/docker/containerd.toml` file is present and
   contains a GRPC Socket address. If so, it starts containerd with the given
   configuration file, and connects using the socket specified in it.
4. The daemon generates a containerd config file, and saves it in a temporary
   location; `/var/run/docker/containerd/containerd.toml` (default), or
   `$XDG_RUNTIME_DIR/containerd/containerd.toml` (rootless)

Notes/Caveats:
----------------------------------------

There's a couple of caveats worth mentioning;

- The daemon performs no validation of the configuration file, other than to
  verify that a GRPC address is present. It is up to the user to set the correct
  configuration options; including specifying the correct location when running
  dockerd in rootless mode.
- containerd's default configuration uses the same location for the socket as
  a containerd instance that's running as a (systemd) service. This can cause
  issues in some situations;
  - if the socket is not cleaned up when containerd exits, the daemon will fail
    to start, as it assumes containerd is already running as a service.
  - if dockerd is started _before_ a containerd service is started, the socket
    will conflict (potentially blocking the containerd service from being
    started).

We can consider producing a warning for the second case, or decide to refuse
the configuration (but that would likely be too restrictive).

How to test this feature
----------------------------------------

Create the `/etc/docker` configuration directory if it doesn't exist, and create
a valid `containerd.toml` file in it (the example below uses containerd's default
configuration);

```bash
mkdir -p /etc/docker
containerd config default > /etc/docker/containerd.toml
```

Start the daemon and verify that it picks up the configuration;

```
dockerd --debug

INFO[2022-08-10T15:30:42.266201214Z] Starting up
INFO[2022-08-10T15:30:42.267341643Z] containerd not running, starting managed containerd
INFO[2022-08-10T15:30:42.311542635Z] using pre-existing containerd configuration file: /etc/docker/containerd.toml
INFO[2022-08-10T15:30:42.313421457Z] libcontainerd: started new containerd process  pid=8542
INFO[2022-08-10T15:30:42.313501681Z] [core] original dial target is: "unix:///run/containerd/containerd.sock"  module=grpc
INFO[2022-08-10T15:30:42.313727993Z] [core] parsed dial target is: {Scheme:unix Authority: Endpoint:run/containerd/containerd.sock URL:{Scheme:unix Opaque: User: Host: Path:/run/containerd/containerd.sock RawPath: ForceQuery:false RawQuery: Fragment: RawFragment:}}  module=grpc
INFO[2022-08-10T15:30:42.313775044Z] [core] Channel authority set to "localhost"   module=grpc
INFO[2022-08-10T15:30:42.313851206Z] [core] ccResolverWrapper: sending update to cc: {[{/run/containerd/containerd.sock  0xc00003a3f0 <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-08-10T15:30:42.313860948Z] [core] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-08-10T15:30:42.313865563Z] [core] Channel switches to new LB policy "pick_first"  module=grpc
INFO[2022-08-10T15:30:42.313979933Z] [core] Subchannel Connectivity change to CONNECTING  module=grpc
INFO[2022-08-10T15:30:42.314101917Z] [core] Subchannel picks a new address "/run/containerd/containerd.sock" to connect  module=grpc
INFO[2022-08-10T15:30:42.314260534Z] [core] Channel Connectivity change to CONNECTING  module=grpc
INFO[2022-08-10T15:30:42.352557205Z] starting containerd                           revision=0197261a30bf81f1ee8e6a4dd2dea0ef95d67ccb version=v1.6.7
...
INFO[2022-08-10T15:30:43.358542169Z] API listen on /var/run/docker.sock
```

Once running, verify that containerd is started with the expected configuration
file and logging level (debug in this case);

```bash
ps auxf
...
root      1147  2.2  1.3 1422952 56236 pts/0   Sl+  16:08   0:00 dockerd --debug
root      1154  1.3  0.8 1407144 33952 ?       Ssl  16:08   0:00  \_ containerd --config /etc/docker/containerd.toml --log-level debug
```

Unlike docker's own configuration for a managed containerd instance, containerd's
default configuration uses `/var/lib/containerd` for data, and `/run/containerd/`
as runtime directory;

```bash
tree /var/lib/containerd
/var/lib/containerd
|-- io.containerd.content.v1.content
|   `-- ingest
|-- io.containerd.metadata.v1.bolt
|   `-- meta.db
|-- io.containerd.runtime.v1.linux
|-- io.containerd.runtime.v2.task
|   `-- moby
|       `-- f3061cb7133927a8bb43279a7e437cd895d210a6a73dff55d94ea358e735f153
|-- io.containerd.snapshotter.v1.btrfs
|-- io.containerd.snapshotter.v1.native
|   `-- snapshots
|-- io.containerd.snapshotter.v1.overlayfs
|   `-- snapshots
`-- tmpmounts

13 directories, 1 file
```

```bash
tree /run/containerd/
/run/containerd/
|-- containerd.sock
|-- containerd.sock.ttrpc
|-- io.containerd.runtime.v1.linux
|-- io.containerd.runtime.v2.task
|   `-- moby
|       `-- f3061cb7133927a8bb43279a7e437cd895d210a6a73dff55d94ea358e735f153
|           |-- address
|           |-- config.json
|           |-- init.pid
|           |-- log
|           |-- log.json
|           |-- options.json
|           |-- rootfs
|           |-- runtime
|           |-- shim-binary-path
|           `-- work -> /var/lib/containerd/io.containerd.runtime.v2.task/moby/f3061cb7133927a8bb43279a7e437cd895d210a6a73dff55d94ea358e735f153
`-- s
    `-- 77e1863a75ec4e750b1471bc638859398f17065e321e50b3a42ccef16bdc1a64

7 directories, 11 files
```

Signed-off-by: Sebastiaan van Stijn <[email protected]>