-
Notifications
You must be signed in to change notification settings - Fork 3.9k
Expand file tree
/
Copy pathconfig.md
More file actions
798 lines (660 loc) · 35.6 KB
/
config.md
File metadata and controls
798 lines (660 loc) · 35.6 KB
Edit and raw actions
OlderNewer
1
# CRI Plugin Config Guide
2
This document provides the description of the CRI plugin configuration.
3
The CRI plugin config is part of the containerd config (default
4
path: `/etc/containerd/config.toml`).
5
6
See [here](https://github.com/containerd/containerd/blob/main/docs/ops.md)
7
for more information about containerd config.
8
9
Note that the `[plugins."io.containerd.grpc.v1.cri"]` section is specific to CRI,
10
and not recognized by other containerd clients such as `ctr`, `nerdctl`, and Docker/Moby.
11
12
## Config versions
13
The content of `/etc/containerd/config.toml` must start with a version header, for example:
14
```toml
15
version = 3
16
```
17
18
The config version 3 was introduced in containerd v2.0.
19
The config version 2 used in containerd 1.x is still supported and automatically
20
converted to the config version 3.
21
22
For the further information, see [`../PLUGINS.md`](../PLUGINS.md).
23
24
## Basic configuration
25
### Cgroup Driver
26
While containerd and Kubernetes use the legacy `cgroupfs` driver for managing cgroups by default,
27
it is recommended to use the `systemd` driver on systemd-based hosts for compliance of
28
[the "single-writer" rule](https://systemd.io/CGROUP_DELEGATION/) of cgroups.
29
30
To configure containerd to use the `systemd` driver, set the following option in `/etc/containerd/config.toml`:
31
+ In containerd 2.x
32
```toml
33
version = 3
34
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
35
SystemdCgroup = true
36
```
37
+ In containerd 1.x
38
```toml
39
version = 2
40
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
41
SystemdCgroup = true
42
```
43
44
In addition to containerd, you have to configure the `KubeletConfiguration` to use the "systemd" cgroup driver.
45
The `KubeletConfiguration` is typically located at `/var/lib/kubelet/config.yaml`:
46
```yaml
47
kind: KubeletConfiguration
48
apiVersion: kubelet.config.k8s.io/v1beta1
49
cgroupDriver: "systemd"
50
```
51
52
kubeadm users should also see [the kubeadm documentation](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/).
53
54
> Note: Kubernetes v1.28 supports automatic detection of the cgroup driver as
55
> an alpha feature. With the `KubeletCgroupDriverFromCRI` kubelet feature gate
56
> enabled, the kubelet automatically detects the cgroup driver from the CRI
57
> runtime and the `KubeletConfiguration` configuration step above is not
58
> needed.
59
>
60
> When determining the cgroup driver, containerd uses the `SystemdCgroup`
61
> setting from runc-based runtime classes, starting from the default runtime
62
> class. If no runc-based runtime classes have been configured containerd
63
> relies on auto-detection based on determining if systemd is running.
64
> Note that all runc-based runtime classes should be configured to have the
65
> same `SystemdCgroup` setting in order to avoid unexpected behavior.
66
>
67
> The automatic cgroup driver configuration for kubelet feature is supported in
68
> containerd v2.0 and later.
69
70
### Snapshotter
71
72
The default snapshotter is set to `overlayfs` (akin to Docker's `overlay2` storage driver):
73
+ In containerd 2.x
74
```toml
75
version = 3
76
[plugins.'io.containerd.cri.v1.images']
77
snapshotter = "overlayfs"
78
```
79
+ In containerd 1.x
80
```toml
81
version = 2
82
[plugins."io.containerd.grpc.v1.cri".containerd]
83
snapshotter = "overlayfs"
84
```
85
86
See [here](https://github.com/containerd/containerd/blob/main/docs/snapshotters) for other supported snapshotters.
87
88
### Runtime classes
89
90
The following example registers custom runtimes into containerd:
91
+ In containerd 2.x
92
```toml
93
version = 3
94
[plugins."io.containerd.cri.v1.runtime".containerd]
95
default_runtime_name = "crun"
96
[plugins."io.containerd.cri.v1.runtime".containerd.runtimes]
97
# crun: https://github.com/containers/crun
98
[plugins."io.containerd.cri.v1.runtime".containerd.runtimes.crun]
99
runtime_type = "io.containerd.runc.v2"
100
[plugins."io.containerd.cri.v1.runtime".containerd.runtimes.crun.options]
101
BinaryName = "/usr/local/bin/crun"
102
# gVisor: https://gvisor.dev/
103
[plugins."io.containerd.cri.v1.runtime".containerd.runtimes.gvisor]
104
runtime_type = "io.containerd.runsc.v1"
105
# Kata Containers: https://katacontainers.io/
106
[plugins."io.containerd.cri.v1.runtime".containerd.runtimes.kata]
107
runtime_type = "io.containerd.kata.v2"
108
```
109
+ In containerd 1.x
110
```toml
111
version = 2
112
[plugins."io.containerd.grpc.v1.cri".containerd]
113
default_runtime_name = "crun"
114
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
115
# crun: https://github.com/containers/crun
116
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.crun]
117
runtime_type = "io.containerd.runc.v2"
118
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.crun.options]
119
BinaryName = "/usr/local/bin/crun"
120
# gVisor: https://gvisor.dev/
121
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.gvisor]
122
runtime_type = "io.containerd.runsc.v1"
123
# Kata Containers: https://katacontainers.io/
124
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
125
runtime_type = "io.containerd.kata.v2"
126
```
127
128
In addition, you have to install the following `RuntimeClass` resources into the cluster
129
with the `cluster-admin` role:
130
131
```yaml
132
apiVersion: node.k8s.io/v1
133
kind: RuntimeClass
134
metadata:
135
name: crun
136
handler: crun
137
---
138
apiVersion: node.k8s.io/v1
139
kind: RuntimeClass
140
metadata:
141
name: gvisor
142
handler: gvisor
143
---
144
apiVersion: node.k8s.io/v1
145
kind: RuntimeClass
146
metadata:
147
name: kata
148
handler: kata
149
```
150
151
To apply a runtime class to a pod, set `.spec.runtimeClassName`:
152
153
```yaml
154
apiVersion: v1
155
kind: Pod
156
spec:
157
runtimeClassName: crun
158
```
159
160
See also [the Kubernetes documentation](https://kubernetes.io/docs/concepts/containers/runtime-class/).
161
162
163
## Image Pull Configuration (since containerd v2.1)
164
165
### Transfer Service for Image Pull
166
167
Starting with containerd v2.1, the CRI plugin uses containerd's Transfer Service for image pull by default, instead of client-based pull.
168
169
To configure Transfer Service, use the following settings in your config.toml:
170
171
```toml
172
[plugins.'io.containerd.transfer.v1.local']
173
# Transfer service specific configurations
174
max_concurrent_downloads = 3
175
unpack_config = { ... }
176
```
177
178
### Local Pull Mode
179
180
If you prefer to use the client-based pull method instead of the Transfer Service, you can set `use_local_image_pull = true` in your CRI image configuration:
181
182
```toml
183
[plugins.'io.containerd.cri.v1.images']
184
use_local_image_pull = true
185
```
186
187
### Configuration differences and automatic fallback to Local Mode
188
189
There are some differences in how image pull configurations are specified between the Transfer Service and Local Pull mode:
190
191
| CRI Image Config Option | Local Pull | Transfer Service Pull |
192
|------------------------|------------|---------------------|
193
| Snapshotter | ✅ Supported | ✅ Supported |
194
| DisableSnapshotAnnotations | ✅ Supported | ⚠️ Must be configured in snapshotter plugin:<br>`[proxy_plugins.stargz.exports]`<br>`enable_remote_snapshot_annotations = "true"` |
195
| ImagePullProgressTimeout | ✅ Supported | ✅ Supported |
196
| DiscardUnpackedLayers | ✅ Supported | ❌ Not Supported |
197
| PinnedImages | ✅ Supported | ✅ Supported |
198
| Registry Settings | ✅ All supported | ⚠️ Only ConfigPath and Headers supported<br>(Mirrors, Configs, Auths not supported, also deprecated) |
199
| ImageDecryption | ❌ Disabled | ❌ Disabled |
200
| MaxConcurrentDownloads | ✅ Uses CRI Image config | ⚠️ Must be configured in transfer service plugin: `plugins."io.containerd.transfer.v1.local"` |
201
| ImagePullWithSyncFs | ✅ Supported | ❌ Not Supported |
202
| StatsCollectPeriod | ✅ Supported | ✅ Supported |
203
204
To ensure compatibility, ***containerd 2.1 automatically detects configuration conflicts and falls back to local image pull mode when necessary***.
205
206
If you have any of the following configurations in your CRI image config, containerd will automatically set `use_local_image_pull = true` and log a warning:
207
208
- `DisableSnapshotAnnotations = false`
209
- `DiscardUnpackedLayers = true`
210
- `Registry.Mirrors` is configured
211
- `Registry.Configs` is configured
212
- `Registry.Auths` is configured
213
- `MaxConcurrentDownloads != 3`
214
- `ImagePullWithSyncFs = true`
215
216
The warning message will indicate which configuration option triggered the fallback and provide guidance on how to properly configure the option when using the Transfer Service.
217
218
## Full configuration
219
The explanation and default value of each configuration item are as follows:
220
+ In containerd 2.x
221
<details>
222
223
<p>
224
225
```toml
226
# containerd has several configuration versions:
227
# - Version 3 (Recommended for containerd 2.x): Introduced in containerd 2.0.
228
# Several plugin IDs have changed in this version.
229
# - Version 2 (Recommended for containerd 1.x): Introduced in containerd 1.3.
230
# Still supported in containerd v2.x.
231
# Plugin IDs are changed to have prefixes like "io.containerd.".
232
# - Version 1 (Default): Introduced in containerd 1.0. Removed in containerd 2.0.
233
version = 3
234
235
[plugins]
236
[plugins.'io.containerd.cri.v1.images']
237
snapshotter = 'overlayfs'
238
disable_snapshot_annotations = true
239
discard_unpacked_layers = false
240
max_concurrent_downloads = 3
241
image_pull_progress_timeout = '5m0s'
242
image_pull_with_sync_fs = false
243
stats_collect_period = 10
244
use_local_image_pull = false
245
246
[plugins.'io.containerd.cri.v1.images'.pinned_images]
247
sandbox = 'registry.k8s.io/pause:3.10.2'
248
249
[plugins.'io.containerd.cri.v1.images'.registry]
250
config_path = ''
251
252
[plugins.'io.containerd.cri.v1.images'.image_decryption]
253
key_model = 'node'
254
255
[plugins.'io.containerd.cri.v1.runtime']
256
enable_selinux = false
257
selinux_category_range = 1024
258
max_container_log_line_size = 16384
259
disable_cgroup = false
260
disable_apparmor = false
261
restrict_oom_score_adj = false
262
disable_proc_mount = false
263
unset_seccomp_profile = ''
264
tolerate_missing_hugetlb_controller = true
265
disable_hugetlb_controller = true
266
device_ownership_from_security_context = false
267
ignore_image_defined_volumes = false
268
netns_mounts_under_state_dir = false
269
enable_unprivileged_ports = true
270
enable_unprivileged_icmp = true
271
enable_cdi = true
272
cdi_spec_dirs = ['/etc/cdi', '/var/run/cdi']
273
drain_exec_sync_io_timeout = '0s'
274
ignore_deprecation_warnings = []
275
stats_collect_period = '1s'
276
stats_retention_period = '2m'
277
278
[plugins.'io.containerd.cri.v1.runtime'.containerd]
279
default_runtime_name = 'runc'
280
ignore_blockio_not_enabled_errors = false
281
ignore_rdt_not_enabled_errors = false
282
283
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes]
284
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
285
runtime_type = 'io.containerd.runc.v2'
286
runtime_path = ''
287
pod_annotations = []
288
container_annotations = []
289
privileged_without_host_devices = false
290
privileged_without_host_devices_all_devices_allowed = false
291
cgroup_writable = false
292
base_runtime_spec = ''
293
cni_conf_dir = ''
294
cni_max_conf_num = 0
295
snapshotter = ''
296
sandboxer = 'podsandbox'
297
io_type = ''
298
299
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
300
BinaryName = ''
301
CriuImagePath = ''
302
CriuWorkPath = ''
303
IoGid = 0
304
IoUid = 0
305
NoNewKeyring = false
306
Root = ''
307
ShimCgroup = ''
308
309
[plugins.'io.containerd.cri.v1.runtime'.cni]
310
# DEPRECATED, use `bin_dirs` instead (since containerd v2.1).
311
bin_dir = ''
312
bin_dirs = ['/opt/cni/bin']
313
conf_dir = '/etc/cni/net.d'
314
max_conf_num = 1
315
setup_serially = false
316
conf_template = ''
317
ip_pref = ''
318
use_internal_loopback = false
319
320
[plugins.'io.containerd.grpc.v1.cri']
321
disable_tcp_service = true
322
stream_server_address = '127.0.0.1'
323
stream_server_port = '0'
324
stream_idle_timeout = '4h0m0s'
325
enable_tls_streaming = false
326
327
[plugins.'io.containerd.grpc.v1.cri'.x509_key_pair_streaming]
328
tls_cert_file = ''
329
tls_key_file = ''
330
```
331
332
</p>
333
</details>
334
335
+ In containerd 1.x
336
<details>
337
338
<p>
339
340
```toml
341
# containerd has several configuration versions:
342
# - Version 3 (Recommended for containerd 2.x): Introduced in containerd 2.0.
343
# Several plugin IDs have changed in this version.
344
# - Version 2 (Recommended for containerd 1.x): Introduced in containerd 1.3.
345
# Still supported in containerd v2.x.
346
# Plugin IDs are changed to have prefixes like "io.containerd.".
347
# - Version 1 (Default): Introduced in containerd 1.0. Removed in containerd 2.0.
348
version = 2
349
350
# The 'plugins."io.containerd.grpc.v1.cri"' table contains all of the server options.
351
[plugins."io.containerd.grpc.v1.cri"]
352
353
# disable_tcp_service disables serving CRI on the TCP server.
354
# Note that a TCP server is enabled for containerd if TCPAddress is set in section [grpc].
355
disable_tcp_service = true
356
357
# stream_server_address is the ip address streaming server is listening on.
358
stream_server_address = "127.0.0.1"
359
360
# stream_server_port is the port streaming server is listening on.
361
stream_server_port = "0"
362
363
# stream_idle_timeout is the maximum time a streaming connection can be
364
# idle before the connection is automatically closed.
365
# The string is in the golang duration format, see:
366
# https://golang.org/pkg/time/#ParseDuration
367
stream_idle_timeout = "4h"
368
369
# enable_selinux indicates to enable the selinux support.
370
enable_selinux = false
371
372
# selinux_category_range allows the upper bound on the category range to be set.
373
# if not specified or set to 0, defaults to 1024 from the selinux package.
374
selinux_category_range = 1024
375
376
# sandbox_image is the image used by sandbox container.
377
sandbox_image = "registry.k8s.io/pause:3.10.2"
378
379
# stats_collect_period is the period (in seconds) of snapshots stats collection.
380
stats_collect_period = 10
381
382
# enable_tls_streaming enables the TLS streaming support.
383
# It generates a self-sign certificate unless the following x509_key_pair_streaming are both set.
384
enable_tls_streaming = false
385
386
# tolerate_missing_hugetlb_controller if set to false will error out on create/update
387
# container requests with huge page limits if the cgroup controller for hugepages is not present.
388
# This helps with supporting Kubernetes <=1.18 out of the box. (default is `true`)
389
tolerate_missing_hugetlb_controller = true
390
391
# ignore_image_defined_volumes ignores volumes defined by the image. Useful for better resource
392
# isolation, security and early detection of issues in the mount configuration when using
393
# ReadOnlyRootFilesystem since containers won't silently mount a temporary volume.
394
ignore_image_defined_volumes = false
395
396
# netns_mounts_under_state_dir places all mounts for network namespaces under StateDir/netns
397
# instead of being placed under the hardcoded directory /var/run/netns. Changing this setting
398
# requires that all containers are deleted.
399
netns_mounts_under_state_dir = false
400
401
# max_container_log_line_size is the maximum log line size in bytes for a container.
402
# Log line longer than the limit will be split into multiple lines. -1 means no
403
# limit.
404
max_container_log_line_size = 16384
405
406
# disable_cgroup indicates to disable the cgroup support.
407
# This is useful when the daemon does not have permission to access cgroup.
408
disable_cgroup = false
409
410
# disable_apparmor indicates to disable the apparmor support.
411
# This is useful when the daemon does not have permission to access apparmor.
412
disable_apparmor = false
413
414
# restrict_oom_score_adj indicates to limit the lower bound of OOMScoreAdj to
415
# the containerd's current OOMScoreAdj.
416
# This is useful when the containerd does not have permission to decrease OOMScoreAdj.
417
restrict_oom_score_adj = false
418
419
# max_concurrent_downloads restricts the number of concurrent downloads for each image.
420
max_concurrent_downloads = 3
421
422
# disable_proc_mount disables Kubernetes ProcMount support. This MUST be set to `true`
423
# when using containerd with Kubernetes <=1.11.
424
disable_proc_mount = false
425
426
# unset_seccomp_profile is the seccomp profile containerd/cri will use if the seccomp
427
# profile requested over CRI is unset (or nil) for a pod/container (otherwise if this field is not set the
428
# default unset profile will map to `unconfined`)
429
# Note: The default unset seccomp profile should not be confused with the seccomp profile
430
# used in CRI when the runtime default seccomp profile is requested. In the later case, the
431
# default is set by the following code (https://github.com/containerd/containerd/blob/main/contrib/seccomp/seccomp_default.go).
432
# To summarize, there are two different seccomp defaults, the unset default used when the CRI request is
433
# set to nil or `unconfined`, and the default used when the runtime default seccomp profile is requested.
434
unset_seccomp_profile = ""
435
436
# enable_unprivileged_ports configures net.ipv4.ip_unprivileged_port_start=0
437
# for all containers which are not using host network
438
# and if it is not overwritten by PodSandboxConfig
439
# Note that before containerd v2.0, this value defaulted to false.
440
# [k8s discussion](https://github.com/kubernetes/kubernetes/issues/102612)
441
enable_unprivileged_ports = true
442
443
# enable_unprivileged_icmp configures net.ipv4.ping_group_range="0 2147483647"
444
# for all containers which are not using host network, are not running in user namespace
445
# and if it is not overwritten by PodSandboxConfig
446
# Note that before containerd v2.0, this value defaulted to false.
447
enable_unprivileged_icmp = true
448
449
# enable_cdi enables support of the Container Device Interface (CDI)
450
# For more details about CDI and the syntax of CDI Spec files please refer to
451
# https://tags.cncf.io/container-device-interface.
452
# TODO: Deprecate this option when either Dynamic Resource Allocation(DRA)
453
# or CDI support for the Device Plugins are graduated to GA.
454
# `Dynamic Resource Allocation` KEP:
455
# https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/3063-dynamic-resource-allocation
456
# `Add CDI devices to device plugin API` KEP:
457
# https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/4009-add-cdi-devices-to-device-plugin-api
458
enable_cdi = true
459
460
# cdi_spec_dirs is the list of directories to scan for CDI spec files
461
# For more details about CDI configuration please refer to
462
# https://tags.cncf.io/container-device-interface#containerd-configuration
463
cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"]
464
465
# drain_exec_sync_io_timeout is the maximum duration to wait for ExecSync API'
466
# IO EOF event after exec init process exits. A zero value means there is no
467
# timeout.
468
#
469
# The string is in the golang duration format, see:
470
# https://golang.org/pkg/time/#ParseDuration
471
#
472
# For example, the value can be '5h', '2h30m', '10s'.
473
drain_exec_sync_io_timeout = "0s"
474
475
# 'plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming' contains a x509 valid key pair to stream with tls.
476
[plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
477
# tls_cert_file is the filepath to the certificate paired with the "tls_key_file"
478
tls_cert_file = ""
479
480
# tls_key_file is the filepath to the private key paired with the "tls_cert_file"
481
tls_key_file = ""
482
483
# 'plugins."io.containerd.grpc.v1.cri".containerd' contains config related to containerd
484
[plugins."io.containerd.grpc.v1.cri".containerd]
485
486
# snapshotter is the default snapshotter used by containerd
487
# for all runtimes, if not overridden by an experimental runtime's snapshotter config.
488
snapshotter = "overlayfs"
489
490
# no_pivot disables pivot-root (linux only), required when running a container in a RamDisk with runc.
491
# This only works for runtime type "io.containerd.runtime.v1.linux".
492
no_pivot = false
493
494
# disable_snapshot_annotations disables to pass additional annotations (image
495
# related information) to snapshotters. These annotations are required by
496
# stargz snapshotter (https://github.com/containerd/stargz-snapshotter)
497
# changed to default true with https://github.com/containerd/containerd/pull/4665 and subsequent service refreshes.
498
disable_snapshot_annotations = true
499
500
# discard_unpacked_layers allows GC to remove layers from the content store after
501
# successfully unpacking these layers to the snapshotter.
502
discard_unpacked_layers = false
503
504
# default_runtime_name is the default runtime name to use.
505
default_runtime_name = "runc"
506
507
# ignore_blockio_not_enabled_errors disables blockio related
508
# errors when blockio support has not been enabled. By default,
509
# trying to set the blockio class of a container via annotations
510
# produces an error if blockio hasn't been enabled. This config
511
# option practically enables a "soft" mode for blockio where these
512
# errors are ignored and the container gets no blockio class.
513
ignore_blockio_not_enabled_errors = false
514
515
# ignore_rdt_not_enabled_errors disables RDT related errors when RDT
516
# support has not been enabled. Intel RDT is a technology for cache and
517
# memory bandwidth management. By default, trying to set the RDT class of
518
# a container via annotations produces an error if RDT hasn't been enabled.
519
# This config option practically enables a "soft" mode for RDT where these
520
# errors are ignored and the container gets no RDT class.
521
ignore_rdt_not_enabled_errors = false
522
523
# 'plugins."io.containerd.grpc.v1.cri".containerd.default_runtime' is the runtime to use in containerd.
524
# DEPRECATED: use `default_runtime_name` and `plugins."io.containerd.grpc.v1.cri".containerd.runtimes` instead.
525
[plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
526
527
# 'plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime' is a runtime to run untrusted workloads on it.
528
# DEPRECATED: use `untrusted` runtime in `plugins."io.containerd.grpc.v1.cri".containerd.runtimes` instead.
529
[plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
530
531
# 'plugins."io.containerd.grpc.v1.cri".containerd.runtimes' is a map from CRI RuntimeHandler strings, which specify types
532
# of runtime configurations, to the matching configurations.
533
# In this example, 'runc' is the RuntimeHandler string to match.
534
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
535
# runtime_type is the runtime type to use in containerd.
536
# The default value is "io.containerd.runc.v2" since containerd 1.4.
537
# The default value was "io.containerd.runc.v1" in containerd 1.3, "io.containerd.runtime.v1.linux" in prior releases.
538
runtime_type = "io.containerd.runc.v2"
539
540
# runtime_path is an optional field that can be used to overwrite path to a shim runtime binary.
541
# When specified, containerd will ignore runtime name field when resolving shim location.
542
# Path must be abs.
543
runtime_path = ""
544
545
# pod_annotations is a list of pod annotations passed to both pod
546
# sandbox as well as container OCI annotations. Pod_annotations also
547
# supports golang path match pattern - https://golang.org/pkg/path/#Match.
548
# e.g. ["runc.com.*"], ["*.runc.com"], ["runc.com/*"].
549
#
550
# For the naming convention of annotation keys, please reference:
551
# * Kubernetes: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set
552
# * OCI: https://github.com/opencontainers/image-spec/blob/main/annotations.md
553
pod_annotations = []
554
555
# container_annotations is a list of container annotations passed through to the OCI config of the containers.
556
# Container annotations in CRI are usually generated by other Kubernetes node components (i.e., not users).
557
# Currently, only device plugins populate the annotations.
558
container_annotations = []
559
560
# privileged_without_host_devices allows overloading the default behaviour of passing host
561
# devices through to privileged containers. This is useful when using a runtime where it does
562
# not make sense to pass host devices to the container when privileged. Defaults to false -
563
# i.e pass host devices through to privileged containers.
564
privileged_without_host_devices = false
565
566
# privileged_without_host_devices_all_devices_allowed allows the allowlisting of all devices when
567
# privileged_without_host_devices is enabled.
568
# In plain privileged mode all host device nodes are added to the container's spec and all devices
569
# are put in the container's device allowlist. This flags is for the modification of the privileged_without_host_devices
570
# option so that even when no host devices are implicitly added to the container, all devices allowlisting is still enabled.
571
# Requires privileged_without_host_devices to be enabled. Defaults to false.
572
privileged_without_host_devices_all_devices_allowed = false
573
574
# cgroup_writable field enables the support for writable cgroups in unprivileged containers with cgroup v2 enabled. When disabled, the cgroup interface (/sys/fs/cgroup) is mounted as read-only, preventing containers from managing their own cgroup hierarchies.
575
cgroup_writable = false
576
577
# base_runtime_spec is a file path to a JSON file with the OCI spec that will be used as the base spec that all
578
# container's are created from.
579
# Use containerd's `ctr oci spec > /etc/containerd/cri-base.json` to output initial spec file.
580
# Spec files are loaded at launch, so containerd daemon must be restarted on any changes to refresh default specs.
581
# Still running containers and restarted containers will still be using the original spec from which that container was created.
582
base_runtime_spec = ""
583
584
# conf_dir is the directory in which the admin places a CNI conf.
585
# this allows a different CNI conf for the network stack when a different runtime is being used.
586
cni_conf_dir = "/etc/cni/net.d"
587
588
# cni_max_conf_num specifies the maximum number of CNI plugin config files to
589
# load from the CNI config directory. By default, only 1 CNI plugin config
590
# file will be loaded. If you want to load multiple CNI plugin config files
591
# set max_conf_num to the number desired. Setting cni_max_config_num to 0 is
592
# interpreted as no limit is desired and will result in all CNI plugin
593
# config files being loaded from the CNI config directory.
594
cni_max_conf_num = 1
595
596
# snapshotter overrides the global default snapshotter to a runtime specific value.
597
# Please be aware that overriding the default snapshotter on a runtime basis is currently an experimental feature.
598
# See https://github.com/containerd/containerd/issues/6657 for context.
599
snapshotter = ""
600
601
# sandboxer is the sandbox controller for the runtime.
602
# The default sandbox controller is the podsandbox controller, which create a "pause" container as a sandbox.
603
# We can create our own "shim" sandbox controller by implementing the sandbox api defined in runtime/sandbox/v1/sandbox.proto in our shim, and specifiy the sandboxer to "shim" here.
604
# We can also run a grpc or ttrpc server to serve the sandbox controller API defined in services/sandbox/v1/sandbox.proto, and define a ProxyPlugin of "sandbox" type, and specify the name of the ProxyPlugin here.
605
sandboxer = ""
606
607
# io_type is the way containerd get stdin/stdout/stderr from container or the execed process.
608
# The default value is "fifo", in which containerd will create a set of named pipes and transfer io by them.
609
# Currently the value of "streaming" is supported, in this way, sandbox should serve streaming api defined in services/streaming/v1/streaming.proto, and containerd will connect to sandbox's endpoint and create a set of streams to it, as channels to transfer io of container or process.
610
io_type = ""
611
612
# 'plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options' is options specific to
613
# "io.containerd.runc.v1" and "io.containerd.runc.v2". Its corresponding options type is:
614
# https://github.com/containerd/containerd/blob/v1.3.2/runtime/v2/runc/options/oci.pb.go#L26 .
615
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
616
# NoPivotRoot disables pivot root when creating a container.
617
NoPivotRoot = false
618
619
# NoNewKeyring disables new keyring for the container.
620
NoNewKeyring = false
621
622
# ShimCgroup places the shim in a cgroup.
623
ShimCgroup = ""
624
625
# IoUid sets the I/O's pipes uid.
626
IoUid = 0
627
628
# IoGid sets the I/O's pipes gid.
629
IoGid = 0
630
631
# BinaryName is the binary name of the runc binary.
632
BinaryName = ""
633
634
# Root is the runc root directory.
635
Root = ""
636
637
# SystemdCgroup enables systemd cgroups.
638
SystemdCgroup = false
639
640
# CriuImagePath is the criu image path
641
CriuImagePath = ""
642
643
# CriuWorkPath is the criu work path.
644
CriuWorkPath = ""
645
646
# 'plugins."io.containerd.grpc.v1.cri".cni' contains config related to cni
647
[plugins."io.containerd.grpc.v1.cri".cni]
648
# bin_dir is the directory in which the binaries for the plugin is kept.
649
bin_dir = "/opt/cni/bin"
650
651
# conf_dir is the directory in which the admin places a CNI conf.
652
conf_dir = "/etc/cni/net.d"
653
654
# max_conf_num specifies the maximum number of CNI plugin config files to
655
# load from the CNI config directory. By default, only 1 CNI plugin config
656
# file will be loaded. If you want to load multiple CNI plugin config files
657
# set max_conf_num to the number desired. Setting max_config_num to 0 is
658
# interpreted as no limit is desired and will result in all CNI plugin
659
# config files being loaded from the CNI config directory.
660
max_conf_num = 1
661
662
# conf_template is the file path of golang template used to generate
663
# cni config.
664
# If this is set, containerd will generate a cni config file from the
665
# template. Otherwise, containerd will wait for the system admin or cni
666
# daemon to drop the config file into the conf_dir.
667
# See the "CNI Config Template" section for more details.
668
conf_template = ""
669
# ip_pref specifies the strategy to use when selecting the main IP address for a pod.
670
# options include:
671
# * ipv4, "" - (default) select the first ipv4 address
672
# * ipv6 - select the first ipv6 address
673
# * cni - use the order returned by the CNI plugins, returning the first IP address from the results
674
ip_pref = "ipv4"
675
# use_internal_loopback specifies if we use the CNI loopback plugin or internal mechanism to set lo to up
676
use_internal_loopback = false
677
678
# 'plugins."io.containerd.grpc.v1.cri".image_decryption' contains config related
679
# to handling decryption of encrypted container images.
680
[plugins."io.containerd.grpc.v1.cri".image_decryption]
681
# key_model defines the name of the key model used for how the cri obtains
682
# keys used for decryption of encrypted container images.
683
# The [decryption document](https://github.com/containerd/containerd/blob/main/docs/cri/decryption.md)
684
# contains additional information about the key models available.
685
#
686
# Set of available string options: {"", "node"}
687
# Omission of this field defaults to the empty string "", which indicates no key model,
688
# disabling image decryption.
689
#
690
# In order to use the decryption feature, additional configurations must be made.
691
# The [decryption document](https://github.com/containerd/containerd/blob/main/docs/cri/decryption.md)
692
# provides information of how to set up stream processors and the containerd imgcrypt decoder
693
# with the appropriate key models.
694
#
695
# Additional information:
696
# * Stream processors: https://github.com/containerd/containerd/blob/main/docs/stream_processors.md
697
# * Containerd imgcrypt: https://github.com/containerd/imgcrypt
698
key_model = "node"
699
700
# 'plugins."io.containerd.grpc.v1.cri".registry' contains config related to
701
# the registry
702
[plugins."io.containerd.grpc.v1.cri".registry]
703
# config_path specifies a directory to look for the registry hosts configuration.
704
#
705
# The cri plugin will look for and use config_path/host-namespace/hosts.toml
706
# configs if present OR load certificate files as laid out in the Docker/Moby
707
# specific layout https://docs.docker.com/engine/security/certificates/
708
#
709
# If config_path is not provided defaults are used.
710
#
711
# *** registry.configs and registry.mirrors that were a part of containerd 1.4
712
# are now DEPRECATED and will only be used if the config_path is not specified.
713
# It is an error to specify both config_path and the deprecated configs or mirrors
714
config_path = "/etc/containerd/certs.d:/etc/docker/certs.d"
715
```
716
717
</p>
718
</details>
719
720
## Registry Configuration
721
722
Here is a simple example for a default registry hosts configuration. Set
723
`config_path = "/etc/containerd/certs.d"` in your config.toml for containerd.
724
Make a directory tree at the config path that includes `docker.io` as a directory
725
representing the host namespace to be configured. Then add a `hosts.toml` file
726
in the `docker.io` to configure the host namespace. It should look like this:
727
```
728
$ tree /etc/containerd/certs.d
729
/etc/containerd/certs.d
730
└── docker.io
731
└── hosts.toml
732
733
$ cat /etc/containerd/certs.d/docker.io/hosts.toml
734
server = "https://docker.io"
735
736
[host."https://registry-1.docker.io"]
737
capabilities = ["pull", "resolve"]
738
```
739
740
To specify a custom certificate:
741
742
```
743
$ cat /etc/containerd/certs.d/192.168.12.34:5000/hosts.toml
744
server = "https://192.168.12.34:5000"
745
746
[host."https://192.168.12.34:5000"]
747
ca = "/path/to/ca.crt"
748
```
749
750
See [`docs/hosts.md`](https://github.com/containerd/containerd/blob/main/docs/hosts.md) for the further information.
751
752
## Untrusted Workload
753
754
The recommended way to run untrusted workload is to use
755
[`RuntimeClass`](https://kubernetes.io/docs/concepts/containers/runtime-class/) api
756
introduced in Kubernetes 1.12 to select RuntimeHandlers configured to run
757
untrusted workload in `plugins."io.containerd.grpc.v1.cri".containerd.runtimes`.
758
759
However, if you are using the legacy `io.kubernetes.cri.untrusted-workload`pod annotation
760
to request a pod be run using a runtime for untrusted workloads, the RuntimeHandler
761
`plugins."io.containerd.grpc.v1.cri"cri.containerd.runtimes.untrusted` must be defined first.
762
When the annotation `io.kubernetes.cri.untrusted-workload` is set to `true` the `untrusted`
763
runtime will be used. For example, see
764
[Create an untrusted pod using Kata Containers](https://github.com/kata-containers/kata-containers/blob/main/docs/how-to/containerd-kata.md#kata-containers-as-the-runtime-for-untrusted-workload).
765
766
## CNI Config Template
767
768
Ideally the cni config should be placed by system admin or cni daemon like calico, weaveworks etc.
769
However, this is useful for the cases when there is no cni daemonset to place cni config.
770
771
The cni config template uses the [golang
772
template](https://golang.org/pkg/text/template/) format. Currently supported
773
values are:
774
* `.PodCIDR` is a string of the first CIDR assigned to the node.
775
* `.PodCIDRRanges` is a string array of all CIDRs assigned to the node. It is
776
usually used for
777
[dualstack](https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/563-dual-stack) support.
778
* `.Routes` is a string array of all routes needed. It is usually used for
779
dualstack support or single stack but IPv4 or IPv6 is decided at runtime.
780
781
The [golang template actions](https://golang.org/pkg/text/template/#hdr-Actions)
782
can be used to render the cni config. For example, you can use the following
783
template to add CIDRs and routes for dualstack in the CNI config:
784
```
785
"ipam": {
786
"type": "host-local",
787
"ranges": [{{range $i, $range := .PodCIDRRanges}}{{if $i}}, {{end}}[{"subnet": "{{$range}}"}]{{end}}],
788
"routes": [{{range $i, $route := .Routes}}{{if $i}}, {{end}}{"dst": "{{$route}}"}{{end}}]
789
}
790
```
791
792
## Deprecation
793
The config options of the CRI plugin follow the [Kubernetes deprecation
794
policy of "admin-facing CLI components"](https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecating-a-flag-or-cli).
795
796
In summary, when a config option is announced to be deprecated:
797
* It is kept functional for 6 months or 1 release (whichever is longer);
798
* A warning is emitted when it is used.