Skip to content

Commit ca71484

Browse files
dmcgowandmcgowan
authored
Merge pull request #2480 from dmcgowan/proxy-plugin-doc
Document plugins
2 parents 578a236 + 1580ec5 commit ca71484

4 files changed

Lines changed: 507 additions & 33 deletions

File tree

‎PLUGINS.md‎

Lines changed: 267 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,267 @@
1+
# containerd Plugins
2+
3+
containerd supports extending its functionality using most of its defined
4+
interfaces. This includes using a customized runtime, snapshotter, content
5+
store, and even adding gRPC interfaces.
6+
7+
## Smart Client Model
8+
9+
containerd has a smart client architecture, meaning any functionality which is
10+
not required by the daemon is done by the client. This includes most high
11+
level interactions such as creating a container's specification, interacting
12+
with an image registry, or loading an image from tar. containerd's Go client
13+
gives a user access to many points of extensions from creating their own
14+
options on container creation to resolving image registry names.
15+
16+
See [containerd's Go documentation](https://godoc.org/github.com/containerd/containerd)
17+
18+
## External Plugins
19+
20+
External plugins allow extending containerd's functionality using an officially
21+
released version of containerd without needing to recompile the daemon to add a
22+
plugin.
23+
24+
containerd allows extensions through two method:
25+
- via a binary available in containerd's PATH
26+
- by configuring containerd to proxy to another gRPC service
27+
28+
### V2 Runtimes
29+
30+
The runtime v2 interface allows resolving runtimes to binaries on the system.
31+
These binaries are used to start the shim process for containerd and allows
32+
containerd to manage those containers using the runtime shim api returned by
33+
the binary.
34+
35+
See [runtime v2 documentation](runtime/v2/README.md)
36+
37+
### Proxy Plugins
38+
39+
A proxy plugin is configured using containerd's config file and will be loaded
40+
alongside the internal plugins when containerd is started. These plugins are
41+
connected to containerd using a local socket serving one of containerd's gRPC
42+
API services. Each plugin is configured with a type and name just as internal
43+
plugins are.
44+
45+
#### Configuration
46+
47+
Update the containerd config file, which by default is at
48+
`/etc/containerd/config.toml`. Add a `[proxy_plugins]` section along with a
49+
section for your given plugin `[proxy_plugins.myplugin]`. The `address` must
50+
refer to a local socket file which the containerd process has access to. The
51+
currently supported types are `snapshot` and `content`.
52+
53+
```
54+
[proxy_plugins]
55+
[proxy_plugins.customsnapshot]
56+
type = "snapshot"
57+
address = "/var/run/mysnapshotter.sock"
58+
```
59+
60+
#### Implementation
61+
62+
Implementing a proxy plugin is as easy as implementing the gRPC API for a
63+
service. For implementing a proxy plugin in Go, look at the go doc for
64+
[content store service](https://godoc.org/github.com/containerd/containerd/api/services/content/v1#ContentServer)
65+
and [snapshotter service](https://godoc.org/github.com/containerd/containerd/api/services/snapshots/v1#SnapshotsServer).
66+
67+
The following example creates a snapshot plugin binary which can be used
68+
with any implementation of
69+
[containerd's Snapshotter interface](https://godoc.org/github.com/containerd/containerd/snapshots#Snapshotter)
70+
```go
71+
package main
72+
73+
import (
74+
"fmt"
75+
"net"
76+
"os"
77+
78+
"google.golang.org/grpc"
79+
80+
snapshotsapi "github.com/containerd/containerd/api/services/snapshots/v1"
81+
"github.com/containerd/containerd/contrib/snapshotservice"
82+
"github.com/containerd/containerd/snapshots/native"
83+
)
84+
85+
func main() {
86+
// Provide a unix address to listen to, this will be the `address`
87+
// in the `proxy_plugin` configuration.
88+
// The root will be used to store the snapshots.
89+
if len(os.Args) < 3 {
90+
fmt.Printf("invalid args: usage: %s <unix addr> <root>\n", os.Args[0])
91+
os.Exit(1)
92+
}
93+
94+
// Create a gRPC server
95+
rpc := grpc.NewServer()
96+
97+
// Configure your custom snapshotter, this example uses the native
98+
// snapshotter and a root directory. Your custom snapshotter will be
99+
// much more useful than using a snapshotter which is already included.
100+
// https://godoc.org/github.com/containerd/containerd/snapshots#Snapshotter
101+
sn, err := native.NewSnapshotter(os.Args[2])
102+
if err != nil {
103+
fmt.Printf("error: %v\n", err)
104+
os.Exit(1)
105+
}
106+
107+
// Convert the snapshotter to a gRPC service,
108+
// example in github.com/containerd/containerd/contrib/snapshotservice
109+
service := snapshotservice.FromSnapshotter(sn)
110+
111+
// Register the service with the gRPC server
112+
snapshotsapi.RegisterSnapshotsServer(rpc, service)
113+
114+
// Listen and serve
115+
l, err := net.Listen("unix", os.Args[1])
116+
if err != nil {
117+
fmt.Printf("error: %v\n", err)
118+
os.Exit(1)
119+
}
120+
if err := rpc.Serve(l); err != nil {
121+
fmt.Printf("error: %v\n", err)
122+
os.Exit(1)
123+
}
124+
}
125+
```
126+
127+
Using the previous configuration and example, you could run a snapshot plugin
128+
with
129+
```
130+
# Start plugin in one terminal
131+
$ go run ./main.go /var/run/mysnapshotter.sock /tmp/snapshots
132+
133+
# Use ctr in another
134+
$ CONTAINERD_SNAPSHOTTER=customsnapshot ctr images pull docker.io/library/alpine:latest
135+
$ tree -L 3 /tmp/snapshots
136+
/tmp/snapshots
137+
|-- metadata.db
138+
`-- snapshots
139+
`-- 1
140+
|-- bin
141+
|-- dev
142+
|-- etc
143+
|-- home
144+
|-- lib
145+
|-- media
146+
|-- mnt
147+
|-- proc
148+
|-- root
149+
|-- run
150+
|-- sbin
151+
|-- srv
152+
|-- sys
153+
|-- tmp
154+
|-- usr
155+
`-- var
156+
157+
18 directories, 1 file
158+
```
159+
160+
## Built-in Plugins
161+
162+
containerd uses plugins internally to ensure that internal implementations are
163+
decoupled, stable, and treated equally with external plugins. To see all the
164+
plugins containerd has, use `ctr plugins ls`
165+
166+
```
167+
$ ctr plugins ls
168+
TYPE ID PLATFORMS STATUS
169+
io.containerd.content.v1 content - ok
170+
io.containerd.snapshotter.v1 btrfs linux/amd64 ok
171+
io.containerd.snapshotter.v1 aufs linux/amd64 error
172+
io.containerd.snapshotter.v1 native linux/amd64 ok
173+
io.containerd.snapshotter.v1 overlayfs linux/amd64 ok
174+
io.containerd.snapshotter.v1 zfs linux/amd64 error
175+
io.containerd.metadata.v1 bolt - ok
176+
io.containerd.differ.v1 walking linux/amd64 ok
177+
io.containerd.gc.v1 scheduler - ok
178+
io.containerd.service.v1 containers-service - ok
179+
io.containerd.service.v1 content-service - ok
180+
io.containerd.service.v1 diff-service - ok
181+
io.containerd.service.v1 images-service - ok
182+
io.containerd.service.v1 leases-service - ok
183+
io.containerd.service.v1 namespaces-service - ok
184+
io.containerd.service.v1 snapshots-service - ok
185+
io.containerd.runtime.v1 linux linux/amd64 ok
186+
io.containerd.runtime.v2 task linux/amd64 ok
187+
io.containerd.monitor.v1 cgroups linux/amd64 ok
188+
io.containerd.service.v1 tasks-service - ok
189+
io.containerd.internal.v1 restart - ok
190+
io.containerd.grpc.v1 containers - ok
191+
io.containerd.grpc.v1 content - ok
192+
io.containerd.grpc.v1 diff - ok
193+
io.containerd.grpc.v1 events - ok
194+
io.containerd.grpc.v1 healthcheck - ok
195+
io.containerd.grpc.v1 images - ok
196+
io.containerd.grpc.v1 leases - ok
197+
io.containerd.grpc.v1 namespaces - ok
198+
io.containerd.grpc.v1 snapshots - ok
199+
io.containerd.grpc.v1 tasks - ok
200+
io.containerd.grpc.v1 version - ok
201+
io.containerd.grpc.v1 cri linux/amd64 ok
202+
```
203+
204+
From the output all the plugins can be seen as well those which did not
205+
successfully load. In this case `aufs` and `zfs` are expected not to load
206+
since they are not support on the machine. The logs will show why it failed,
207+
but you can also get more details using the `-d` option.
208+
209+
```
210+
$ ctr plugins ls -d id==aufs id==zfs
211+
Type: io.containerd.snapshotter.v1
212+
ID: aufs
213+
Platforms: linux/amd64
214+
Exports:
215+
root /var/lib/containerd/io.containerd.snapshotter.v1.aufs
216+
Error:
217+
Code: Unknown
218+
Message: modprobe aufs failed: "modprobe: FATAL: Module aufs not found in directory /lib/modules/4.17.2-1-ARCH\n": exit status 1
219+
220+
Type: io.containerd.snapshotter.v1
221+
ID: zfs
222+
Platforms: linux/amd64
223+
Exports:
224+
root /var/lib/containerd/io.containerd.snapshotter.v1.zfs
225+
Error:
226+
Code: Unknown
227+
Message: path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter
228+
```
229+
230+
The error message which the plugin returned explains why the plugin was unable
231+
to load.
232+
233+
#### Configuration
234+
235+
Plugins are configured using the `[plugins]` section of containerd's config.
236+
Every plugin can have its own section using the pattern `[plugins.<plugin id>]`.
237+
238+
example configuration
239+
```
240+
[plugins]
241+
[plugins.cgroups]
242+
no_prometheus = false
243+
[plugins.cri]
244+
stream_server_address = ""
245+
stream_server_port = "10010"
246+
enable_selinux = false
247+
sandbox_image = "k8s.gcr.io/pause:3.1"
248+
stats_collect_period = 10
249+
systemd_cgroup = false
250+
[plugins.cri.containerd]
251+
snapshotter = "overlayfs"
252+
[plugins.cri.containerd.default_runtime]
253+
runtime_type = "io.containerd.runtime.v1.linux"
254+
runtime_engine = ""
255+
runtime_root = ""
256+
[plugins.cri.containerd.untrusted_workload_runtime]
257+
runtime_type = ""
258+
runtime_engine = ""
259+
runtime_root = ""
260+
[plugins.cri.cni]
261+
bin_dir = "/opt/cni/bin"
262+
conf_dir = "/etc/cni/net.d"
263+
[plugins.cri.registry]
264+
[plugins.cri.registry.mirrors]
265+
[plugins.cri.registry.mirrors."docker.io"]
266+
endpoint = ["https://registry-1.docker.io"]
267+
```

‎README.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -184,6 +184,28 @@ defer task.Delete(context)
184184
err := task.Start(context)
185185
```
186186

187+
### Snapshot Plugins
188+
189+
In addition to the built-in Snapshot plugins in containerd, additional external
190+
plugins can be configured using GRPC. An external plugin is made available using
191+
the configured name and appears as a plugin alongside the built-in ones.
192+
193+
To add an external snapshot plugin, add the plugin to containerd's config file
194+
(by default at `/etc/containerd/config.toml`). The string following
195+
`proxy_plugin.` will be used as the name of the snapshotter and the address
196+
should refer to a socket with a GRPC listener serving containerd's Snapshot
197+
GRPC API. Remember to restart containerd for any configuration changes to take
198+
effect.
199+
200+
```
201+
[proxy_plugins]
202+
[proxy_plugins.customsnapshot]
203+
type = "snapshot"
204+
address = "/var/run/mysnapshotter.sock"
205+
```
206+
207+
See [PLUGINS.md](PLUGINS.md) for how to create plugins
208+
187209
### Releases and API Stability
188210

189211
Please see [RELEASES.md](RELEASES.md) for details on versioning and stability

0 commit comments

Comments
 (0)