[Sandbox API] CRI status cleanup by mxpv · Pull Request #7731 · containerd/containerd

mxpv · 2022-11-28T19:18:21Z

Sandbox info includes a lot of metadata about pause container (runtime handler, snapshotter info, etc), this PR moves that code to podsandbox package and extends Sandbox.Status APIs to pass new fields.
Don't access pause container when running a pod container.
Minor fixes to get sandboxed shims running.

Signed-off-by: Maksym Pavlenko <[email protected]>

mxpv · 2022-11-28T21:34:48Z

/test pull-containerd-sandboxed-node-e2e

dmcgowan · 2022-11-29T07:25:01Z

-	uint32 exit_status = 4;
-	google.protobuf.Timestamp exited_at = 5;
-	google.protobuf.Any extra = 6;
+	map<string, string> info = 4;


The API now has a map field info which has an "info" field of a json encoded value. Is there more fields intended there or having the field flattened out?

This is how it was originally structured, so I'm preserving existing behavior here.
Flattening sounds like a good idea. I'm not sure how much do we care about backward compatibility here. @mikebrow any thought on this? ^^

See bottom of this note for an example verbose pod status..

In the cri podsandbox implementation.. for status requests the PodSandboxStatusResponse includes Status and Info.. Status is the formal/required stuff and has quite a few fields see below for the first level. By contrast info is semi adhock optional content provided when verbose is status is requested.
We fill info with stuff like cni results which need to be "stored" with cri pod run requests.. ** At this point in time none of the "names" in the info name value pair list have been formalized at the CRI level.

I think it's fair to have this info map also include "stuff" the controller may want to be placed in the verbose status response so long as we don't duplicate the items in the map..

If anything feels like it should be formalized in the sandboxed controller .. or is already in the PodSandboxStatus struct then we need to consider continuing to flatten those fields into this controller status response record. I'm good with this direction..

Here is "our" version of the info map struct ( I see also in the moved code below..)

type SandboxInfo struct { Pid uint32 `json:"pid"` Status string `json:"processStatus"` NetNSClosed bool `json:"netNamespaceClosed"` Image string `json:"image"` SnapshotKey string `json:"snapshotKey"` Snapshotter string `json:"snapshotter"` // Note: a new field `RuntimeHandler` has been added into the CRI PodSandboxStatus struct, and // should be set. This `RuntimeHandler` field will be deprecated after containerd 1.3 (tracked // in https://github.com/containerd/cri/issues/1064). RuntimeHandler string `json:"runtimeHandler"` // see the Note above RuntimeType string `json:"runtimeType"` RuntimeOptions interface{} `json:"runtimeOptions"` Config *runtime.PodSandboxConfig `json:"config"` RuntimeSpec *runtimespec.Spec `json:"runtimeSpec"` CNIResult *cni.Result `json:"cniResult"` }

the formal status struct follows:

type PodSandboxStatus struct { // ID of the sandbox. Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"` // Metadata of the sandbox. Metadata *PodSandboxMetadata `protobuf:"bytes,2,opt,name=metadata,proto3" json:"metadata,omitempty"` // State of the sandbox. State PodSandboxState `protobuf:"varint,3,opt,name=state,proto3,enum=runtime.v1alpha2.PodSandboxState" json:"state,omitempty"` // Creation timestamp of the sandbox in nanoseconds. Must be > 0. CreatedAt int64 `protobuf:"varint,4,opt,name=created_at,json=createdAt,proto3" json:"created_at,omitempty"` // Network contains network status if network is handled by the runtime. Network *PodSandboxNetworkStatus `protobuf:"bytes,5,opt,name=network,proto3" json:"network,omitempty"` // Linux-specific status to a pod sandbox. Linux *LinuxPodSandboxStatus `protobuf:"bytes,6,opt,name=linux,proto3" json:"linux,omitempty"` // Labels are key-value pairs that may be used to scope and select individual resources. Labels map[string]string `protobuf:"bytes,7,rep,name=labels,proto3" json:"labels,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` // Unstructured key-value map holding arbitrary metadata. // Annotations MUST NOT be altered by the runtime; the value of this field // MUST be identical to that of the corresponding PodSandboxConfig used to // instantiate the pod sandbox this status represents. Annotations map[string]string `protobuf:"bytes,8,rep,name=annotations,proto3" json:"annotations,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` // runtime configuration used for this PodSandbox. RuntimeHandler string `protobuf:"bytes,9,opt,name=runtime_handler,json=runtimeHandler,proto3" json:"runtime_handler,omitempty"` XXX_NoUnkeyedLiteral struct{} `json:"-"` XXX_sizecache int32 `json:"-"` }

pull cri podsanbox status:

mike@ubnt:~/crictl-test$ sudo crictl inspectp 8 { "status": { "id": "8f9ec14c8c927fc516444052653a936fbd22d74d04b016e2809ddd2eb75a6cfa", "metadata": { "attempt": 1, "name": "busybox-sandbox", "namespace": "default", "uid": "hdishd83djaidwnduwk28bcsb" }, "state": "SANDBOX_READY", "createdAt": "2022-11-29T12:07:44.196650393-06:00", "network": { "additionalIps": [ { "ip": "2001:4860:4860::11" } ], "ip": "10.88.0.17" }, "linux": { "namespaces": { "options": { "ipc": "POD", "network": "POD", "pid": "POD", "targetId": "" } } }, "labels": {}, "annotations": {}, "runtimeHandler": "" }, "info": { "pid": 48095, "processStatus": "running", "netNamespaceClosed": false, "image": "registry.k8s.io/pause:3.8", "snapshotKey": "8f9ec14c8c927fc516444052653a936fbd22d74d04b016e2809ddd2eb75a6cfa", "snapshotter": "overlayfs", "runtimeHandler": "", "runtimeType": "io.containerd.runc.v2", "runtimeOptions": {}, "config": { "metadata": { "name": "busybox-sandbox", "uid": "hdishd83djaidwnduwk28bcsb", "namespace": "default", "attempt": 1 }, "linux": {} }, "runtimeSpec": { "ociVersion": "1.0.2-dev", "process": { "user": { "uid": 65535, "gid": 65535 }, "args": [ "/pause" ], "env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ], "cwd": "/", "capabilities": { "bounding": [ "CAP_CHOWN", "CAP_DAC_OVERRIDE", "CAP_FSETID", "CAP_FOWNER", "CAP_MKNOD", "CAP_NET_RAW", "CAP_SETGID", "CAP_SETUID", "CAP_SETFCAP", "CAP_SETPCAP", "CAP_NET_BIND_SERVICE", "CAP_SYS_CHROOT", "CAP_KILL", "CAP_AUDIT_WRITE" ], "effective": [ "CAP_CHOWN", "CAP_DAC_OVERRIDE", "CAP_FSETID", "CAP_FOWNER", "CAP_MKNOD", "CAP_NET_RAW", "CAP_SETGID", "CAP_SETUID", "CAP_SETFCAP", "CAP_SETPCAP", "CAP_NET_BIND_SERVICE", "CAP_SYS_CHROOT", "CAP_KILL", "CAP_AUDIT_WRITE" ], "permitted": [ "CAP_CHOWN", "CAP_DAC_OVERRIDE", "CAP_FSETID", "CAP_FOWNER", "CAP_MKNOD", "CAP_NET_RAW", "CAP_SETGID", "CAP_SETUID", "CAP_SETFCAP", "CAP_SETPCAP", "CAP_NET_BIND_SERVICE", "CAP_SYS_CHROOT", "CAP_KILL", "CAP_AUDIT_WRITE" ] }, "noNewPrivileges": true, "oomScoreAdj": -998 }, "root": { "path": "rootfs", "readonly": true }, "mounts": [ { "destination": "/proc", "type": "proc", "source": "proc", "options": [ "nosuid", "noexec", "nodev" ] }, { "destination": "/dev", "type": "tmpfs", "source": "tmpfs", "options": [ "nosuid", "strictatime", "mode=755", "size=65536k" ] }, { "destination": "/dev/pts", "type": "devpts", "source": "devpts", "options": [ "nosuid", "noexec", "newinstance", "ptmxmode=0666", "mode=0620", "gid=5" ] }, { "destination": "/dev/shm", "type": "tmpfs", "source": "shm", "options": [ "nosuid", "noexec", "nodev", "mode=1777", "size=65536k" ] }, { "destination": "/dev/mqueue", "type": "mqueue", "source": "mqueue", "options": [ "nosuid", "noexec", "nodev" ] }, { "destination": "/sys", "type": "sysfs", "source": "sysfs", "options": [ "nosuid", "noexec", "nodev", "ro" ] }, { "destination": "/dev/shm", "type": "bind", "source": "/run/containerd/io.containerd.grpc.v1.cri/sandboxes/8f9ec14c8c927fc516444052653a936fbd22d74d04b016e2809ddd2eb75a6cfa/shm", "options": [ "rbind", "ro" ] }, { "destination": "/etc/resolv.conf", "type": "bind", "source": "/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/8f9ec14c8c927fc516444052653a936fbd22d74d04b016e2809ddd2eb75a6cfa/resolv.conf", "options": [ "rbind", "ro" ] } ], "annotations": { "io.kubernetes.cri.container-type": "sandbox", "io.kubernetes.cri.sandbox-id": "8f9ec14c8c927fc516444052653a936fbd22d74d04b016e2809ddd2eb75a6cfa", "io.kubernetes.cri.sandbox-log-directory": "", "io.kubernetes.cri.sandbox-name": "busybox-sandbox", "io.kubernetes.cri.sandbox-namespace": "default" }, "linux": { "resources": { "devices": [ { "allow": false, "access": "rwm" } ], "cpu": { "shares": 2 } }, "cgroupsPath": "/k8s.io/8f9ec14c8c927fc516444052653a936fbd22d74d04b016e2809ddd2eb75a6cfa", "namespaces": [ { "type": "pid" }, { "type": "ipc" }, { "type": "uts" }, { "type": "mount" }, { "type": "network", "path": "/var/run/netns/cni-96eb786a-73b1-2237-88cd-185181cad3fe" } ], "maskedPaths": [ "/proc/acpi", "/proc/asound", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/sys/firmware", "/proc/scsi" ], "readonlyPaths": [ "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ] } }, "cniResult": { "Interfaces": { "cni0": { "IPConfigs": null, "Mac": "ce:01:90:a2:bf:41", "Sandbox": "" }, "eth0": { "IPConfigs": [ { "IP": "10.88.0.17", "Gateway": "10.88.0.1" }, { "IP": "2001:4860:4860::11", "Gateway": "2001:4860:4860::1" } ], "Mac": "a6:5e:17:ae:af:10", "Sandbox": "/var/run/netns/cni-96eb786a-73b1-2237-88cd-185181cad3fe" }, "lo": { "IPConfigs": [ { "IP": "127.0.0.1", "Gateway": "" }, { "IP": "::1", "Gateway": "" } ], "Mac": "00:00:00:00:00:00", "Sandbox": "/var/run/netns/cni-96eb786a-73b1-2237-88cd-185181cad3fe" }, "veth620affdb": { "IPConfigs": null, "Mac": "92:ee:61:ae:1f:08", "Sandbox": "" } }, "DNS": [ {}, {} ], "Routes": [ { "dst": "0.0.0.0/0" }, { "dst": "::/0" } ] } } }

mikebrow

LGTM

mxpv added 2 commits November 28, 2022 10:13

[sandbox] Update Controller.Status protos

a6d1d53

Signed-off-by: Maksym Pavlenko <[email protected]>

[sandbox] Move sandbox info to podsandbox controller

cc111ee

Signed-off-by: Maksym Pavlenko <[email protected]>

mxpv force-pushed the cri branch from da79535 to e2f5fba Compare November 28, 2022 20:06

[sandbox] Don't access pause container when creating pod container

9a53a6c

Signed-off-by: Maksym Pavlenko <[email protected]>

mxpv force-pushed the cri branch from e2f5fba to 9a53a6c Compare November 28, 2022 20:49

mxpv added 2 commits November 28, 2022 13:27

[sandbox] Specify sandbox ID when using sandboxed shims

dbc6d33

Signed-off-by: Maksym Pavlenko <[email protected]>

[sandbox] Fix panic when waiting for sandbox controller

9f4ba48

Signed-off-by: Maksym Pavlenko <[email protected]>

mxpv marked this pull request as ready for review November 28, 2022 21:34

containerd deleted a comment from k8s-ci-robot Nov 28, 2022

mxpv added the area/cri Container Runtime Interface (CRI) label Nov 28, 2022

mxpv added this to the 1.7 milestone Nov 28, 2022

dmcgowan reviewed Nov 29, 2022

View reviewed changes

dmcgowan self-assigned this Nov 30, 2022

mikebrow approved these changes Nov 30, 2022

View reviewed changes

dmcgowan approved these changes Dec 1, 2022

View reviewed changes

dmcgowan merged commit 51195ad into containerd:main Dec 1, 2022

mxpv deleted the cri branch December 1, 2022 21:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Sandbox API] CRI status cleanup#7731

[Sandbox API] CRI status cleanup#7731
dmcgowan merged 5 commits intocontainerd:mainfrom
mxpv:cri

mxpv commented Nov 28, 2022 •

edited

Loading

Uh oh!

mxpv commented Nov 28, 2022

Uh oh!

dmcgowan Nov 29, 2022

Uh oh!

mxpv Nov 29, 2022

Uh oh!

mikebrow Nov 30, 2022 •

edited

Loading

Uh oh!

mikebrow left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mxpv commented Nov 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mxpv commented Nov 28, 2022

Uh oh!

dmcgowan Nov 29, 2022

Choose a reason for hiding this comment

Uh oh!

mxpv Nov 29, 2022

Choose a reason for hiding this comment

Uh oh!

mikebrow Nov 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mikebrow left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mxpv commented Nov 28, 2022 •

edited

Loading

mikebrow Nov 30, 2022 •

edited

Loading