You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm facing an issue in minikube which writes a containerd config.toml file, restarts the containerd using systemctl, and then tries to query containerd using crictl, all in less than a second.
The problem is that when containerd is restarted and data dir is quite huge (>2 GB in my case), socket is created but containerd is not fully initialized, so "crictl info" will fail with
rpc error: code = Unknown desc = server is not initialized yet
In journald, I see the following (note that minikube uses sudo to run systemctl and crictl, so you'll see a lot of non-containerd info)
Mar 02 18:24:12 localhost sudo[4011]: root : TTY=pts/0 ; PWD=/ops/kubernetes-package ; USER=root ; COMMAND=/usr/bin/systemctl restart containerd
Mar 02 18:24:12 localhost sudo[4011]: pam_unix(sudo:session): session opened for user root by root(uid=0)
Mar 02 18:24:12 localhost systemd[1]: Created slice User Slice of root.
Mar 02 18:24:12 localhost systemd[1]: Starting User Manager for UID 0...
Mar 02 18:24:12 localhost systemd-logind[878]: New session c11 of user root.
Mar 02 18:24:12 localhost systemd[1]: Started Session c11 of user root.
Mar 02 18:24:12 localhost systemd[4015]: pam_unix(systemd-user:session): session opened for user root by (uid=0)
Mar 02 18:24:12 localhost systemd[4015]: Reached target Sockets.
Mar 02 18:24:12 localhost systemd[4015]: Reached target Timers.
Mar 02 18:24:12 localhost systemd[4015]: Reached target Paths.
Mar 02 18:24:12 localhost systemd[4015]: Reached target Basic System.
Mar 02 18:24:12 localhost systemd[4015]: Reached target Default.
Mar 02 18:24:12 localhost systemd[4015]: Startup finished in 18ms.
Mar 02 18:24:12 localhost systemd[1]: Started User Manager for UID 0.
Mar 02 18:24:12 localhost systemd[1]: Stopping containerd container runtime...
Mar 02 18:24:12 localhost containerd[3541]: time="2021-03-02T18:24:12.418917421+01:00" level=info msg="Stop CRI service"
Mar 02 18:24:12 localhost containerd[3541]: time="2021-03-02T18:24:12.432746715+01:00" level=info msg="Stop CRI service"
Mar 02 18:24:12 localhost containerd[3541]: time="2021-03-02T18:24:12.432772387+01:00" level=info msg="Event monitor stopped"
Mar 02 18:24:12 localhost containerd[3541]: time="2021-03-02T18:24:12.432800653+01:00" level=info msg="Stream server stopped"
Mar 02 18:24:12 localhost systemd[1]: Stopped containerd container runtime.
Mar 02 18:24:12 localhost systemd[1]: Starting containerd container runtime...
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.451884151+01:00" level=info msg="starting containerd" revision=269548fa27e0089a8b8278fc4fc781d7f65a939b version=v1.4.3
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.480983793+01:00" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.481034962+01:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.482298362+01:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/4.12.14-122.41-de
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.482334715+01:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.482692848+01:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.482723296+01:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.482738043+01:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.482760169+01:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.482840644+01:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.483162385+01:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /HRAP/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.co
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.483182179+01:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.483202201+01:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.483212621+01:00" level=info msg="metadata content store policy set" policy=shared
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.483305692+01:00" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.483325056+01:00" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.483387137+01:00" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.483420340+01:00" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.483437346+01:00" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.483451217+01:00" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.483485567+01:00" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.483501341+01:00" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.483516514+01:00" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.483530655+01:00" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.483543877+01:00" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.483599495+01:00" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.483648819+01:00" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.484080037+01:00" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.484120119+01:00" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.484174145+01:00" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.484191106+01:00" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.484205089+01:00" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.484217452+01:00" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.484229735+01:00" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.484244440+01:00" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.484258407+01:00" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.484272229+01:00" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.484285080+01:00" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.484316508+01:00" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.484333465+01:00" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.484348408+01:00" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.484360377+01:00" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.484486951+01:00" level=warning msg="`default_runtime` is deprecated, please use `default_runtime_name` to reference the default configuration you have defined in `runtimes`"
Mar 02 18:24:12 localhost systemd[1]: Started containerd container runtime.
Mar 02 18:24:12 localhost sudo[4011]: pam_unix(sudo:session): session closed for user root
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.484548363+01:00" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName:default DefaultRuntime:{Type:io.containerd.runtime.v1.linux Engine: PodAnnotations:[] ContainerAnnotations:[]
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.484626645+01:00" level=info msg="Connect containerd service"
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.484720771+01:00" level=info msg="Get image filesystem path \"/HRAP/lib/containerd/io.containerd.snapshotter.v1.overlayfs\""
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.486372250+01:00" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.486453955+01:00" level=info msg="Start subscribing containerd event"
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.486509778+01:00" level=info msg="Start recovering state"
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.486578710+01:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.486624642+01:00" level=info msg=serving... address=/run/containerd/containerd.sock
Mar 02 18:24:12 localhost containerd[4024]: time="2021-03-02T18:24:12.486670275+01:00" level=info msg="containerd successfully booted in 0.036825s"
Mar 02 18:24:12 localhost systemd-logind[878]: Removed session c11.
Mar 02 18:24:12 localhost systemd[1]: Stopping User Manager for UID 0...
Mar 02 18:24:12 localhost systemd[4015]: Reached target Shutdown.
Mar 02 18:24:12 localhost systemd[4015]: Starting Exit the Session...
Mar 02 18:24:12 localhost systemd[4015]: Stopped target Default.
Mar 02 18:24:12 localhost systemd[4015]: Stopped target Basic System.
Mar 02 18:24:12 localhost systemd[4015]: Stopped target Paths.
Mar 02 18:24:12 localhost systemd[4015]: Stopped target Timers.
Mar 02 18:24:12 localhost systemd[4015]: Stopped target Sockets.
Mar 02 18:24:12 localhost systemd[4015]: Received SIGRTMIN+24 from PID 4047 (kill).
Mar 02 18:24:12 localhost systemd[4016]: pam_unix(systemd-user:session): session closed for user root
Mar 02 18:24:12 localhost systemd[1]: Stopped User Manager for UID 0.
Mar 02 18:24:12 localhost systemd[1]: Removed slice User Slice of root.
Mar 02 18:24:12 localhost sudo[4064]: root : TTY=pts/0 ; PWD=/ops/kubernetes-package ; USER=root ; COMMAND=/usr/bin/crictl info
Mar 02 18:24:12 localhost sudo[4064]: pam_unix(sudo:session): session opened for user root by root(uid=0)
Mar 02 18:24:12 localhost systemd[1]: Created slice User Slice of root.
Mar 02 18:24:12 localhost systemd[1]: Starting User Manager for UID 0...
Mar 02 18:24:12 localhost systemd-logind[878]: New session c12 of user root.
Mar 02 18:24:12 localhost systemd[1]: Started Session c12 of user root.
Mar 02 18:24:12 localhost systemd[4065]: pam_unix(systemd-user:session): session opened for user root by (uid=0)
Mar 02 18:24:12 localhost systemd[4065]: Reached target Sockets.
Mar 02 18:24:12 localhost systemd[4065]: Reached target Timers.
Mar 02 18:24:12 localhost systemd[4065]: Reached target Paths.
Mar 02 18:24:12 localhost systemd[4065]: Reached target Basic System.
Mar 02 18:24:12 localhost systemd[4065]: Reached target Default.
Mar 02 18:24:12 localhost systemd[4065]: Startup finished in 17ms.
Mar 02 18:24:12 localhost systemd[1]: Started User Manager for UID 0.
Mar 02 18:24:12 localhost sudo[4064]: pam_unix(sudo:session): session closed for user root
Mar 02 18:24:12 localhost systemd-logind[878]: Removed session c12.
Mar 02 18:24:12 localhost systemd[1]: Stopping User Manager for UID 0...
Mar 02 18:24:12 localhost systemd[4065]: Reached target Shutdown.
Mar 02 18:24:12 localhost systemd[4065]: Stopped target Default.
Mar 02 18:24:12 localhost systemd[4065]: Starting Exit the Session...
Mar 02 18:24:12 localhost systemd[4065]: Stopped target Basic System.
Mar 02 18:24:12 localhost systemd[4065]: Stopped target Paths.
Mar 02 18:24:12 localhost systemd[4065]: Stopped target Timers.
Mar 02 18:24:12 localhost systemd[4065]: Stopped target Sockets.
Mar 02 18:24:12 localhost systemd[4065]: Received SIGRTMIN+24 from PID 4080 (kill).
Mar 02 18:24:12 localhost systemd[4066]: pam_unix(systemd-user:session): session closed for user root
Mar 02 18:24:12 localhost systemd[1]: Stopped User Manager for UID 0.
Mar 02 18:24:12 localhost systemd[1]: Removed slice User Slice of root.
Mar 02 18:24:13 localhost containerd[4024]: time="2021-03-02T18:24:13.030195225+01:00" level=info msg="Start event monitor"
Mar 02 18:24:13 localhost containerd[4024]: time="2021-03-02T18:24:13.030260098+01:00" level=info msg="Start snapshots syncer"
Mar 02 18:24:13 localhost containerd[4024]: time="2021-03-02T18:24:13.030272534+01:00" level=info msg="Start cni network conf syncer"
Mar 02 18:24:13 localhost containerd[4024]: time="2021-03-02T18:24:13.030279405+01:00" level=info msg="Start streaming server"
As you can see, there is less than a second period between creation of the socket and successful boot, and the streaming server. The "sudo crictl info" command in the middle fails, and I can run it after minikube fails ...
Any idea how to handle this? Minikube dev has already added checking for presence of the socket before invoking crictl, but it appears that socket is created before containerd is fully initialized. Is there a way to perform a check whether container is fully started rather than spamming "crictl info" for fixed period of time?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
I'm facing an issue in minikube which writes a containerd config.toml file, restarts the containerd using systemctl, and then tries to query containerd using crictl, all in less than a second.
The problem is that when containerd is restarted and data dir is quite huge (>2 GB in my case), socket is created but containerd is not fully initialized, so "crictl info" will fail with
In journald, I see the following (note that minikube uses sudo to run systemctl and crictl, so you'll see a lot of non-containerd info)
As you can see, there is less than a second period between creation of the socket and successful boot, and the streaming server. The "sudo crictl info" command in the middle fails, and I can run it after minikube fails ...
Any idea how to handle this? Minikube dev has already added checking for presence of the socket before invoking crictl, but it appears that socket is created before containerd is fully initialized. Is there a way to perform a check whether container is fully started rather than spamming "crictl info" for fixed period of time?
Beta Was this translation helpful? Give feedback.
All reactions