Skip to content

containerd crash. #1865

@Random-Liu

Description

@Random-Liu

Log on cri-containerd side:

I1203 19:46:46.985333    1092 instrumented_service.go:185] ExecSync for "a970277beec1b5090bb2b275ccfed47fd2322314b309f32b7e44176824e7de4a" with command [test -f /data/statefulset-continue] and timeout 1 (s)
E1203 19:46:47.297301    1092 instrumented_service.go:188] ExecSync for "a970277beec1b5090bb2b275ccfed47fd2322314b309f32b7e44176824e7de4a" failed, error: failed to exec in container: failed to get container spec: transport is closing: unavailable
E1203 19:46:47.297380    1092 events.go:123] failed to stop container "fbba1e7d9d0b64b4db05b4c83bf5b55f467c0aaf8d19099d0e639f2a02682a8a": transport is closing: unavailable
E1203 19:46:47.297391    1092 events.go:65] Failed to handle event stream: rpc error: code = Unavailable desc = transport is closing
I1203 19:46:47.297408    1092 service.go:250] Stop cri-containerd service
I1203 19:46:47.297441    1092 service.go:240] Event monitor stopped
E1203 19:46:47.297477    1092 snapshots.go:62] Failed to sync snapshot stats: walk all snapshots failed: transport is closing: unavailable
E1203 19:46:47.297500    1092 instrumented_service.go:52] ListPodSandbox failed, error: failed to list sandbox containers: rpc error: code = Canceled desc = context canceled
E1203 19:46:47.297528    1092 service.go:207] Failed to start streaming server: http: Server closed
I1203 19:46:47.297538    1092 service.go:242] Stream server stopped
E1203 19:46:47.297552    1092 service.go:226] Failed to serve grpc grpc request: accept unix /var/run/cri-containerd.sock: use of closed network connection
I1203 19:46:47.297560    1092 service.go:244] GRPC server stopped
I1203 19:46:47.297567    1092 service.go:250] Stop cri-containerd service
cri-containerd.service: Service hold-off time over, scheduling restart.
Stopped Kubernetes containerd CRI shim.
Started Kubernetes containerd CRI shim.
I1203 19:46:52.666402   25997 cri_containerd.go:100] Run cri-containerd &{Config:{ContainerdConfig:{RootDir:/var/lib/containerd Snapshotter:overlayfs Endpoint:/run/containerd/containerd.sock Runtime:io.containerd.runtime.v1.linux RuntimeEngine: RuntimeRoot:} CniConfig:{NetworkPluginBinDir:/home/kubernetes/bin NetworkPluginConfDir:/etc/cni/net.d} SocketPath:/var/run/cri-containerd.sock RootDir:/var/lib/cri-containerd StreamServerAddress: StreamServerPort:10010 CgroupPath: EnableSelinux:false SandboxImage:gcr.io/google_containers/pause:3.0 StatsCollectPeriod:10 SystemdCgroup:false OOMScore:-999 EnableProfiling:true ProfilingPort:10011 ProfilingAddress:127.0.0.1} ConfigFilePath:/etc/cri-containerd/config.toml}
I1203 19:46:52.666905   25997 cri_containerd.go:104] Start profiling server
I1203 19:46:52.667445   25997 cri_containerd.go:108] Run cri-containerd grpc server on socket "/var/run/cri-containerd.sock"
Error: failed to create CRI containerd service: failed to initialize containerd client with endpoint "/run/containerd/containerd.sock": failed to dial "/run/containerd/containerd.sock": dial unix /run/containerd/containerd.sock: connect: connection refused
Usage:
  cri-containerd [flags]
  cri-containerd [command]
Available Commands:
  default-config Print default toml config of cri-containerd.
  help           Help about any command
  load           Load an image from a tar archive.
  version        Print cri-containerd version information.
Flags:
      --alsologtostderr                    log to standard error as well as files
      --cgroup-path string                 The cgroup that cri-containerd is part of. Cri-containerd is not placed in a cgroup if none is specified.
      --config string                      Path to the config file. (default "/etc/cri-containerd/config.toml")
      --containerd-endpoint string         Path to the containerd endpoint. (default "/run/containerd/containerd.sock")
      --containerd-root-dir string         Root directory path where containerd stores persistent data. (default "/var/lib/containerd")
      --containerd-runtime string          The runtime used by containerd. (default "io.containerd.runtime.v1.linux")
      --containerd-runtime-engine string   Runtime engine used by containerd. Defaults to containerd's default if not specified.
      --containerd-runtime-root string     The directory used by containerd for runtime state. Defaults to containerd's default if not specified.
      --containerd-snapshotter string      The snapshotter used by containerd. (default "overlayfs")
      --enable-selinux                     Enable selinux support. By default not enabled.
  -h, --help                               help for cri-containerd
      --log_backtrace_at traceLocation     when logging hits line file:N, emit a stack trace (default :0)
      --log_dir string                     If non-empty, write log files in this directory
      --logtostderr                        log to standard error instead of files (default false)
      --network-bin-dir string             The directory for putting network binaries. (default "/opt/cni/bin")
      --network-conf-dir string            The directory for putting network plugin configuration files. (default "/etc/cni/net.d")
      --oom-score int                      Adjust the cri-containerd's oom score. (default -999)
      --profiling                          Enable profiling via web interface host:port/debug/pprof/. (default true)
      --profiling-addr string              Profiling address for web interface host:port/debug/pprof/. (default "127.0.0.1")
      --profiling-port string              Profiling port for web interface host:port/debug/pprof/. (default "10011")
      --root-dir string                    Root directory path for cri-containerd managed files (metadata checkpoint etc). (default "/var/lib/cri-containerd")
      --sandbox-image string               The image used by sandbox container. (default "gcr.io/google_containers/pause:3.0")
      --socket-path string                 Path to the socket which cri-containerd serves on. (default "/var/run/cri-containerd.sock")
      --stats-collect-period int           The period (in seconds) of snapshots stats collection. (default 10)
      --stderrthreshold severity           logs at or above this threshold go to stderr (default 2)
      --stream-addr string                 The ip address streaming server is listening on. The default host interface is used if not specified.
      --stream-port string                 The port streaming server is listening on. (default "10010")
      --systemd-cgroup                     Enables systemd cgroup support. By default not enabled.
  -v, --v Level                            log level for V logs (default 0)
      --vmodule moduleSpec                 comma-separated list of pattern=N settings for file-filtered logging
Use "cri-containerd [command] --help" for more information about a command.
cri-containerd.service: Main process exited, code=exited, status=1/FAILURE
cri-containerd.service: Unit entered failed state.
cri-containerd.service: Failed with result 'exit-code'.
cri-containerd.service: Service hold-off time over, scheduling restart.
Stopped Kubernetes containerd CRI shim.
Started Kubernetes containerd CRI shim.
I1203 19:46:57.847103   26057 cri_containerd.go:100] Run cri-containerd &{Config:{ContainerdConfig:{RootDir:/var/lib/containerd Snapshotter:overlayfs Endpoint:/run/containerd/containerd.sock Runtime:io.containerd.runtime.v1.linux RuntimeEngine: RuntimeRoot:} CniConfig:{NetworkPluginBinDir:/home/kubernetes/bin NetworkPluginConfDir:/etc/cni/net.d} SocketPath:/var/run/cri-containerd.sock RootDir:/var/lib/cri-containerd StreamServerAddress: StreamServerPort:10010 CgroupPath: EnableSelinux:false SandboxImage:gcr.io/google_containers/pause:3.0 StatsCollectPeriod:10 SystemdCgroup:false OOMScore:-999 EnableProfiling:true ProfilingPort:10011 ProfilingAddress:127.0.0.1} ConfigFilePath:/etc/cri-containerd/config.toml}
I1203 19:46:57.849122   26057 cri_containerd.go:104] Start profiling server
I1203 19:46:57.849152   26057 cri_containerd.go:108] Run cri-containerd grpc server on socket "/var/run/cri-containerd.sock"
I1203 19:46:57.858897   26057 service.go:155] Get device uuid "34c2d19c-feb1-4d79-86d0-088a3a41b931" for image filesystem "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs"
time="2017-12-03T19:46:57Z" level=info msg="CNI network k8s-pod-network (type=calico) is used from /etc/cni/net.d/10-calico.conflist"
time="2017-12-03T19:46:57Z" level=info msg="CNI network k8s-pod-network (type=calico) is used from /etc/cni/net.d/10-calico.conflist"
I1203 19:46:57.859761   26057 interface.go:360] Looking for default routes with IPv4 addresses
I1203 19:46:57.859778   26057 interface.go:365] Default route transits interface "eth0"
I1203 19:46:57.860133   26057 interface.go:174] Interface eth0 is up
I1203 19:46:57.860233   26057 interface.go:222] Interface "eth0" has 2 addresses :[10.128.0.6/32 fe80::4001:aff:fe80:6/64].
I1203 19:46:57.860257   26057 interface.go:189] Checking addr  10.128.0.6/32.
I1203 19:46:57.860267   26057 interface.go:196] IP found 10.128.0.6
I1203 19:46:57.860278   26057 interface.go:228] Found valid IPv4 address 10.128.0.6 for interface "eth0".
I1203 19:46:57.860287   26057 interface.go:371] Found active IP 10.128.0.6
I1203 19:46:57.860777   26057 service.go:182] Start cri-containerd service
I1203 19:46:57.860787   26057 service.go:184] Start recovering state

Log on containerd side:

fatal error: concurrent map iteration and map write
goroutine 56996 [running]:
runtime.throw(0x55b96504e743, 0x26)
	/usr/local/go/src/runtime/panic.go:605 +0x97 fp=0xc424083568 sp=0xc424083548 pc=0x55b964956d77
runtime.mapiternext(0xc424083668)
	/usr/local/go/src/runtime/hashmap.go:778 +0x6f3 fp=0xc424083600 sp=0xc424083568 pc=0x55b964934d13
github.com/containerd/containerd/runtime.(*TaskList).GetAll(0xc4201ebe50, 0x55b9658db580, 0xc421c36c90, 0x55b964939c9a, 0x55b96545f680, 0x55b96547ed00, 0xc424083700, 0x55b9658db580)
	/go/src/github.com/containerd/containerd/runtime/task_list.go:61 +0x110 fp=0xc4240836d8 sp=0xc424083600 pc=0x55b964ed8300
github.com/containerd/containerd/linux.(*Runtime).Tasks(0xc420061920, 0x55b9658db580, 0xc421c36c90, 0x55b9658db580, 0xc421c36c90, 0x0, 0x55b9658de600, 0xc4202a0400)
	/go/src/github.com/containerd/containerd/linux/runtime.go:343 +0x45 fp=0xc424083728 sp=0xc4240836d8 pc=0x55b964f9e5c5
github.com/containerd/containerd/services/tasks.(*service).List(0xc4202107b0, 0x7f93231d0270, 0xc421c36c90, 0xc4239de370, 0xc4202107b0, 0x55b96504f86a, 0x4)
	/go/src/github.com/containerd/containerd/services/tasks/service.go:280 +0x142 fp=0xc4240837f0 sp=0xc424083728 pc=0x55b964edba62
github.com/containerd/containerd/api/services/tasks/v1._Tasks_List_Handler.func1(0x7f93231d0270, 0xc421c36c90, 0x55b9654b6000, 0xc4239de370, 0xc423a54f50, 0x55b965938b38, 0x55b9653f83a0, 0xc4239de3b0)
	/go/src/github.com/containerd/containerd/api/services/tasks/v1/tasks.pb.go:698 +0x88 fp=0xc424083838 sp=0xc4240837f0 pc=0x55b964ed4ce8
github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.UnaryServerInterceptor(0x7f93231d0270, 0xc421c36c90, 0x55b9654b6000, 0xc4239de370, 0xc4228ac160, 0xc4228ac180, 0x50, 0x48, 0xc4228ac140, 0xc424083968)
	/go/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:29 +0xd4 fp=0xc4240838f0 sp=0xc424083838 pc=0x55b9650261c4
github.com/containerd/containerd/server.interceptor(0x7f93231d0270, 0xc421c36bd0, 0x55b9654b6000, 0xc4239de370, 0xc4228ac160, 0xc4228ac180, 0x0, 0xc4240839e0, 0x55b96493bfda, 0x50)
	/go/src/github.com/containerd/containerd/server/server.go:267 +0x1d2 fp=0xc424083978 sp=0xc4240838f0 pc=0x55b96502a482
github.com/containerd/containerd/api/services/tasks/v1._Tasks_List_Handler(0x55b965509840, 0xc4202107b0, 0x7f93231d0270, 0xc421c36ab0, 0xc423a54e10, 0x55b965521428, 0x0, 0x0, 0x1000, 0x0)
	/go/src/github.com/containerd/containerd/api/services/tasks/v1/tasks.pb.go:700 +0x16f fp=0xc4240839f0 sp=0xc424083978 pc=0x55b964eb5c8f
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc420081e00, 0x55b9658de6c0, 0xc420346000, 0xc4218c8a00, 0xc420211110, 0x55b9658c5d58, 0x0, 0x0, 0x0)
	/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:826 +0xab6 fp=0xc424083d90 sp=0xc4240839f0 pc=0x55b964dc1816
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleStream(0xc420081e00, 0x55b9658de6c0, 0xc420346000, 0xc4218c8a00, 0x0)
	/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1023 +0x152a fp=0xc424083f80 sp=0xc424083d90 pc=0x55b964dc505a
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc42030e0d0, 0xc420081e00, 0x55b9658de6c0, 0xc420346000, 0xc4218c8a00)
	/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:572 +0xa1 fp=0xc424083fb8 sp=0xc424083f80 pc=0x55b964dcb0e1
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc424083fc0 sp=0xc424083fb8 pc=0x55b964989051
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:570 +0xa3
goroutine 1 [chan receive, 10 minutes]:
main.main.func1(0xc42009d340, 0xc42009d340, 0xc42004bb4f)
	/go/src/github.com/containerd/containerd/cmd/containerd/main.go:134 +0x878
github.com/containerd/containerd/vendor/github.com/urfave/cli.HandleAction(0x55b96540c700, 0x55b965522638, 0xc42009d340, 0xc420060d20, 0x0)
	/go/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:502 +0xd4
github.com/containerd/containerd/vendor/github.com/urfave/cli.(*App).Run(0xc42016ca80, 0xc420010090, 0x3, 0x3, 0x0, 0x0)
	/go/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:268 +0x655
main.main()
...

containerd.log: https://storage.googleapis.com/kubernetes-jenkins/logs/ci-cri-containerd-e2e-gci-gce/628/build-log.txt
cri-containerd.log: https://storage.googleapis.com/kubernetes-jenkins/logs/ci-cri-containerd-e2e-gci-gce/628/artifacts/bootstrap-e2e-minion-group-xnmw/cri-containerd.log

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions