I1203 19:46:46.985333 1092 instrumented_service.go:185] ExecSync for "a970277beec1b5090bb2b275ccfed47fd2322314b309f32b7e44176824e7de4a" with command [test -f /data/statefulset-continue] and timeout 1 (s)
E1203 19:46:47.297301 1092 instrumented_service.go:188] ExecSync for "a970277beec1b5090bb2b275ccfed47fd2322314b309f32b7e44176824e7de4a" failed, error: failed to exec in container: failed to get container spec: transport is closing: unavailable
E1203 19:46:47.297380 1092 events.go:123] failed to stop container "fbba1e7d9d0b64b4db05b4c83bf5b55f467c0aaf8d19099d0e639f2a02682a8a": transport is closing: unavailable
E1203 19:46:47.297391 1092 events.go:65] Failed to handle event stream: rpc error: code = Unavailable desc = transport is closing
I1203 19:46:47.297408 1092 service.go:250] Stop cri-containerd service
I1203 19:46:47.297441 1092 service.go:240] Event monitor stopped
E1203 19:46:47.297477 1092 snapshots.go:62] Failed to sync snapshot stats: walk all snapshots failed: transport is closing: unavailable
E1203 19:46:47.297500 1092 instrumented_service.go:52] ListPodSandbox failed, error: failed to list sandbox containers: rpc error: code = Canceled desc = context canceled
E1203 19:46:47.297528 1092 service.go:207] Failed to start streaming server: http: Server closed
I1203 19:46:47.297538 1092 service.go:242] Stream server stopped
E1203 19:46:47.297552 1092 service.go:226] Failed to serve grpc grpc request: accept unix /var/run/cri-containerd.sock: use of closed network connection
I1203 19:46:47.297560 1092 service.go:244] GRPC server stopped
I1203 19:46:47.297567 1092 service.go:250] Stop cri-containerd service
cri-containerd.service: Service hold-off time over, scheduling restart.
Stopped Kubernetes containerd CRI shim.
Started Kubernetes containerd CRI shim.
I1203 19:46:52.666402 25997 cri_containerd.go:100] Run cri-containerd &{Config:{ContainerdConfig:{RootDir:/var/lib/containerd Snapshotter:overlayfs Endpoint:/run/containerd/containerd.sock Runtime:io.containerd.runtime.v1.linux RuntimeEngine: RuntimeRoot:} CniConfig:{NetworkPluginBinDir:/home/kubernetes/bin NetworkPluginConfDir:/etc/cni/net.d} SocketPath:/var/run/cri-containerd.sock RootDir:/var/lib/cri-containerd StreamServerAddress: StreamServerPort:10010 CgroupPath: EnableSelinux:false SandboxImage:gcr.io/google_containers/pause:3.0 StatsCollectPeriod:10 SystemdCgroup:false OOMScore:-999 EnableProfiling:true ProfilingPort:10011 ProfilingAddress:127.0.0.1} ConfigFilePath:/etc/cri-containerd/config.toml}
I1203 19:46:52.666905 25997 cri_containerd.go:104] Start profiling server
I1203 19:46:52.667445 25997 cri_containerd.go:108] Run cri-containerd grpc server on socket "/var/run/cri-containerd.sock"
Error: failed to create CRI containerd service: failed to initialize containerd client with endpoint "/run/containerd/containerd.sock": failed to dial "/run/containerd/containerd.sock": dial unix /run/containerd/containerd.sock: connect: connection refused
Usage:
cri-containerd [flags]
cri-containerd [command]
Available Commands:
default-config Print default toml config of cri-containerd.
help Help about any command
load Load an image from a tar archive.
version Print cri-containerd version information.
Flags:
--alsologtostderr log to standard error as well as files
--cgroup-path string The cgroup that cri-containerd is part of. Cri-containerd is not placed in a cgroup if none is specified.
--config string Path to the config file. (default "/etc/cri-containerd/config.toml")
--containerd-endpoint string Path to the containerd endpoint. (default "/run/containerd/containerd.sock")
--containerd-root-dir string Root directory path where containerd stores persistent data. (default "/var/lib/containerd")
--containerd-runtime string The runtime used by containerd. (default "io.containerd.runtime.v1.linux")
--containerd-runtime-engine string Runtime engine used by containerd. Defaults to containerd's default if not specified.
--containerd-runtime-root string The directory used by containerd for runtime state. Defaults to containerd's default if not specified.
--containerd-snapshotter string The snapshotter used by containerd. (default "overlayfs")
--enable-selinux Enable selinux support. By default not enabled.
-h, --help help for cri-containerd
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--logtostderr log to standard error instead of files (default false)
--network-bin-dir string The directory for putting network binaries. (default "/opt/cni/bin")
--network-conf-dir string The directory for putting network plugin configuration files. (default "/etc/cni/net.d")
--oom-score int Adjust the cri-containerd's oom score. (default -999)
--profiling Enable profiling via web interface host:port/debug/pprof/. (default true)
--profiling-addr string Profiling address for web interface host:port/debug/pprof/. (default "127.0.0.1")
--profiling-port string Profiling port for web interface host:port/debug/pprof/. (default "10011")
--root-dir string Root directory path for cri-containerd managed files (metadata checkpoint etc). (default "/var/lib/cri-containerd")
--sandbox-image string The image used by sandbox container. (default "gcr.io/google_containers/pause:3.0")
--socket-path string Path to the socket which cri-containerd serves on. (default "/var/run/cri-containerd.sock")
--stats-collect-period int The period (in seconds) of snapshots stats collection. (default 10)
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
--stream-addr string The ip address streaming server is listening on. The default host interface is used if not specified.
--stream-port string The port streaming server is listening on. (default "10010")
--systemd-cgroup Enables systemd cgroup support. By default not enabled.
-v, --v Level log level for V logs (default 0)
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
Use "cri-containerd [command] --help" for more information about a command.
cri-containerd.service: Main process exited, code=exited, status=1/FAILURE
cri-containerd.service: Unit entered failed state.
cri-containerd.service: Failed with result 'exit-code'.
cri-containerd.service: Service hold-off time over, scheduling restart.
Stopped Kubernetes containerd CRI shim.
Started Kubernetes containerd CRI shim.
I1203 19:46:57.847103 26057 cri_containerd.go:100] Run cri-containerd &{Config:{ContainerdConfig:{RootDir:/var/lib/containerd Snapshotter:overlayfs Endpoint:/run/containerd/containerd.sock Runtime:io.containerd.runtime.v1.linux RuntimeEngine: RuntimeRoot:} CniConfig:{NetworkPluginBinDir:/home/kubernetes/bin NetworkPluginConfDir:/etc/cni/net.d} SocketPath:/var/run/cri-containerd.sock RootDir:/var/lib/cri-containerd StreamServerAddress: StreamServerPort:10010 CgroupPath: EnableSelinux:false SandboxImage:gcr.io/google_containers/pause:3.0 StatsCollectPeriod:10 SystemdCgroup:false OOMScore:-999 EnableProfiling:true ProfilingPort:10011 ProfilingAddress:127.0.0.1} ConfigFilePath:/etc/cri-containerd/config.toml}
I1203 19:46:57.849122 26057 cri_containerd.go:104] Start profiling server
I1203 19:46:57.849152 26057 cri_containerd.go:108] Run cri-containerd grpc server on socket "/var/run/cri-containerd.sock"
I1203 19:46:57.858897 26057 service.go:155] Get device uuid "34c2d19c-feb1-4d79-86d0-088a3a41b931" for image filesystem "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs"
time="2017-12-03T19:46:57Z" level=info msg="CNI network k8s-pod-network (type=calico) is used from /etc/cni/net.d/10-calico.conflist"
time="2017-12-03T19:46:57Z" level=info msg="CNI network k8s-pod-network (type=calico) is used from /etc/cni/net.d/10-calico.conflist"
I1203 19:46:57.859761 26057 interface.go:360] Looking for default routes with IPv4 addresses
I1203 19:46:57.859778 26057 interface.go:365] Default route transits interface "eth0"
I1203 19:46:57.860133 26057 interface.go:174] Interface eth0 is up
I1203 19:46:57.860233 26057 interface.go:222] Interface "eth0" has 2 addresses :[10.128.0.6/32 fe80::4001:aff:fe80:6/64].
I1203 19:46:57.860257 26057 interface.go:189] Checking addr 10.128.0.6/32.
I1203 19:46:57.860267 26057 interface.go:196] IP found 10.128.0.6
I1203 19:46:57.860278 26057 interface.go:228] Found valid IPv4 address 10.128.0.6 for interface "eth0".
I1203 19:46:57.860287 26057 interface.go:371] Found active IP 10.128.0.6
I1203 19:46:57.860777 26057 service.go:182] Start cri-containerd service
I1203 19:46:57.860787 26057 service.go:184] Start recovering state
fatal error: concurrent map iteration and map write
goroutine 56996 [running]:
runtime.throw(0x55b96504e743, 0x26)
/usr/local/go/src/runtime/panic.go:605 +0x97 fp=0xc424083568 sp=0xc424083548 pc=0x55b964956d77
runtime.mapiternext(0xc424083668)
/usr/local/go/src/runtime/hashmap.go:778 +0x6f3 fp=0xc424083600 sp=0xc424083568 pc=0x55b964934d13
github.com/containerd/containerd/runtime.(*TaskList).GetAll(0xc4201ebe50, 0x55b9658db580, 0xc421c36c90, 0x55b964939c9a, 0x55b96545f680, 0x55b96547ed00, 0xc424083700, 0x55b9658db580)
/go/src/github.com/containerd/containerd/runtime/task_list.go:61 +0x110 fp=0xc4240836d8 sp=0xc424083600 pc=0x55b964ed8300
github.com/containerd/containerd/linux.(*Runtime).Tasks(0xc420061920, 0x55b9658db580, 0xc421c36c90, 0x55b9658db580, 0xc421c36c90, 0x0, 0x55b9658de600, 0xc4202a0400)
/go/src/github.com/containerd/containerd/linux/runtime.go:343 +0x45 fp=0xc424083728 sp=0xc4240836d8 pc=0x55b964f9e5c5
github.com/containerd/containerd/services/tasks.(*service).List(0xc4202107b0, 0x7f93231d0270, 0xc421c36c90, 0xc4239de370, 0xc4202107b0, 0x55b96504f86a, 0x4)
/go/src/github.com/containerd/containerd/services/tasks/service.go:280 +0x142 fp=0xc4240837f0 sp=0xc424083728 pc=0x55b964edba62
github.com/containerd/containerd/api/services/tasks/v1._Tasks_List_Handler.func1(0x7f93231d0270, 0xc421c36c90, 0x55b9654b6000, 0xc4239de370, 0xc423a54f50, 0x55b965938b38, 0x55b9653f83a0, 0xc4239de3b0)
/go/src/github.com/containerd/containerd/api/services/tasks/v1/tasks.pb.go:698 +0x88 fp=0xc424083838 sp=0xc4240837f0 pc=0x55b964ed4ce8
github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.UnaryServerInterceptor(0x7f93231d0270, 0xc421c36c90, 0x55b9654b6000, 0xc4239de370, 0xc4228ac160, 0xc4228ac180, 0x50, 0x48, 0xc4228ac140, 0xc424083968)
/go/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:29 +0xd4 fp=0xc4240838f0 sp=0xc424083838 pc=0x55b9650261c4
github.com/containerd/containerd/server.interceptor(0x7f93231d0270, 0xc421c36bd0, 0x55b9654b6000, 0xc4239de370, 0xc4228ac160, 0xc4228ac180, 0x0, 0xc4240839e0, 0x55b96493bfda, 0x50)
/go/src/github.com/containerd/containerd/server/server.go:267 +0x1d2 fp=0xc424083978 sp=0xc4240838f0 pc=0x55b96502a482
github.com/containerd/containerd/api/services/tasks/v1._Tasks_List_Handler(0x55b965509840, 0xc4202107b0, 0x7f93231d0270, 0xc421c36ab0, 0xc423a54e10, 0x55b965521428, 0x0, 0x0, 0x1000, 0x0)
/go/src/github.com/containerd/containerd/api/services/tasks/v1/tasks.pb.go:700 +0x16f fp=0xc4240839f0 sp=0xc424083978 pc=0x55b964eb5c8f
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc420081e00, 0x55b9658de6c0, 0xc420346000, 0xc4218c8a00, 0xc420211110, 0x55b9658c5d58, 0x0, 0x0, 0x0)
/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:826 +0xab6 fp=0xc424083d90 sp=0xc4240839f0 pc=0x55b964dc1816
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleStream(0xc420081e00, 0x55b9658de6c0, 0xc420346000, 0xc4218c8a00, 0x0)
/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1023 +0x152a fp=0xc424083f80 sp=0xc424083d90 pc=0x55b964dc505a
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc42030e0d0, 0xc420081e00, 0x55b9658de6c0, 0xc420346000, 0xc4218c8a00)
/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:572 +0xa1 fp=0xc424083fb8 sp=0xc424083f80 pc=0x55b964dcb0e1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc424083fc0 sp=0xc424083fb8 pc=0x55b964989051
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:570 +0xa3
goroutine 1 [chan receive, 10 minutes]:
main.main.func1(0xc42009d340, 0xc42009d340, 0xc42004bb4f)
/go/src/github.com/containerd/containerd/cmd/containerd/main.go:134 +0x878
github.com/containerd/containerd/vendor/github.com/urfave/cli.HandleAction(0x55b96540c700, 0x55b965522638, 0xc42009d340, 0xc420060d20, 0x0)
/go/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:502 +0xd4
github.com/containerd/containerd/vendor/github.com/urfave/cli.(*App).Run(0xc42016ca80, 0xc420010090, 0x3, 0x3, 0x0, 0x0)
/go/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:268 +0x655
main.main()
...
Log on cri-containerd side:
Log on containerd side:
containerd.log: https://storage.googleapis.com/kubernetes-jenkins/logs/ci-cri-containerd-e2e-gci-gce/628/build-log.txt
cri-containerd.log: https://storage.googleapis.com/kubernetes-jenkins/logs/ci-cri-containerd-e2e-gci-gce/628/artifacts/bootstrap-e2e-minion-group-xnmw/cri-containerd.log