Containerd panics with fatal error: concurrent map writes when pull images concurrently #8742
Description
Description
Containerd paniced when during concurrent image pulling. I believe the following is the root cause, but feel free to correct me if I missed anything:
Since c.config.Registry.Headers is a global map, concurrent image pull could result in concurrent map.Set
containerd/pkg/cri/server/image_pull.go
Line 133 in 9b4ed8a
options.Headers.Set("User-Agent", "containerd/"+version.Version) could result to concurrent map write
https://github.com/containerd/containerd/blob/9b4ed8acc2a04a3f8df140e79052d18b750d757e/remotes/docker/resolver.go#L153C1-L153C1
Steps to reproduce the issue
We had a client using CRI client calling PullImage almost at the same time and can get a repro.
Describe the results you received and expected
Got a panic when pulling images concurrently when calling CRI.PullImage API. Expecting pullImage to be thread safe
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: fatal error: concurrent map writes
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: goroutine 339 [running]:
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: runtime.throw({0x55c6b3076599?, 0x10?})
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /usr/local/go/src/runtime/panic.go:992 +0x71 fp=0xc000174c08 sp=0xc000174bd8 pc=0x55c6b1df5e91
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: runtime.mapassign_faststr(0x55c6b306211c?, 0x55c6b306404b?, {0x55c6b306211c, 0xa})
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /usr/local/go/src/runtime/map_faststr.go:212 +0x39c fp=0xc000174c70 sp=0xc000174c08 pc=0x55c6b1dd21bc
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: net/textproto.MIMEHeader.Set(...)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /usr/local/go/src/net/textproto/header.go:22
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: net/http.Header.Set(...)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /usr/local/go/src/net/http/header.go:40
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/containerd/containerd/remotes/docker.NewResolver({0xc0006b08b8, 0xc000038750, {0x7efc08032300, 0xc0006b08d0}, {0x0, 0x0}, 0x0, 0x0, 0x0, 0x0})
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/remotes/docker/resolver.go:145 +0x225 fp=0xc000174e48 sp=0xc000174c70 pc=0x55c6b25724a5
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/containerd/containerd/pkg/cri/server.(*criService).PullImage(0xc000601800, {0x55c6b3902d70, 0xc00055a0c0}, 0xc00082c180)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/pkg/cri/server/image_pull.go:104 +0x2a5 fp=0xc000175228 sp=0xc000174e48 pc=0x55c6b2f17825
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/containerd/containerd/pkg/cri/server.(*instrumentedAlphaService).PullImage(0xc00011a7b0, {0x55c6b3902d70, 0xc0005b3e90}, 0xc000634f00)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/pkg/cri/server/instrumented_service.go:966 +0x277 fp=0xc000175480 sp=0xc000175228 pc=0x55c6b2f2d1f7
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: k8s.io/cri-api/pkg/apis/runtime/v1alpha2._ImageService_PullImage_Handler.func1({0x55c6b3902d70, 0xc0005b3e90}, {0x55c6b3871400?, 0xc000634f00})
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/vendor/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.pb.go:9855 +0x78 fp=0xc0001754c0 sp=0xc000175480 pc=0x55c6b2de24d8
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/containerd/containerd/services/server.unaryNamespaceInterceptor({0x55c6b3902d70, 0xc0005b3e90}, {0x55c6b3871400, 0xc000634f00}, 0x3?, 0xc0006b07e0)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/services/server/namespace.go:31 +0x6b fp=0xc0001754f0 sp=0xc0001754c0 pc=0x55c6b3012dcb
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x55c6b3902d70?, 0xc0005b3e90?}, {0x55c6b3871400?, 0xc000634f00?})
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25 +0x3a fp=0xc000175530 sp=0xc0001754f0 pc=0x55c6b3009a9a
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1({0x55c6b3902d70, 0xc0005b3e90}, {0x55c6b3871400, 0xc000634f00}, 0x0?, 0xc00082c020)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107 +0x87 fp=0xc000175590 sp=0xc000175530 pc=0x55c6b300c5e7
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x55c6b3902d70?, 0xc0005b3e90?}, {0x55c6b3871400?, 0xc000634f00?})
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25 +0x3a fp=0xc0001755d0 sp=0xc000175590 pc=0x55c6b3009a9a
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x55c6b3902d70, 0xc0005b3c20}, {0x55c6b3871400, 0xc000634f00}, 0xc00082c000, 0xc00082c040)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:325 +0x664 fp=0xc000175a10 sp=0xc0001755d0 pc=0x55c6b3010204
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x55c6b3902d70?, 0xc0005b3c20?}, {0x55c6b3871400?, 0xc000634f00?})
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25 +0x3a fp=0xc000175a50 sp=0xc000175a10 pc=0x55c6b3009a9a
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x55c6b3902d70, 0xc0005b3c20}, {0x55c6b3871400, 0xc000634f00}, 0xc000470af0?, 0x55c6b36b3620?)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0xbf fp=0xc000175aa8 sp=0xc000175a50 pc=0x55c6b300993f
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: k8s.io/cri-api/pkg/apis/runtime/v1alpha2._ImageService_PullImage_Handler({0x55c6b38a8f40?, 0xc00011a7b0}, {0x55c6b3902d70, 0xc0005b3c20}, 0xc000516480, 0xc0002e41b0)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/vendor/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.pb.go:9857 +0x138 fp=0xc000175b00 sp=0xc000175aa8 pc=0x55c6b2de2398
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: google.golang.org/grpc.(*Server).processUnaryRPC(0xc000140a80, {0x55c6b3908ba0, 0xc0006c3520}, 0xc000638120, 0xc000158600, 0x55c6b44e71f0, 0x0)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1283 +0xcfd fp=0xc000175e48 sp=0xc000175b00 pc=0x55c6b2458e5d
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: google.golang.org/grpc.(*Server).handleStream(0xc000140a80, {0x55c6b3908ba0, 0xc0006c3520}, 0xc000638120, 0x0)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1620 +0xa1b fp=0xc000175f68 sp=0xc000175e48 pc=0x55c6b245d4bb
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: google.golang.org/grpc.(*Server).serveStreams.func1.2()
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:922 +0x98 fp=0xc000175fe0 sp=0xc000175f68 pc=0x55c6b2456978
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: runtime.goexit()
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /usr/local/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000175fe8 sp=0xc000175fe0 pc=0x55c6b1e2abc1
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:920 +0x28a
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: goroutine 1 [chan receive]:
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/containerd/containerd/cmd/containerd/command.App.func1(0xc00016e2c0?)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/cmd/containerd/command/main.go:279 +0x134c
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/urfave/cli.HandleAction({0x55c6b361e2e0?, 0x55c6b38dfac8?}, 0xc00003c230?)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:523 +0xa8
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/urfave/cli.(*App).Run(0xc000140700, {0xc00003c230, 0x1, 0x1})
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:285 +0x818
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: main.main()
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/containerd/containerd/cmd/containerd/main.go:33 +0x37
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: goroutine 11 [chan receive]:
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: k8s.io/klog/v2.(*loggingT).flushDaemon(0x0?)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/vendor/k8s.io/klog/v2/klog.go:1181 +0x6a
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: created by k8s.io/klog/v2.init.0
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: /go/src/github.com/containerd/containerd/vendor/k8s.io/klog/v2/klog.go:420 +0xf6
What version of containerd are you using?
1.6.9.4
Any other relevant information
No response
Show configuration if it is related to CRI plugin.
No response