Dockerd panic: [2023-10-30T19:05:44.292642262Z][dockerd][I] panic: runtime error: invalid memory address or nil pointer dereference
[2023-10-30T19:05:44.292697769Z][dockerd][I] [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2add536]
[2023-10-30T19:05:44.292705699Z][dockerd][I]
[2023-10-30T19:05:44.292705963Z][dockerd][I] goroutine 297251 [running]:
[2023-10-30T19:05:44.292711163Z][dockerd][I] io.(*PipeWriter).Write(...)
[2023-10-30T19:05:44.292711417Z][dockerd][I] /usr/local/go/src/io/pipe.go:165
[2023-10-30T19:05:44.292713012Z][dockerd][I] github.com/containerd/containerd/remotes/docker.(*pushWriter).Commit(0xc001565570, {0x0?, 0xc34e40?}, 0x0, {0xc017f28a00, 0x47}, {0xc1481d821119c3ee?, 0x7a38d2c3f5?, 0x3a605a0?})
[2023-10-30T19:05:44.292714437Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/remotes/docker/pusher.go:441 +0x76
[2023-10-30T19:05:44.292716408Z][dockerd][I] github.com/containerd/containerd/content.Copy({0xc5cac0, 0xc018140c30}, {0xc63168, 0xc001565570}, {0xc3d3e0, 0xc0180fe090}, 0x0, {0xc017f28a00, 0x47}, {0x0, ...})
[2023-10-30T19:05:44.292716948Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/content/helpers.go:186 +0x3df
[2023-10-30T19:05:44.292718684Z][dockerd][I] github.com/containerd/containerd/remotes.push({0xc5cac0, 0xc018140c30}, {0x7f16504bff18, 0xc018087a70}, {0xc3e9c0?, 0xc0180ce8a0?}, {{0xc017f30990, 0x2c}, {0xc017f28a00, 0x47}, ...})
[2023-10-30T19:05:44.292718930Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/remotes/handlers.go:199 +0x5bc
[2023-10-30T19:05:44.292720358Z][dockerd][I] github.com/containerd/containerd/remotes.PushHandler.func1({0xc5ca18, 0xc017f47a90}, {{0xc017f30990, 0x2c}, {0xc017f28a00, 0x47}, 0x0, {0x0, 0x0, 0x0}, ...})
[2023-10-30T19:05:44.292720937Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/remotes/handlers.go:166 +0x2a5
[2023-10-30T19:05:44.292722473Z][dockerd][I] github.com/containerd/containerd/images.HandlerFunc.Handle(0xc017f28a00?, {0xc5ca18?, 0xc017f47a90?}, {{0xc017f30990, 0x2c}, {0xc017f28a00, 0x47}, 0x0, {0x0, 0x0, ...}, ...})
[2023-10-30T19:05:44.292722818Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/images/handlers.go:59 +0x7f
[2023-10-30T19:05:44.292724433Z][dockerd][I] github.com/containerd/containerd/images.Handlers.func1({0xc5ca18, 0xc017f47a90}, {{0xc017f30990, 0x2c}, {0xc017f28a00, 0x47}, 0x0, {0x0, 0x0, 0x0}, ...})
[2023-10-30T19:05:44.292724687Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/images/handlers.go:69 +0x15e
[2023-10-30T19:05:44.292762509Z][dockerd][I] github.com/containerd/containerd/images.HandlerFunc.Handle(0xc3c901?, {0xc5ca18?, 0xc017f47a90?}, {{0xc017f30990, 0x2c}, {0xc017f28a00, 0x47}, 0x0, {0x0, 0x0, ...}, ...})
[2023-10-30T19:05:44.292763387Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/images/handlers.go:59 +0x7f
[2023-10-30T19:05:44.292765983Z][dockerd][I] github.com/containerd/containerd/images.Handlers.func1({0xc5ca18, 0xc017f47a90}, {{0xc017f30990, 0x2c}, {0xc017f28a00, 0x47}, 0x0, {0x0, 0x0, 0x0}, ...})
[2023-10-30T19:05:44.292766218Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/images/handlers.go:69 +0x15e
[2023-10-30T19:05:44.292767842Z][dockerd][I] github.com/containerd/containerd/images.HandlerFunc.Handle(0xc001121ec8?, {0xc5ca18?, 0xc017f47a90?}, {{0xc017f30990, 0x2c}, {0xc017f28a00, 0x47}, 0x0, {0x0, 0x0, ...}, ...})
[2023-10-30T19:05:44.292768075Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/images/handlers.go:59 +0x7f
[2023-10-30T19:05:44.292769615Z][dockerd][I] github.com/containerd/containerd/images.Dispatch.func1()
[2023-10-30T19:05:44.292770438Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/images/handlers.go:168 +0xd6
[2023-10-30T19:05:44.292772048Z][dockerd][I] golang.org/x/sync/errgroup.(*Group).Go.func1()
[2023-10-30T19:05:44.292772303Z][dockerd][I] /go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
[2023-10-30T19:05:44.292774110Z][dockerd][I] created by golang.org/x/sync/errgroup.(*Group).Go
[2023-10-30T19:05:44.292774360Z][dockerd][I] /go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5
[2023-10-30T19:05:44.344824989Z][dockerd][I] EOF
Obviously we would love to keep the faster behavior - but a crash should be resolved if nothing else. #44369 could allow us to correctly probe the daemon storage to see if we should do the work or not.
Description
Found by @natalieparellano when testing containerd beta. https://github.com/buildpacks/imgutil has this undocumented fast path for
ImageLoad. The idea, I believe, is that if we think the base layers should exist in the daemon we can send a 0 size tar header for those layers and the overlay storage system would end up finding the referenced layers by diffid I believe. The new containerd storage layer hard crashes the daemon with an EOF.Reproduce
Result:
Expected behavior
I would expect the
ImageLoadto fail with an error, allowing the caller to handle the error. In this caseimgutilhas a slow path for downloading base layers if we fail to load the image into the daemon.Obviously we would love to keep the faster behavior - but a crash should be resolved if nothing else. #44369 could allow us to correctly probe the daemon storage to see if we should do the work or not.
docker version
docker info
Additional Info
No response