Skip to content

[Regression on main] Segfaults at runtime/restart/monitor/monitor.go:226 #6831

@AkihiroSuda

Description

@AkihiroSuda

Description

The restart monitor segfaults during restarting the daemon.
Seems a regression in #6744

# journalctl -u containerd -o cat -f
containerd.service: Scheduled restart job, restart counter is at 19.
Stopped containerd container runtime.
Starting containerd container runtime...
time="2022-04-20T15:30:40+09:00" level=warning msg="deprecated version : `1`, please switch to version `2`"
time="2022-04-20T15:30:40.197183901+09:00" level=info msg="starting containerd" revision=fe7f0aeef42b9cd8c5b0c45f889bbbf9ad766ab2 version=v1.6.0-276-gfe7f0aeef
time="2022-04-20T15:30:40.213830418+09:00" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
time="2022-04-20T15:30:40.213929276+09:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
time="2022-04-20T15:30:40.215907767+09:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/5.13.0-39-generic\\n\"): skip plugin" type=io.containerd.snapshotter.v1
time="2022-04-20T15:30:40.215941253+09:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
time="2022-04-20T15:30:40.216112999+09:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2022-04-20T15:30:40.216126889+09:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
time="2022-04-20T15:30:40.216136520+09:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2022-04-20T15:30:40.216145048+09:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
time="2022-04-20T15:30:40.216165051+09:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
time="2022-04-20T15:30:40.216286150+09:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
time="2022-04-20T15:30:40.216400450+09:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2022-04-20T15:30:40.216424586+09:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.stargz\"..." type=io.containerd.snapshotter.v1
time="2022-04-20T15:30:40.216523275+09:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
time="2022-04-20T15:30:40.216544861+09:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
time="2022-04-20T15:30:40.216554152+09:00" level=info msg="metadata content store policy set" policy=shared
time="2022-04-20T15:30:40.216683336+09:00" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
time="2022-04-20T15:30:40.216705589+09:00" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
time="2022-04-20T15:30:40.216714500+09:00" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
time="2022-04-20T15:30:40.216734656+09:00" level=info msg="loading plugin \"io.containerd.lease.v1.manager\"..." type=io.containerd.lease.v1
time="2022-04-20T15:30:40.216744862+09:00" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
time="2022-04-20T15:30:40.216757520+09:00" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
time="2022-04-20T15:30:40.216767736+09:00" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
time="2022-04-20T15:30:40.216777474+09:00" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
time="2022-04-20T15:30:40.216787070+09:00" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
time="2022-04-20T15:30:40.216798446+09:00" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
time="2022-04-20T15:30:40.216808224+09:00" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
time="2022-04-20T15:30:40.216867167+09:00" level=info msg="loading plugin \"io.containerd.runtime.v2.shim\"..." type=io.containerd.runtime.v2
time="2022-04-20T15:30:40.216877084+09:00" level=info msg="loading plugin \"io.containerd.service.v1.sandbox-controller-service\"..." type=io.containerd.service.v1
time="2022-04-20T15:30:40.216885808+09:00" level=info msg="loading plugin \"io.containerd.service.v1.sandbox-store-service\"..." type=io.containerd.service.v1
time="2022-04-20T15:30:40.216893933+09:00" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
time="2022-04-20T15:30:40.216903377+09:00" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
time="2022-04-20T15:30:40.216929797+09:00" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
time="2022-04-20T15:30:40.217188514+09:00" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
time="2022-04-20T15:30:40.217213216+09:00" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
time="2022-04-20T15:30:40.217222681+09:00" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
time="2022-04-20T15:30:40.217271210+09:00" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
time="2022-04-20T15:30:40.217280999+09:00" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
time="2022-04-20T15:30:40.217290514+09:00" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
time="2022-04-20T15:30:40.217297809+09:00" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
time="2022-04-20T15:30:40.217305978+09:00" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
time="2022-04-20T15:30:40.217314295+09:00" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
time="2022-04-20T15:30:40.217322018+09:00" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
time="2022-04-20T15:30:40.217329491+09:00" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
time="2022-04-20T15:30:40.217341320+09:00" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
time="2022-04-20T15:30:40.217378328+09:00" level=info msg="loading plugin \"io.containerd.grpc.v1.sandbox-controllers\"..." type=io.containerd.grpc.v1
time="2022-04-20T15:30:40.217391110+09:00" level=info msg="loading plugin \"io.containerd.grpc.v1.sandboxes\"..." type=io.containerd.grpc.v1
time="2022-04-20T15:30:40.217399800+09:00" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
time="2022-04-20T15:30:40.217408926+09:00" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
time="2022-04-20T15:30:40.217417179+09:00" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
time="2022-04-20T15:30:40.217425356+09:00" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x98 pc=0x56380058ee32]
goroutine 27 [running]:
github.com/containerd/containerd/runtime/restart/monitor.(*monitor).monitor(0xc000010228, {0x56380169c2e8, 0xc0007ae0c0})
        /home/suda/gopath/src/github.com/containerd/containerd/runtime/restart/monitor/monitor.go:226 +0x272
github.com/containerd/containerd/runtime/restart/monitor.(*monitor).reconcile.func1()
        /home/suda/gopath/src/github.com/containerd/containerd/runtime/restart/monitor/monitor.go:187 +0xab
created by github.com/containerd/containerd/runtime/restart/monitor.(*monitor).reconcile
        /home/suda/gopath/src/github.com/containerd/containerd/runtime/restart/monitor/monitor.go:184 +0x18c
containerd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
containerd.service: Failed with result 'exit-code'.
Failed to start containerd container runtime.

Steps to reproduce the issue

# nerdctl version
Client:
 Version:       v0.18.0
 Git commit:    77276ff0fffad3f855ab9f2f5a4ad5527ef76485

Server:
 containerd:
  Version:      v1.6.0-276-gfe7f0aeef
  GitCommit:    fe7f0aeef42b9cd8c5b0c45f889bbbf9ad766ab2

# nerdctl run -d --restart=always --name nginx -p 80:80 nginx:alpine
cd753d109812f9bb06f609db858e32cea1c01bc4ed72382709db4d5f9a9195f2

# systemctl kill -s KILL containerd

# journalctl -u containerd -o cat -f
...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x98 pc=0x56380058ee32]
goroutine 27 [running]:
github.com/containerd/containerd/runtime/restart/monitor.(*monitor).monitor(0xc000010228, {0x56380169c2e8, 0xc0007ae0c0})
        /home/suda/gopath/src/github.com/containerd/containerd/runtime/restart/monitor/monitor.go:226 +0x272
github.com/containerd/containerd/runtime/restart/monitor.(*monitor).reconcile.func1()
        /home/suda/gopath/src/github.com/containerd/containerd/runtime/restart/monitor/monitor.go:187 +0xab
created by github.com/containerd/containerd/runtime/restart/monitor.(*monitor).reconcile
        /home/suda/gopath/src/github.com/containerd/containerd/runtime/restart/monitor/monitor.go:184 +0x18c
containerd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
containerd.service: Failed with result 'exit-code'.
Failed to start containerd container runtime.

Describe the results you received and expected

Received: segfault
Expected: shout not segfault

What version of containerd are you using?

containerd github.com/containerd/containerd v1.6.0-276-gfe7f0aeef fe7f0ae

Any other relevant information

No response

Show configuration if it is related to CRI plugin.

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions