Skip to content

LXD 5.21.2 crashed  #14706

New issue
New issue
Closed
Bug
#14771
Closed
LXD 5.21.2 crashed #14706
Bug
#14771
Assignees
tomponline
Milestone
lxd-6.3
@mastier

Description

@mastier
mastier
opened on Dec 20, 2024

Required information

  • Distribution: Ubuntu
  • Distribution version: 22.04
  • The output of "snap list --all lxd core20 core22 core24 snaps":
$ snap list --all lxd core20 core22 core24 snaps
Name    Version         Rev    Tracking       Publisher   Notes
core20  20240911        2434   latest/stable  canonical✓  base
core22  20241119        1722   latest/stable  canonical✓  base
core24  20240920        609    latest/stable  canonical✓  base
lxd     5.0.4-497fe1e   31333  5.21/stable    canonical✓  disabled
lxd     5.21.2-084c8c8  31214  5.21/stable    canonical✓  -
  • The output of "lxc info" or if that fails:
    • Kernel version: 6.8.0-49-generic
    • LXC version: 5.21.2 LTS
    • LXD version: 5.21.2 LTS
    • Storage backend in use: dir

Issue description

I was deploying Openstack cloud with Juju. Some containers on one of the nodes failed to get the IP address (4 of them).
After restarting containers the could get the address normally.
Around the time of the deployment I saw that LXD crashed in the logs. So that might be related

Dec 19 17:54:12 controller-2 lxd.daemon[17490]: => First LXD execution on this system
Dec 19 17:54:12 controller-2 lxd.daemon[17490]: => LXD is ready
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: fatal error: concurrent map read and map write
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: goroutine 2928 [running]:
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: main.hoistReq.func1({0x265fe88, 0xc000279340}, 0xc00082f900)
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         github.com/canonical/lxd/lxd/devlxd.go:277 +0xf0
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: net/http.HandlerFunc.ServeHTTP(0xc00082f540?, {0x265fe88?, 0xc000279340?}, 0x538389?)
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         net/http/server.go:2220 +0x29
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: github.com/gorilla/mux.(*Router).ServeHTTP(0xc0007a6600, {0x265fe88, 0xc000279340}, 0xc00082f180)
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         github.com/gorilla/[email protected]/mux.go:212 +0x1e2
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: net/http.serverHandler.ServeHTTP({0xc000a1c330?}, {0x265fe88?, 0xc000279340?}, 0x6?)
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         net/http/server.go:3210 +0x8e
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: net/http.(*conn).serve(0xc0007fb440, {0x26647e8, 0xc000a1c2d0})
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         net/http/server.go:2092 +0x5d0
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: created by net/http.(*Server).Serve in goroutine 131
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         net/http/server.go:3360 +0x485
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: goroutine 1 [select, 2 minutes]:
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: main.(*cmdDaemon).Run(0xc0012bc078, 0xc00119f998?, {0xc0012cbac0, 0x0, 0x0?})
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         github.com/canonical/lxd/lxd/main_daemon.go:83 +0x5b2
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: github.com/spf13/cobra.(*Command).execute(0xc0012ce008, {0xc0001c8010, 0x4, 0x4})
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         github.com/spf13/[email protected]/command.go:985 +0xaaa
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: github.com/spf13/cobra.(*Command).ExecuteC(0xc0012ce008)
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         github.com/spf13/[email protected]/command.go:1117 +0x3ff
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: github.com/spf13/cobra.(*Command).Execute(...)
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         github.com/spf13/[email protected]/command.go:1041
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: main.main()
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         github.com/canonical/lxd/lxd/main.go:225 +0x1d98
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: goroutine 98 [syscall, 2 minutes]:
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: os/signal.signal_recv()
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         runtime/sigqueue.go:152 +0x29
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: os/signal.loop()
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         os/signal/signal_unix.go:23 +0x13
ec 19 17:56:33 controller-2 lxd.daemon[17656]: created by os/signal.Notify.func1.1 in goroutine 1
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         os/signal/signal.go:151 +0x1f
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: goroutine 32 [select, 2 minutes]:
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: github.com/mdlayher/socket.rwT[...].func2()
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         github.com/mdlayher/[email protected]/conn.go:778 +0xb5
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: created by github.com/mdlayher/socket.rwT[...] in goroutine 132
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         github.com/mdlayher/[email protected]/conn.go:775 +0x4f9
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: goroutine 43 [select, 2 minutes]:
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: database/sql.(*DB).connectionOpener(0xc00081b790, {0x2664820, 0xc000052190})
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         database/sql/sql.go:1253 +0x87
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: created by database/sql.OpenDB in goroutine 1
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         database/sql/sql.go:833 +0x130
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: goroutine 130 [IO wait]:
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: internal/poll.runtime_pollWait(0x7d9290128990, 0x72)
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         runtime/netpoll.go:351 +0x85
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: internal/poll.(*pollDesc).wait(0xc000cb6500?, 0xc00021d180?, 0x0)
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         internal/poll/fd_poll_runtime.go:84 +0x27
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: internal/poll.(*pollDesc).waitRead(...)
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         internal/poll/fd_poll_runtime.go:89
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: internal/poll.(*FD).Accept(0xc000cb6500)
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         internal/poll/fd_unix.go:620 +0x295
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: net.(*netFD).accept(0xc000cb6500)
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         net/fd_unix.go:172 +0x29
Dec 19 17:56:33 controller-2 lxd.daemon[17656]: net.(*UnixListener).accept(0x41dee5?)
Dec 19 17:56:33 controller-2 lxd.daemon[17656]:         net/unixsock_posix.go:172 +0x16

Steps to reproduce

None in particular, I guess you would need to check the traceback

Information to attach

  • Any relevant kernel output (dmesg)
  • [ x] Container log (lxc info NAME --show-log)
    One of the containers that requres restart
$ sudo lxc info --show-log juju-920055-4-lxd-6

   CPU usage (in seconds): 123
 Memory usage:
   Memory (current): 410.77MiB
 Network usage:
   eth0:
     Type: broadcast
     State: UP
     Host interface: veth9a0a9126
     MAC address: 00:16:3e:d0:1f:6b
     MTU: 9000
     Bytes received: 1.27MB
     Bytes sent: 1.22kB
     Packets received: 9519
     Packets sent: 16
     IP addresses:
       inet6: fe80::216:3eff:fed0:1f6b/64 (link)
   eth1:
     Type: broadcast
     State: UP
     Host interface: vethca47b59e
     MAC address: 00:16:3e:10:38:6e
     MTU: 1500
     Bytes received: 183.02kB
     Bytes sent: 1.76kB
     Packets received: 2953
     Packets sent: 24
     IP addresses:
       inet:  10.50.221.4/24 (global)
       inet6: fe80::216:3eff:fe10:386e/64 (link)
   eth2:
     Type: broadcast
     State: UP
     Host interface: vethac2c02bc
     MAC address: 00:16:3e:e4:54:14
     MTU: 1500
     Bytes received: 72.75MB
     Bytes sent: 824.27kB
     Packets received: 18909
     Packets sent: 7474
     IP addresses:
       inet:  10.50.193.130/24 (global)
       inet6: fe80::216:3eff:fee4:5414/64 (link)
   lo:
     Type: loopback
     State: UP
     MTU: 65536
     Bytes received: 1.32MB
     Bytes sent: 1.32MB
     Packets received: 17212
     Packets sent: 17212
     IP addresses:
     inet:  127.0.0.1/8 (local)
     inet6: ::1/128 (local)

Log:

lxc juju-920055-4-lxd-6 20241220103937.512 WARN     idmap_utils - ../src/src/lxc/idmap_utils.c:lxc_map_ids:165 - newuidmap binary is missing
lxc juju-920055-4-lxd-6 20241220103937.513 WARN     idmap_utils - ../src/src/lxc/idmap_utils.c:lxc_map_ids:171 - newgidmap binary is missing
lxc juju-920055-4-lxd-6 20241220103937.514 WARN     idmap_utils - ../src/src/lxc/idmap_utils.c:lxc_map_ids:165 - newuidmap binary is missing
lxc juju-920055-4-lxd-6 20241220103937.514 WARN     idmap_utils - ../src/src/lxc/idmap_utils.c:lxc_map_ids:171 - newgidmap binary is missing

  • [x ] Container configuration (lxc config show NAME --expanded)

  • [ x ] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)

ubuntu@controller-2:~$ sudo cat /var/snap/lxd/common/lxd/logs/lxd.log
time="2024-12-19T17:56:35Z" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
ubuntu@controller-2:~$ sudo cat /var/snap/lxd/common/lxd/logs/lxd.log.1
time="2024-12-19T17:54:08Z" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
  • Output of the client with --debug
  • Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Bug

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions