Is it platform specific
broadcom
Importance or Severity
Medium
Description of the bug
Below error seen on sonic-mgmt voq tests (test_voq_chassis_app_db_consistency, test_voq_disrupts):
2025 May 29 19:12:10.954804 ixr-x3b-5 NOTICE kernel: [ 0.017736] Kernel command line: BOOT_IMAGE=/image-HEAD.1044465-msft-2503-5a2624582/boot/vmlinuz-6.1.0-29-2-amd64 root=UUID=99f1be09-5129-4845-9da1-ef081a968521 rw console=tty0 console=ttyS0,115200n8 quiet processor.max_cstate=1 amd_idle.max_cstate=0 net.ifnames=0 biosdevname=0 loop=image-HEAD.1044465-msft-2503-5a2624582/fs.squashfs loopfstype=squashfs systemd.unified_cgroup_hierarchy=0 apparmor=1 security=apparmor varlog_size=4096 usbcore.autosuspend=-1 amd_iommu=off crashkernel=8G-:1G idle=poll
2025 May 29 19:12:16.538690 ixr-x3b-5 ERR dockerd: nss_tacplus: failed to connect TACACS+ server 10.250.17.253:49, ret=-8: Operation now in progress
2025 May 29 19:12:16.538758 ixr-x3b-5 ERR dockerd: tac_connect_single: connection to 10.250.17.253:49 failed: Network is unreachable
2025 May 29 19:12:16.538806 ixr-x3b-5 ERR dockerd: nss_tacplus: failed to connect TACACS+ server 10.250.17.253:49, ret=-9: Network is unreachable
2025 May 29 19:12:26.655748 ixr-x3b-5 ERR libnetwork-setkey: tac_connect_single: connection to 10.250.17.253:49 failed: Network is unreachable
2025 May 29 19:12:26.655828 ixr-x3b-5 ERR libnetwork-setkey: nss_tacplus: failed to connect TACACS+ server 10.250.17.253:49, ret=-9: Network is unreachable
2025 May 29 19:12:26.655871 ixr-x3b-5 ERR libnetwork-setkey: tac_connect_single: connection to 10.250.17.253:49 failed: Network is unreachable
2025 May 29 19:12:26.655912 ixr-x3b-5 ERR libnetwork-setkey: nss_tacplus: failed to connect TACACS+ server 10.250.17.253:49, ret=-9: Network is unreachable
2025 May 29 19:12:26.656455 ixr-x3b-5 ERR libnetwork-setkey: tac_connect_single: connection to 10.250.17.253:49 failed: Network is unreachable
2025 May 29 19:12:26.656536 ixr-x3b-5 ERR libnetwork-setkey: nss_tacplus: failed to connect TACACS+ server 10.250.17.253:49, ret=-9: Network is unreachable
2025 May 29 19:12:26.656577 ixr-x3b-5 ERR libnetwork-setkey: tac_connect_single: connection to 10.250.17.253:49 failed: Network is unreachable
2025 May 29 19:12:26.656615 ixr-x3b-5 ERR libnetwork-setkey: nss_tacplus: failed to connect TACACS+ server 10.250.17.253:49, ret=-9: Network is unreachable
Steps to Reproduce
1.Execute the below tests :
test_voq_chassis_app_db_consistency.py & test_voq_disrupts.py
2. Verify the syslog after reboot is triggered.
Actual Behavior and Expected Behavior
This error occurs only on reboot. During the up -path when the system starts coming up, TACACS+ server reachability check is done very early in the process and hence, errors out
Relevant log output
2025 May 29 19:12:26.459550 ixr-x3b-5 INFO database.sh[2227]: Starting existing database0 container
2025 May 29 19:12:26.462068 ixr-x3b-5 INFO database.sh[2228]: Starting existing database1 container
2025 May 29 19:12:26.489973 ixr-x3b-5 INFO (udev-worker)[2379]: Network interface NamePolicy= disabled on kernel command line.
2025 May 29 19:12:26.490214 ixr-x3b-5 INFO kernel: [ 20.857267] docker0: port 1(vethcc84c68) entered blocking state
2025 May 29 19:12:26.490227 ixr-x3b-5 INFO kernel: [ 20.857274] docker0: port 1(vethcc84c68) entered disabled state
2025 May 29 19:12:26.490229 ixr-x3b-5 INFO kernel: [ 20.857361] device vethcc84c68 entered promiscuous mode
2025 May 29 19:12:26.490351 ixr-x3b-5 INFO (udev-worker)[2378]: Network interface NamePolicy= disabled on kernel command line.
2025 May 29 19:12:26.491083 ixr-x3b-5 INFO dockerd[1223]: time="2025-05-29T19:12:26.490704531Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
2025 May 29 19:12:26.491129 ixr-x3b-5 INFO dockerd[1223]: time="2025-05-29T19:12:26.490739231Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
2025 May 29 19:12:26.493824 ixr-x3b-5 INFO (udev-worker)[2382]: Network interface NamePolicy= disabled on kernel command line.
2025 May 29 19:12:26.494213 ixr-x3b-5 INFO kernel: [ 20.863201] docker0: port 2(vethada0afc) entered blocking state
2025 May 29 19:12:26.494221 ixr-x3b-5 INFO kernel: [ 20.863209] docker0: port 2(vethada0afc) entered disabled state
2025 May 29 19:12:26.494223 ixr-x3b-5 INFO kernel: [ 20.863332] device vethada0afc entered promiscuous mode
2025 May 29 19:12:26.494224 ixr-x3b-5 INFO kernel: [ 20.863508] docker0: port 2(vethada0afc) entered blocking state
2025 May 29 19:12:26.494226 ixr-x3b-5 INFO kernel: [ 20.863514] docker0: port 2(vethada0afc) entered forwarding state
2025 May 29 19:12:26.496758 ixr-x3b-5 INFO dockerd[1223]: time="2025-05-29T19:12:26.496671013Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
2025 May 29 19:12:26.496826 ixr-x3b-5 INFO dockerd[1223]: time="2025-05-29T19:12:26.496708063Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
2025 May 29 19:12:26.506222 ixr-x3b-5 INFO kernel: [ 20.876123] docker0: port 2(vethada0afc) entered disabled state
2025 May 29 19:12:26.506716 ixr-x3b-5 INFO (udev-worker)[2383]: Network interface NamePolicy= disabled on kernel command line.
2025 May 29 19:12:26.536525 ixr-x3b-5 INFO containerd[1049]: time="2025-05-29T19:12:26.535989378Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
2025 May 29 19:12:26.536587 ixr-x3b-5 INFO containerd[1049]: time="2025-05-29T19:12:26.536076358Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
2025 May 29 19:12:26.536619 ixr-x3b-5 INFO containerd[1049]: time="2025-05-29T19:12:26.536094418Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
2025 May 29 19:12:26.536669 ixr-x3b-5 INFO containerd[1049]: time="2025-05-29T19:12:26.536282598Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/1d760594c1d4e93aa167693f238f69bfac153b6ee2b887461aa69456fa7dec5d pid=2425 runtime=io.containerd.runc.v2
2025 May 29 19:12:26.560905 ixr-x3b-5 INFO containerd[1049]: time="2025-05-29T19:12:26.560755268Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
2025 May 29 19:12:26.560975 ixr-x3b-5 INFO containerd[1049]: time="2025-05-29T19:12:26.560834198Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
2025 May 29 19:12:26.561022 ixr-x3b-5 INFO containerd[1049]: time="2025-05-29T19:12:26.560853508Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
2025 May 29 19:12:26.561110 ixr-x3b-5 INFO containerd[1049]: time="2025-05-29T19:12:26.561055628Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/d96075766c2f01bb6e9caa87abe3938b974dd91e7d56fb6f3bc198194c69e6fc pid=2452 runtime=io.containerd.runc.v2
2025 May 29 19:12:26.655748 ixr-x3b-5 ERR libnetwork-setkey: tac_connect_single: connection to 10.250.17.253:49 failed: Network is unreachable
2025 May 29 19:12:26.655828 ixr-x3b-5 ERR libnetwork-setkey: nss_tacplus: failed to connect TACACS+ server 10.250.17.253:49, ret=-9: Network is unreachable
2025 May 29 19:12:26.655871 ixr-x3b-5 ERR libnetwork-setkey: tac_connect_single: connection to 10.250.17.253:49 failed: Network is unreachable
2025 May 29 19:12:26.655912 ixr-x3b-5 ERR libnetwork-setkey: nss_tacplus: failed to connect TACACS+ server 10.250.17.253:49, ret=-9: Network is unreachable
2025 May 29 19:12:26.656455 ixr-x3b-5 ERR libnetwork-setkey: tac_connect_single: connection to 10.250.17.253:49 failed: Network is unreachable
2025 May 29 19:12:26.656536 ixr-x3b-5 ERR libnetwork-setkey: nss_tacplus: failed to connect TACACS+ server 10.250.17.253:49, ret=-9: Network is unreachable
2025 May 29 19:12:26.656577 ixr-x3b-5 ERR libnetwork-setkey: tac_connect_single: connection to 10.250.17.253:49 failed: Network is unreachable
2025 May 29 19:12:26.656615 ixr-x3b-5 ERR libnetwork-setkey: nss_tacplus: failed to connect TACACS+ server 10.250.17.253:49, ret=-9: Network is unreachable
Output of show version, show techsupport
Attach files (if any)
No response
Is it platform specific
broadcom
Importance or Severity
Medium
Description of the bug
Below error seen on sonic-mgmt voq tests (test_voq_chassis_app_db_consistency, test_voq_disrupts):
2025 May 29 19:12:10.954804 ixr-x3b-5 NOTICE kernel: [ 0.017736] Kernel command line: BOOT_IMAGE=/image-HEAD.1044465-msft-2503-5a2624582/boot/vmlinuz-6.1.0-29-2-amd64 root=UUID=99f1be09-5129-4845-9da1-ef081a968521 rw console=tty0 console=ttyS0,115200n8 quiet processor.max_cstate=1 amd_idle.max_cstate=0 net.ifnames=0 biosdevname=0 loop=image-HEAD.1044465-msft-2503-5a2624582/fs.squashfs loopfstype=squashfs systemd.unified_cgroup_hierarchy=0 apparmor=1 security=apparmor varlog_size=4096 usbcore.autosuspend=-1 amd_iommu=off crashkernel=8G-:1G idle=poll
2025 May 29 19:12:16.538690 ixr-x3b-5 ERR dockerd: nss_tacplus: failed to connect TACACS+ server 10.250.17.253:49, ret=-8: Operation now in progress
2025 May 29 19:12:16.538758 ixr-x3b-5 ERR dockerd: tac_connect_single: connection to 10.250.17.253:49 failed: Network is unreachable
2025 May 29 19:12:16.538806 ixr-x3b-5 ERR dockerd: nss_tacplus: failed to connect TACACS+ server 10.250.17.253:49, ret=-9: Network is unreachable
2025 May 29 19:12:26.655748 ixr-x3b-5 ERR libnetwork-setkey: tac_connect_single: connection to 10.250.17.253:49 failed: Network is unreachable
2025 May 29 19:12:26.655828 ixr-x3b-5 ERR libnetwork-setkey: nss_tacplus: failed to connect TACACS+ server 10.250.17.253:49, ret=-9: Network is unreachable
2025 May 29 19:12:26.655871 ixr-x3b-5 ERR libnetwork-setkey: tac_connect_single: connection to 10.250.17.253:49 failed: Network is unreachable
2025 May 29 19:12:26.655912 ixr-x3b-5 ERR libnetwork-setkey: nss_tacplus: failed to connect TACACS+ server 10.250.17.253:49, ret=-9: Network is unreachable
2025 May 29 19:12:26.656455 ixr-x3b-5 ERR libnetwork-setkey: tac_connect_single: connection to 10.250.17.253:49 failed: Network is unreachable
2025 May 29 19:12:26.656536 ixr-x3b-5 ERR libnetwork-setkey: nss_tacplus: failed to connect TACACS+ server 10.250.17.253:49, ret=-9: Network is unreachable
2025 May 29 19:12:26.656577 ixr-x3b-5 ERR libnetwork-setkey: tac_connect_single: connection to 10.250.17.253:49 failed: Network is unreachable
2025 May 29 19:12:26.656615 ixr-x3b-5 ERR libnetwork-setkey: nss_tacplus: failed to connect TACACS+ server 10.250.17.253:49, ret=-9: Network is unreachable
Steps to Reproduce
1.Execute the below tests :
test_voq_chassis_app_db_consistency.py & test_voq_disrupts.py
2. Verify the syslog after reboot is triggered.
Actual Behavior and Expected Behavior
This error occurs only on reboot. During the up -path when the system starts coming up, TACACS+ server reachability check is done very early in the process and hence, errors out
Relevant log output
Output of
show version,show techsupportAttach files (if any)
No response