Description
When the switch has scaled configuration, the below logs are seen during config reload. It happens because of high CPU contention during config reload. The logs don't happen during reboot.
Feb 8 21:13:46.695474 r-spider-05 ERR syncd#SDK: :- threadFunction: time span WD exceeded 30955 ms for create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000
Feb 8 21:13:46.695474 r-spider-05 ERR syncd#SDK: :- logEventData: op: create, key: SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000
Feb 8 21:13:46.695474 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_INIT_SWITCH: true
Feb 8 21:13:46.695474 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_FDB_EVENT_NOTIFY: 0x55dabccda690
Feb 8 21:13:46.695474 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_PORT_STATE_CHANGE_NOTIFY: 0x55dabccda6a0
Feb 8 21:13:46.695474 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_BFD_SESSION_STATE_CHANGE_NOTIFY: 0x55dabccda6b0
Feb 8 21:13:46.695525 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_SWITCH_SHUTDOWN_REQUEST_NOTIFY: 0x55dabccda6c0
Feb 8 21:13:48.653822 r-spider-05 ERR syncd#SDK: :- setEndTime: event 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000' took 32913 ms to execute
Feb 8 21:13:48.654048 r-spider-05 ERR syncd#SDK: :- logEventData: op: create, key: SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000
Feb 8 21:13:48.654230 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_INIT_SWITCH: true
Feb 8 21:13:48.654405 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_FDB_EVENT_NOTIFY: 0x55dabccda690
Feb 8 21:13:48.654971 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_PORT_STATE_CHANGE_NOTIFY: 0x55dabccda6a0
Feb 8 21:13:48.655165 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_BFD_SESSION_STATE_CHANGE_NOTIFY: 0x55dabccda6b0
Feb 8 21:13:48.655338 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_SWITCH_SHUTDOWN_REQUEST_NOTIFY: 0x55dabccda6c0
Steps to reproduce the issue:
- Have scaled configs and enable all services
- Perform config reload
Describe the results you received:
Error logs seen in syslog
Describe the results you expected:
No error logs.
Output of show version:
SONiC Software Version: SONiC.202111.10-f08866b66_Internal
Distribution: Debian 11.2
Kernel: 5.10.0-8-2-amd64
Build commit: f08866b66
Build date: Mon Feb 7 08:15:17 UTC 2022
Built by: sw-r2d2-bot@r-build-sonic-ci02-241
Platform: x86_64-mlnx_msn2410-r0
HwSKU: ACS-MSN2410
ASIC: mellanox
ASIC Count: 1
Serial Number: MT1921X01546
Model Number: MSN2410-CB2FO
Hardware Revision: A2
Uptime: 21:19:17 up 13 min, 2 users, load average: 1.59, 2.45, 1.90
Docker images:
REPOSITORY TAG IMAGE ID SIZE
docker-teamd 202111.10-f08866b66_Internal a6461b4fc1b1 438MB
docker-teamd latest a6461b4fc1b1 438MB
docker-sflow 202111.10-f08866b66_Internal 502f7aeb5296 439MB
docker-sflow latest 502f7aeb5296 439MB
docker-orchagent 202111.10-f08866b66_Internal 22c1035163ec 457MB
docker-orchagent latest 22c1035163ec 457MB
docker-nat 202111.10-f08866b66_Internal bfaaeef59e80 441MB
docker-nat latest bfaaeef59e80 441MB
docker-macsec 202111.10-f08866b66_Internal 19ce40820a13 441MB
docker-macsec latest 19ce40820a13 441MB
docker-fpm-frr 202111.10-f08866b66_Internal f76fdb1c1625 457MB
docker-fpm-frr latest f76fdb1c1625 457MB
docker-syncd-mlnx 202111.10-f08866b66_Internal 9187d53ee421 1.01GB
docker-syncd-mlnx latest 9187d53ee421 1.01GB
docker-platform-monitor 202111.10-f08866b66_Internal b9bce6dd4fad 809MB
docker-platform-monitor latest b9bce6dd4fad 809MB
docker-snmp 202111.10-f08866b66_Internal 8a604e09da49 465MB
docker-snmp latest 8a604e09da49 465MB
docker-dhcp-relay latest 7c4f66877495 436MB
docker-sonic-mgmt-framework 202111.10-f08866b66_Internal 8e4823d8d271 578MB
docker-sonic-mgmt-framework latest 8e4823d8d271 578MB
docker-sonic-telemetry 202111.10-f08866b66_Internal df6917e0f648 511MB
docker-sonic-telemetry latest df6917e0f648 511MB
docker-router-advertiser 202111.10-f08866b66_Internal ab565ec647e8 423MB
docker-router-advertiser latest ab565ec647e8 423MB
docker-mux 202111.10-f08866b66_Internal 4132719ce52e 475MB
docker-mux latest 4132719ce52e 475MB
docker-lldp 202111.10-f08866b66_Internal 5b758f8ab7b9 463MB
docker-lldp latest 5b758f8ab7b9 463MB
docker-database 202111.10-f08866b66_Internal 3d69079016cc 423MB
docker-database latest 3d69079016cc 423MB
urm.nvidia.com/sw-nbu-sws-sonic-docker/sonic-wjh 1.2.0-202111-internal-1 259fd9a16dbc 469MB
Output of show techsupport:
(paste your output here or download and attach the file here )
Additional information you deem important (e.g. issue happens only occasionally):
sysdump_test_core_functionality_with_reboot.tar.gz
Description
When the switch has scaled configuration, the below logs are seen during config reload. It happens because of high CPU contention during config reload. The logs don't happen during reboot.
Feb 8 21:13:46.695474 r-spider-05 ERR syncd#SDK: :- threadFunction: time span WD exceeded 30955 ms for create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000
Feb 8 21:13:46.695474 r-spider-05 ERR syncd#SDK: :- logEventData: op: create, key: SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000
Feb 8 21:13:46.695474 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_INIT_SWITCH: true
Feb 8 21:13:46.695474 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_FDB_EVENT_NOTIFY: 0x55dabccda690
Feb 8 21:13:46.695474 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_PORT_STATE_CHANGE_NOTIFY: 0x55dabccda6a0
Feb 8 21:13:46.695474 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_BFD_SESSION_STATE_CHANGE_NOTIFY: 0x55dabccda6b0
Feb 8 21:13:46.695525 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_SWITCH_SHUTDOWN_REQUEST_NOTIFY: 0x55dabccda6c0
Feb 8 21:13:48.653822 r-spider-05 ERR syncd#SDK: :- setEndTime: event 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000' took 32913 ms to execute
Feb 8 21:13:48.654048 r-spider-05 ERR syncd#SDK: :- logEventData: op: create, key: SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000
Feb 8 21:13:48.654230 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_INIT_SWITCH: true
Feb 8 21:13:48.654405 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_FDB_EVENT_NOTIFY: 0x55dabccda690
Feb 8 21:13:48.654971 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_PORT_STATE_CHANGE_NOTIFY: 0x55dabccda6a0
Feb 8 21:13:48.655165 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_BFD_SESSION_STATE_CHANGE_NOTIFY: 0x55dabccda6b0
Feb 8 21:13:48.655338 r-spider-05 ERR syncd#SDK: :- logEventData: fv: SAI_SWITCH_ATTR_SWITCH_SHUTDOWN_REQUEST_NOTIFY: 0x55dabccda6c0
Steps to reproduce the issue:
Describe the results you received:
Error logs seen in syslog
Describe the results you expected:
No error logs.
Output of
show version:Output of
show techsupport:Additional information you deem important (e.g. issue happens only occasionally):
sysdump_test_core_functionality_with_reboot.tar.gz