Skip to content

Warmboot fails with OA not ready for warm restart - pending SET oper on VXLAN table #12361

Open
Open
Warmboot fails with OA not ready for warm restart - pending SET oper on VXLAN table#12361
Assignees
skbhava
Labels
BRCMIssue for 202205Issue for 202305Triagedthis issue has been triagedwarmboot 👢
@vaibhavhd

Description

@vaibhavhd
vaibhavhd
opened on Oct 11, 2022

Description

When warmboot is issue with CPA, RESTARTCHECK fails with OA continuously being busy on a pending task VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000

This issue has surfaced after a recent PR to fix another issue with interface creation during CPA:
sonic-net/sonic-utilities#2398

Oct  7 18:43:52.863105 str-7260cx3-acs-1 ERR swss#orchagent: :- handleSaiGetStatus: Encountered failure in get operation, SAI API: SAI_API_MIRROR, status: SAI_STATUS_INVALID_PARAMETER
Oct  7 18:43:52.892196 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:52.892219 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1


Oct  7 18:43:55.045293 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Oct  7 18:43:55.045427 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Oct  7 18:43:55.045441 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Oct  7 18:43:55.045566 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY

Steps to reproduce the issue:

  1. Install latest master or 202205 image on a DUT
  2. Issue warmboot with -c option (with CPA)
  3. warmboot will fail with RESTARTCHECK error

Describe the results you received:

Detailed logs:

Oct  7 18:43:46.150801 str-7260cx3-acs-1 NOTICE admin: Saving counters folder before warmboot...
Oct  7 18:43:46.904575 str-7260cx3-acs-1 INFO syncd#syncd: [none] SAI_API_FDB:_brcm_sai_fdb_event_cb:212 fdbEvent: add (1) for mac 72-06-00-01-05-85 vid:0x3e8, port:0x6b lagid:0x0 flags:0x10440 flags2:0x0 lag:false station flags 0x0
Oct  7 18:43:46.904575 str-7260cx3-acs-1 INFO syncd#syncd: [none] SAI_API_NEIGHBOR:brcm_sai_xgs_common_neighbor_mac_db_update:104 Mac entry not found - skip nbr processing. add 1, dir 4
Oct  7 18:43:47.055934 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 192.168.1.249, 72:06:00:01:05:03 on Vlan1000
Oct  7 18:43:47.057228 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNextHop: Created next hop 192.168.1.249 on Vlan1000
Oct  7 18:43:47.143650 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 192.168.2.75, 72:06:00:01:05:85 on Vlan1000
Oct  7 18:43:47.144652 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNextHop: Created next hop 192.168.2.75 on Vlan1000
Oct  7 18:43:47.172837 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:47.173026 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:47.269332 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:47.363618 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:47.463093 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:47.545223 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:47.638356 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:47.739209 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:49.417956 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 192.168.2.91, 72:06:00:01:06:01 on Vlan1000
Oct  7 18:43:49.419900 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNextHop: Created next hop 192.168.2.91 on Vlan1000
Oct  7 18:43:49.590846 str-7260cx3-acs-1 DEBUG check_db_integrity.py: Database integrity checks passed.
Oct  7 18:43:50.050771 str-7260cx3-acs-1 INFO admin: Checking that ASIC configuration has not changed
Oct  7 18:43:50.775064 str-7260cx3-acs-1 INFO admin: ASIC config unchanged, current and destination SONiC version are the same
Oct  7 18:43:51.606226 str-7260cx3-acs-1 NOTICE admin: Setting up control plane assistant: 10.64.247.17 ...
Oct  7 18:43:51.742819 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 192.168.2.13, 72:06:00:01:05:23 on Vlan1000
Oct  7 18:43:51.747218 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNextHop: Created next hop 192.168.2.13 on Vlan1000
Oct  7 18:43:51.853751 str-7260cx3-acs-1 NOTICE swss#vxlanmgrd: :- doVxlanTunnelCreateTask: Create vxlan tunnel neigh_adv
Oct  7 18:43:51.854593 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addOperation: Vxlan tunnel 'neigh_adv' was added
Oct  7 18:43:51.871876 str-7260cx3-acs-1 INFO systemd-udevd[12173]: Using default interface naming scheme 'v247'.
Oct  7 18:43:51.872008 str-7260cx3-acs-1 INFO systemd-udevd[12173]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Oct  7 18:43:51.874828 str-7260cx3-acs-1 INFO kernel: [  524.277440] Bridge: port 114(neigh_adv-1000) entered blocking state
Oct  7 18:43:51.874853 str-7260cx3-acs-1 INFO kernel: [  524.277443] Bridge: port 114(neigh_adv-1000) entered disabled state
Oct  7 18:43:51.874855 str-7260cx3-acs-1 INFO kernel: [  524.278070] device neigh_adv-1000 entered promiscuous mode
Oct  7 18:43:51.893539 str-7260cx3-acs-1 INFO kernel: [  524.295032] Bridge: port 114(neigh_adv-1000) entered blocking state
Oct  7 18:43:51.893552 str-7260cx3-acs-1 INFO kernel: [  524.295036] Bridge: port 114(neigh_adv-1000) entered forwarding state
Oct  7 18:43:51.893852 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:51.895978 str-7260cx3-acs-1 WARNING swss#orchagent: :- createTunnelHw: creation src = 0
Oct  7 18:43:51.895978 str-7260cx3-acs-1 NOTICE swss#orchagent: :- create_tunnel: create_tunnel:encapmaplist[0]=0x290000000019c8
Oct  7 18:43:51.895978 str-7260cx3-acs-1 NOTICE swss#orchagent: :- create_tunnel: create_tunnel:encapmaplist[1]=0x290000000019ca
Oct  7 18:43:52.243588 str-7260cx3-acs-1 NOTICE syncd#syncd: :- threadFunction: time span 173 ms for 'create:SAI_OBJECT_TYPE_TUNNEL_MAP_ENTRY:oid:0x3b0000000019cd'
Oct  7 18:43:52.854634 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addOperation: Vxlan tunnel map entry 'map_1' for tunnel 'neigh_adv' was created
Oct  7 18:43:52.863105 str-7260cx3-acs-1 ERR swss#orchagent: :- handleSaiGetStatus: Encountered failure in get operation, SAI API: SAI_API_MIRROR, status: SAI_STATUS_INVALID_PARAMETER
Oct  7 18:43:52.886935 str-7260cx3-acs-1 NOTICE swss#orchagent: :- attach: Attached next hop observer of route 192.168.8.0/25 for destination IP 192.168.8.1
Oct  7 18:43:52.887226 str-7260cx3-acs-1 NOTICE swss#orchagent: :- updateNextHop: Updating mirror session neighbor_advertiser with route 192.168.8.0/25
Oct  7 18:43:52.887226 str-7260cx3-acs-1 NOTICE swss#orchagent: :- updateNextHop:     next hop IPs: 10.0.0.33@PortChannel101,10.0.0.35@PortChannel102,10.0.0.37@PortChannel103,10.0.0.39@PortChannel104
Oct  7 18:43:52.887497 str-7260cx3-acs-1 NOTICE swss#orchagent: :- updateNextHop: Updated mirror session state db neighbor_advertiser nexthop to 10.0.0.33@PortChannel101
Oct  7 18:43:52.887497 str-7260cx3-acs-1 NOTICE swss#orchagent: :- getNeighborInfo: Mirror session neighbor_advertiser neighbor is PortChannel101
Oct  7 18:43:52.890866 str-7260cx3-acs-1 NOTICE swss#orchagent: :- activateSession: Activated mirror session neighbor_advertiser
Oct  7 18:43:52.891029 str-7260cx3-acs-1 NOTICE swss#orchagent: :- createEntry: Created mirror session neighbor_advertiser
Oct  7 18:43:52.892196 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:52.892219 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:52.892437 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:52.892437 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:52.892504 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:52.892572 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:52.892572 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:52.892755 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:52.892789 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:52.892819 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:53.158619 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:53.158619 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:53.468052 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:53.468079 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.158617 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.158617 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.468096 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.468096 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.519625 str-7260cx3-acs-1 INFO syncd#syncd: [none] SAI_API_FDB:_brcm_sai_fdb_event_cb:212 fdbEvent: add (1) for mac 72-06-00-01-05-08 vid:0x3e8, port:0x11 lagid:0x0 flags:0x10440 flags2:0x0 lag:false station flags 0x0
Oct  7 18:43:54.519833 str-7260cx3-acs-1 INFO syncd#syncd: [none] SAI_API_NEIGHBOR:brcm_sai_xgs_common_neighbor_mac_db_update:104 Mac entry not found - skip nbr processing. add 1, dir 4
Oct  7 18:43:54.519984 str-7260cx3-acs-1 INFO syncd#syncd: [none] SAI_API_FDB:_brcm_sai_fdb_event_cb:212 fdbEvent: add (1) for mac 72-06-00-01-06-09 vid:0x3e8, port:0x33 lagid:0x0 flags:0x10440 flags2:0x0 lag:false station flags 0x0
Oct  7 18:43:54.519984 str-7260cx3-acs-1 INFO syncd#syncd: [none] SAI_API_NEIGHBOR:brcm_sai_xgs_common_neighbor_mac_db_update:104 Mac entry not found - skip nbr processing. add 1, dir 4
Oct  7 18:43:54.520534 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.520841 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.520841 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.520841 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.520993 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.521096 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.521096 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.521120 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.521440 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.521457 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.521578 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.521651 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.521950 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.522006 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.522060 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.522060 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.558309 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 192.168.1.254, 72:06:00:01:05:08 on Vlan1000
Oct  7 18:43:54.563929 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNextHop: Created next hop 192.168.1.254 on Vlan1000
Oct  7 18:43:54.564205 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.564248 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.688020 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 192.168.2.99, 72:06:00:01:06:09 on Vlan1000
Oct  7 18:43:54.691448 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNextHop: Created next hop 192.168.2.99 on Vlan1000
Oct  7 18:43:54.691560 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.691637 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.895228 str-7260cx3-acs-1 NOTICE swss#orchagent: :- add: Successfully created ACL rule rule_arp in table EVERFLOW
Oct  7 18:43:54.895760 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.895760 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.904555 str-7260cx3-acs-1 NOTICE swss#orchagent: :- add: Successfully created ACL rule rule_nd in table EVERFLOW
Oct  7 18:43:54.904938 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.904938 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.917359 str-7260cx3-acs-1 NOTICE admin: Pausing orchagent ...
Oct  7 18:43:55.003186 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:55.044774 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: Wait time for response from orchagent set to 2000 milliseconds
Oct  7 18:43:55.044774 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: Number of retries for the request to orchagent is set to 5
Oct  7 18:43:55.045123 str-7260cx3-acs-1 INFO swss#orchagent_restart_check: :- subscribe: subscribed to RESTARTCHECKREPLY
Oct  7 18:43:55.045123 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 0
Oct  7 18:43:55.045293 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Oct  7 18:43:55.045293 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Oct  7 18:43:55.045390 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.045406 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:55.045427 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Oct  7 18:43:55.045441 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Oct  7 18:43:55.045448 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Oct  7 18:43:55.045566 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Oct  7 18:43:55.045582 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 1
Oct  7 18:43:55.045661 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Oct  7 18:43:55.045673 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Oct  7 18:43:55.045722 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.045734 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:55.045818 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Oct  7 18:43:55.045850 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Oct  7 18:43:55.045878 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Oct  7 18:43:55.045878 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Oct  7 18:43:55.045900 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 2
Oct  7 18:43:55.046002 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Oct  7 18:43:55.046002 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Oct  7 18:43:55.046141 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.046191 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:55.046191 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Oct  7 18:43:55.046191 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Oct  7 18:43:55.046191 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Oct  7 18:43:55.046211 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Oct  7 18:43:55.046211 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 3
Oct  7 18:43:55.046348 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Oct  7 18:43:55.046348 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Oct  7 18:43:55.046442 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.046442 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:55.046480 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Oct  7 18:43:55.046480 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Oct  7 18:43:55.046560 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Oct  7 18:43:55.046578 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Oct  7 18:43:55.046593 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 4
Oct  7 18:43:55.046684 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Oct  7 18:43:55.046684 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Oct  7 18:43:55.046736 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.046736 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:55.046809 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Oct  7 18:43:55.046826 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Oct  7 18:43:55.046826 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Oct  7 18:43:55.046956 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Oct  7 18:43:55.046956 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 5
Oct  7 18:43:55.046985 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Oct  7 18:43:55.047000 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Oct  7 18:43:55.047041 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.047053 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:55.047074 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Oct  7 18:43:55.047090 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Oct  7 18:43:55.047090 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Oct  7 18:43:55.047287 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Oct  7 18:43:55.053655 str-7260cx3-acs-1 NOTICE admin: warm-reboot failure (10) cleanup ...
Oct  7 18:43:55.058739 str-7260cx3-acs-1 NOTICE admin: Tearing down control plane assistant: 10.64.247.17 ...
Oct  7 18:43:55.158736 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.158736 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:55.210556 str-7260cx3-acs-1 NOTICE swss#orchagent: :- remove: Successfully deleted ACL rule rule_arp in table EVERFLOW
Oct  7 18:43:55.210556 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.210556 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:55.213256 str-7260cx3-acs-1 NOTICE swss#orchagent: :- remove: Successfully deleted ACL rule rule_nd in table EVERFLOW
Oct  7 18:43:55.213281 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.213288 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:55.214870 str-7260cx3-acs-1 WARNING kernel: [  527.614422] linux-bcm-knet (2731): Unsupported command (type=1, opcode=52)
Oct  7 18:43:55.468034 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.468034 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:56.158839 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:56.158914 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:56.468302 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:56.468302 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:57.095922 str-7260cx3-acs-1 ALERT dhcp_relay#dhcpmon[47]: dhcpmon detected disparity in DHCP Relay behavior. Duration: 378 (sec) for vlan: 'Agg-Vlan1000'
Oct  7 18:43:57.095922 str-7260cx3-acs-1 NOTICE dhcp_relay#dhcpmon[47]: [    Agg-Vlan1000-Snapshot rx/tx] Discover:       998/        0, Offer:         0/        0, Request:         0/        0, ACK:         0/        0
Oct  7 18:43:57.095922 str-7260cx3-acs-1 NOTICE dhcp_relay#dhcpmon[47]: [    Agg-Vlan1000- Current rx/tx] Discover:      1053/        0, Offer:         0/        0, Request:         0/        0, ACK:         0/        0
Oct  7 18:43:57.158941 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:57.158941 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:57.468201 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:57.468201 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:58.158844 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:58.158899 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:58.226899 str-7260cx3-acs-1 INFO kernel: [  530.624862] Bridge: port 114(neigh_adv-1000) entered disabled state
Oct  7 18:43:58.230023 str-7260cx3-acs-1 NOTICE swss#orchagent: :- deactivateSession: Deactivated mirror session neighbor_advertiser
Oct  7 18:43:58.230233 str-7260cx3-acs-1 NOTICE swss#orchagent: :- detach: Detached next hop observer for destination IP 192.168.8.1
Oct  7 18:43:58.230539 str-7260cx3-acs-1 NOTICE swss#orchagent: :- deleteEntry: Removed mirror session neighbor_advertiser
Oct  7 18:43:58.230832 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:58.230986 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:58.234817 str-7260cx3-acs-1 INFO kernel: [  530.634999] device neigh_adv-1000 left promiscuous mode
Oct  7 18:43:58.234836 str-7260cx3-acs-1 INFO kernel: [  530.635021] Bridge: port 114(neigh_adv-1000) entered disabled state
Oct  7 18:43:58.244055 str-7260cx3-acs-1 NOTICE syncd#syncd: :- threadFunction: time span 11 ms for 'remove:SAI_OBJECT_TYPE_TUNNEL_MAP_ENTRY:oid:0x3b0000000019cd'
Oct  7 18:43:58.251226 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:58.287772 str-7260cx3-acs-1 NOTICE swss#vxlanmgrd: :- doVxlanTunnelDeleteTask: Delete vxlan tunnel neigh_adv
Oct  7 18:43:58.394301 str-7260cx3-acs-1 NOTICE swss#orchagent: :- delOperation: vni count = 0
Oct  7 18:43:58.398438 str-7260cx3-acs-1 NOTICE swss#orchagent: :- delOperation: Vxlan tunnel map entry 'map_1' for tunnel 'neigh_adv' was removed
Oct  7 18:43:58.398554 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:58.398554 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:58.399020 str-7260cx3-acs-1 NOTICE swss#orchagent: :- delOperation: Vxlan tunnel 'neigh_adv' was removed
Oct  7 18:43:58.399124 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:58.399124 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:58.399583 str-7260cx3-acs-1 WARNING swss#orchagent: :- delOperation: RemoteVniDel getTunnelPort Fails: 192.168.8.1
Oct  7 18:43:58.759667 str-7260cx3-acs-1 NOTICE admin: Cancel warm-reboot: code (0)

Describe the results you expected:

CPA creation should succeed and warmboot should proceed.

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

Metadata

Metadata

Assignees

Labels

BRCMIssue for 202205Issue for 202305Triagedthis issue has been triagedwarmboot 👢

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions