[202012][test_fib] test_fib fails #8519
Description
Description
test_fib constantly fails on nightly.
Steps to reproduce the issue:
This issue could be manually reproduced:
NOTE: both icmp_responder and garp_service are stopped.
- build TCP packet with dest IP that has nexthop as one of the t1s
- dest IP as
100.1.0.29
root@str2-7050cx3-acs-06:~# show ip route 100.1.0.29
Routing entry for 100.1.0.29/32
Known via "bgp", distance 20, metric 0, best
Last update 03:09:20 ago
* 10.0.0.57, via PortChannel0001, weight 1
- build tcp packet that has dest MAC as the upper ToR
>>> pkt0
<Ether dst=d4:af:f7:4d:a4:44 src=d6:7e:c6:35:6a:22 type=0x800 |<IP ihl=None tos=0x0 id=1 frag=0 ttl=64 proto=tcp src=30.0.0.1 dst=100.1.0.29 |<TCP sport=30302 dport=50446 flags=S |<Raw load='\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-' |>>>>
- send the TCP packet to the upper ToR from the T1's intercepted ports
eth34connects to the upper ToR'sPortChannel0004
>>> sendp(pkt0, iface='eth34', count=10000)
- could not received from any packets from the ptf intercepted port that connects to
PortChannel0001
root@str2-7050cx3-acs-06:~# portstat
Last cached time was 2021-08-18 13:52:51.142751
IFACE STATE RX_OK RX_BPS RX_UTIL RX_ERR RX_DRP RX_OVR TX_OK TX_BPS TX_UTIL TX_ERR TX_DRP TX_OVR
----------- ------- ------- ---------- --------- -------- -------- -------- ------- ---------- --------- -------- -------- --------
Ethernet0 X 0 0.00 B/s 0.00% 0 0 0 0 0.00 B/s 0.00% 0 0 0
Ethernet4 U 0 0.00 B/s 0.00% 0 0 0 476 732.93 B/s 0.00% 0 0 0
Ethernet8 U 0 0.00 B/s 0.00% 0 0 0 477 734.47 B/s 0.00% 0 0 0
Ethernet12 U 0 0.00 B/s 0.00% 0 0 0 477 734.47 B/s 0.00% 0 0 0
Ethernet16 U 0 0.00 B/s 0.00% 0 0 0 477 734.47 B/s 0.00% 0 0 0
Ethernet20 U 0 0.00 B/s 0.00% 0 0 0 477 734.47 B/s 0.00% 0 0 0
Ethernet24 U 0 0.00 B/s 0.00% 0 0 0 477 734.47 B/s 0.00% 0 0 0
Ethernet28 U 0 0.00 B/s 0.00% 0 0 0 477 734.47 B/s 0.00% 0 0 0
Ethernet32 U 0 0.00 B/s 0.00% 0 0 0 477 734.47 B/s 0.00% 0 0 0
Ethernet36 U 0 0.00 B/s 0.00% 0 0 0 477 734.49 B/s 0.00% 0 0 0
Ethernet40 U 0 0.00 B/s 0.00% 0 0 0 477 734.49 B/s 0.00% 0 0 0
Ethernet44 U 0 0.00 B/s 0.00% 0 0 0 487 749.87 B/s 0.00% 0 0 0
Ethernet48 U 0 0.00 B/s 0.00% 0 0 0 487 749.87 B/s 0.00% 0 0 0
Ethernet52 U 0 0.00 B/s 0.00% 0 0 0 487 749.87 B/s 0.00% 0 0 0
Ethernet56 U 0 0.00 B/s 0.00% 0 0 0 487 749.87 B/s 0.00% 0 0 0
Ethernet60 U 0 0.00 B/s 0.00% 0 0 0 487 749.87 B/s 0.00% 0 0 0
Ethernet64 U 0 0.00 B/s 0.00% 0 0 0 487 749.87 B/s 0.00% 0 0 0
Ethernet68 U 0 0.00 B/s 0.00% 0 0 0 487 749.87 B/s 0.00% 0 0 0
Ethernet72 U 0 0.00 B/s 0.00% 0 0 0 487 749.87 B/s 0.00% 0 0 0
Ethernet76 U 0 0.00 B/s 0.00% 0 0 0 487 749.87 B/s 0.00% 0 0 0
Ethernet80 U 0 0.00 B/s 0.00% 0 0 0 487 749.87 B/s 0.00% 0 0 0
Ethernet84 U 0 0.00 B/s 0.00% 0 0 0 487 749.87 B/s 0.00% 0 0 0
Ethernet88 U 0 0.00 B/s 0.00% 0 0 0 487 749.87 B/s 0.00% 0 0 0
Ethernet92 U 0 0.00 B/s 0.00% 0 0 0 487 749.87 B/s 0.00% 0 0 0
Ethernet96 U 0 0.00 B/s 0.00% 0 0 0 487 749.87 B/s 0.00% 0 0 0
Ethernet100 X 0 0.00 B/s 0.00% 0 0 0 0 0.00 B/s 0.00% 0 0 0
Ethernet104 X 0 0.00 B/s 0.00% 0 0 0 0 0.00 B/s 0.00% 0 0 0
Ethernet108 X 0 0.00 B/s 0.00% 0 0 0 0 0.00 B/s 0.00% 0 0 0
Ethernet112 U 7 17.22 B/s 0.00% 0 0 0 7 19.15 B/s 0.00% 0 0 0
Ethernet116 U 7 17.22 B/s 0.00% 0 0 0 7 19.15 B/s 0.00% 0 0 0
Ethernet120 U 7 17.22 B/s 0.00% 0 0 0 7 19.15 B/s 0.00% 0 0 0
Ethernet124 U 10,007 22.83 KB/s 0.00% 0 0 0 7 19.15 B/s 0.00% 0 0 0
Ethernet128 X 0 0.00 B/s 0.00% 0 0 0 0 0.00 B/s 0.00% 0 0 0
Ethernet132 X 0 0.00 B/s 0.00% 0 0 0 0 0.00 B/s 0.00% 0 0 0
Describe the results you received:
- the upper ToR doesn't forward the packet to its nexthops
Describe the results you expected:
- the upper ToR should forward the packets
Output of show version:
SONiC version: SONiC.20201231.16
Other observations
Some other observation:
- During the trials, the upper ToR is the standby ToR.
- If we send the tcp packet(destination as
100.1.0.29to the lower ToR(active ToR), the packet gets forwarded. - If we start
icmp_responderandgarp_service, the issue is gone as the TCP packet forwarded to the upper ToR gets routed as the lower ToR.
Metadata
Metadata
Assignees
Labels
No labels