Skip to content

On DualToR Active-Active testbed, 9 failures in test_everflow_ipv6.py due to packets not being received on expected port #9983

@yanjundeng

Description

@yanjundeng

Description
When running test_everflow_ipv6.py ,
after configuring everflow mirroring on the duts, 19 out of 20 testcases are failing because the expected packet is not being received on the expected ports

AssertionError: Did not receive expected packet on any of ports [62, 63] for device 0.

Steps to reproduce the issue:

  1. run "test_everflow_ipv6.py" on dualtor active-active testbeds based on 202205 mgmt

Analysis
As to dualtor Active-Active, the everflow test script only configure ACL rule on one DUT.
Yet, according to debug in our test, the pkts are sent to another DUT which has no ACL rule and mirror config.

For example, the mirror session and everflow rules are only configured on tor40-1 during everflow ipv6 test:

cisco@tor40-2:~$ show mirror_session
ERSPAN Sessions
Name Status SRC IP DST IP GRE DSCP TTL Queue Policer Monitor Port SRC Port Direction


SPAN Sessions
Name Status DST Port SRC Port Direction Queue Policer


cisco@tor40-2:~$ show acl rule
Table Rule Priority Action Match Status


cisco@tor40-2:~$

cisco@tor40-1:~$ show acl rule
Table Rule Priority Action Match Status


EVERFLOWV6 RULE_1 9999 MIRROR INGRESS: test_session_1 SRC_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:0002/128 Active
EVERFLOWV6 RULE_2 9998 MIRROR INGRESS: test_session_1 DST_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:0003/128 Active
EVERFLOWV6 RULE_3 9997 MIRROR INGRESS: test_session_1 IP_PROTOCOL: 126 Active
EVERFLOWV6 RULE_4 9996 MIRROR INGRESS: test_session_1 L4_SRC_PORT: 9000 Active
EVERFLOWV6 RULE_5 9995 MIRROR INGRESS: test_session_1 L4_DST_PORT: 9001 Active
EVERFLOWV6 RULE_6 9994 MIRROR INGRESS: test_session_1 L4_SRC_PORT_RANGE: 10000-10499 Active
EVERFLOWV6 RULE_7 9993 MIRROR INGRESS: test_session_1 L4_DST_PORT_RANGE: 10500-10999 Active
EVERFLOWV6 RULE_8 9992 MIRROR INGRESS: test_session_1 TCP_FLAGS: 0x1b/0x1b Active
EVERFLOWV6 RULE_9 9991 MIRROR INGRESS: test_session_1 DSCP: 37 Active
EVERFLOWV6 RULE_10 9990 MIRROR INGRESS: test_session_1 DST_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:0005/128 Active
IP_PROTOCOL: 6
L4_DST_PORT_RANGE: 11500-11999
L4_SRC_PORT: 11200
SRC_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:0004/128
EVERFLOWV6 RULE_11 9989 MIRROR INGRESS: test_session_1 DST_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:0004/128 Active
IP_PROTOCOL: 6
L4_DST_PORT: 11200
L4_SRC_PORT_RANGE: 11500-11999
SRC_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:0005/128
EVERFLOWV6 RULE_12 9988 MIRROR INGRESS: test_session_1 DST_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:0007/128 Active
IP_PROTOCOL: 6
SRC_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:0006/128
TCP_FLAGS: 0x02/0x02
EVERFLOWV6 RULE_13 9987 MIRROR INGRESS: test_session_1 DST_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:0006/128 Active
IP_PROTOCOL: 6
SRC_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:0007/128
TCP_FLAGS: 0x12/0x12
EVERFLOWV6 RULE_14 9986 MIRROR INGRESS: test_session_1 DST_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:0009/128 Active
IP_PROTOCOL: 6
L4_DST_PORT: 443
L4_SRC_PORT: 12000
SRC_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:0008/128
TCP_FLAGS: 0x02/0x02
EVERFLOWV6 RULE_15 9985 MIRROR INGRESS: test_session_1 DST_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:0008/128 Active
IP_PROTOCOL: 6
L4_DST_PORT: 12000
L4_SRC_PORT: 443
SRC_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:0009/128
TCP_FLAGS: 0x12/0x12
EVERFLOWV6 RULE_16 9984 MIRROR INGRESS: test_session_1 DSCP: 8 Active
DST_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:000b/128
IP_PROTOCOL: 17
L4_DST_PORT: 514
L4_SRC_PORT: 12001
SRC_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:000a/128
EVERFLOWV6 RULE_17 9983 MIRROR INGRESS: test_session_1 DSCP: 8 Active
DST_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:000a/128
IP_PROTOCOL: 17
L4_DST_PORT: 12001
L4_SRC_PORT: 514
SRC_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:000b/128
EVERFLOWV6 RULE_18 9982 MIRROR INGRESS: test_session_1 DST_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:000d/128 Active
SRC_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:000c/128
EVERFLOWV6 RULE_20 9980 MIRROR INGRESS: test_session_1 DST_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:0010/128 Active
IP_PROTOCOL: 6
L4_DST_PORT: 12007
L4_SRC_PORT: 12006
SRC_IPV6: 2002:0225:7c6b:b000:0000:0000:0000:0000/64
EVERFLOWV6 RULE_21 9979 MIRROR INGRESS: test_session_1 DST_IPV6: 2002:0225:7c6b:b000:0000:0000:0000:0000/64 Active
IP_PROTOCOL: 6
L4_DST_PORT: 12009
L4_SRC_PORT: 12008
SRC_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:0010/128
EVERFLOWV6 RULE_22 9978 MIRROR INGRESS: test_session_1 DST_IPV6: 2002:0225:7c6b:d000:0000:0000:0000:0000/64 Active
IP_PROTOCOL: 6
L4_DST_PORT: 12011
L4_SRC_PORT: 12010
SRC_IPV6: 2002:0225:7c6b:c000:0000:0000:0000:0000/64
EVERFLOWV6 RULE_23 9977 MIRROR INGRESS: test_session_1 DST_IPV6: 2002:0225:7c6b:f000:0000:0000:0000:0000/81 Active
IP_PROTOCOL: 6
L4_DST_PORT: 12013
L4_SRC_PORT: 12012
SRC_IPV6: 2002:0225:7c6b:e000:0000:0000:0000:0000/57
EVERFLOWV6 RULE_24 9976 MIRROR INGRESS: test_session_1 DST_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:001d/128 Active
L4_DST_PORT: 12003
L4_SRC_PORT: 12002
SRC_IPV6: 2002:0225:7c6b:a982:d48b:230e:f271:001c/128
cisco@tor40-1:~$ show mirror_session
ERSPAN Sessions
Name Status SRC IP DST IP GRE DSCP TTL Queue Policer Monitor Port SRC Port Direction


test_session_1 active 1.1.1.1 2.2.2.2 35006 8 4 Ethernet56

Yet the pkts are sent to tor40-2, which has no ACL rules.

Describe the results you expected:
As to Active-Active dualtor test, we should config ACL rules and mirror session on both DUTs.
Or else, If pkt is sent to the dut which has no everflow config, this pkt could not be mirrored.

If we only choose one DUT to test and config ACL rules, we must ensure the pkts are sent to this DUT.

Should we also fall back to active-standby mode to test this case?

Additional information you deem important:

**Output of `show version`:**

cisco@tor40-2:~$ show ver

SONiC Software Version: SONiC.azure_cisco_202205.6334-dirty-20230907.053006
SONiC OS Version: 11
Distribution: Debian 11.7
Kernel: 5.10.0-18-2-amd64
Build commit: 3c1a5d820
Build date: Thu Sep 7 14:32:27 UTC 2023
Built by: sonicci@sonic-ci-11-lnx

Platform: x86_64-8102_64h_o-r0
HwSKU: Cisco-8102-C64
ASIC: cisco-8000
ASIC Count: 1
Serial Number: FLM26400GJB
Model Number: 8102-64H-O
Hardware Revision: 1.0
Uptime: 21:01:36 up 49 min, 2 users, load average: 0.87, 0.95, 1.00
Date: Wed 13 Sep 2023 21:01:36

(paste your output here)
```

**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions