Is it platform specific
generic
Importance or Severity
High
Description of the bug
The PFC - test_pfc_pause_lossless_with_snappi has testcases to test with single lossless priority, multiple lossless priority and same testcases with reboot combinations.
Following is the definition of reboot testcase:
@pytest.mark.disable_loganalyzer
def test_pfc_pause_single_lossless_prio_reboot(snappi_api, # noqa: F811
conn_graph_facts, # noqa: F811
fanout_graph_facts_multidut, # noqa: F811
duthosts,
localhost,
enum_one_dut_lossless_prio_with_completeness_level, # noqa: F811
prio_dscp_map, # noqa: F811
lossless_prio_list, # noqa: F811
all_prio_list, # noqa: F811
get_snappi_ports, # noqa: F811
tbinfo, # noqa: F811
setup_ports_and_dut, # noqa: F811
disable_pfcwd, # noqa: F811
reboot_duts): # noqa: F811
Here as part of the initial test-setup, the disable_pfcwd happens which disables PFCWD and credit-WD on the DUT and saves the configuration.
Then the system undergoes reboot with saved configuration.
@pytest.fixture(params=['warm', 'cold', 'fast'])
def reboot_duts(setup_ports_and_dut, localhost, request):
reboot_type = request.param
_, _, snappi_ports = setup_ports_and_dut
skip_warm_reboot(snappi_ports[0]['duthost'], reboot_type)
skip_warm_reboot(snappi_ports[1]['duthost'], reboot_type)
def save_config_and_reboot(node, results=None):
up_bgp_neighbors = node.get_bgp_neighbors_per_asic("established")
logger.info("Issuing a {} reboot on the dut {}".format(reboot_type, node.hostname))
node.shell("mkdir /etc/sonic/orig_configs; mv /etc/sonic/config_db* /etc/sonic/orig_configs/")
node.shell("sudo config save -y")
reboot(node, localhost, reboot_type=reboot_type, safe_reboot=True)
logger.info("Wait until the system is stable")
wait_until(180, 20, 0, node.critical_services_fully_started)
wait_until(180, 20, 0, check_interface_status_of_up_ports, node)
wait_until(300, 10, 0, node.check_bgp_session_state_all_asics, up_bgp_neighbors, "established")
Once it comes up, PFCWD is disabled however, due to reboot, the credit-WD configuration is cleared and reset back to the original as it is not persistent across reboots.
This causes the lossless priority frames (test flow) to continuously send packets even though pause frames are being sent. The run_traffic in traffic_generation.py does not detect the test-stop and hence will timeout with following error:
> pytest_assert(attempts < max_attempts,
"Flows do not stop in {} seconds".format(max_attempts))
E Failed: Flows do not stop in 20 seconds
Workaround:
- Simple workaround is to shift the order in which the reboot_dut and disable_pfcwd is called in the test:
@pytest.mark.disable_loganalyzer
def test_pfc_pause_single_lossless_prio_reboot(snappi_api, # noqa: F811
conn_graph_facts, # noqa: F811
fanout_graph_facts_multidut, # noqa: F811
duthosts,
localhost,
enum_one_dut_lossless_prio_with_completeness_level, # noqa: F811
prio_dscp_map, # noqa: F811
lossless_prio_list, # noqa: F811
all_prio_list, # noqa: F811
get_snappi_ports, # noqa: F811
tbinfo, # noqa: F811
reboot_duts, # noqa: F811
setup_ports_and_dut, # noqa: F811
disable_pfcwd): # noqa: F811
Move the reboot_duts ahead of disable_pfcwd and this should work.
PR #18105 created to handle this issue.
Steps to Reproduce
- Put breakpoint for traffic_generation.py in function run_traffic before the test-starts.
- Check the PFCWD and Credit-WD config on the DUT.
- Start the reboot case.
- On the breakpoint, check the PFCWD and Credit-WD on the DUT.
- Start the testcase manually.
- Check if the test-flows stop or not after the configured interval.
- Disable Credit-WD manually on the DUT.
- Re-run the same testcase manually.
- Check if the test-flows stop or not.
Actual Behavior and Expected Behavior
Expected behavior:
The IXIA should stop sending packets to the DUT.
Actual behavior:
IXIA keeps on sending packets as Credit-WD is enabled on reboot.
Relevant log output
Output of show version
Attach files (if any)
No response
Is it platform specific
generic
Importance or Severity
High
Description of the bug
The PFC - test_pfc_pause_lossless_with_snappi has testcases to test with single lossless priority, multiple lossless priority and same testcases with reboot combinations.
Following is the definition of reboot testcase:
Here as part of the initial test-setup, the disable_pfcwd happens which disables PFCWD and credit-WD on the DUT and saves the configuration.
Then the system undergoes reboot with saved configuration.
Once it comes up, PFCWD is disabled however, due to reboot, the credit-WD configuration is cleared and reset back to the original as it is not persistent across reboots.
This causes the lossless priority frames (test flow) to continuously send packets even though pause frames are being sent. The run_traffic in traffic_generation.py does not detect the test-stop and hence will timeout with following error:
Workaround:
Move the reboot_duts ahead of disable_pfcwd and this should work.
PR #18105 created to handle this issue.
Steps to Reproduce
Actual Behavior and Expected Behavior
Expected behavior:
The IXIA should stop sending packets to the DUT.
Actual behavior:
IXIA keeps on sending packets as Credit-WD is enabled on reboot.
Relevant log output
Output of
show versionAttach files (if any)
No response