Description of PR

Summary:
Fixes # (issue)
Fix pfcwd/test_pfcwd_warm_reboot.py.

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• Test case(new/improvement)

Approach

What is the motivation for this PR?

In some platforms and topologies(more ports, more possibility to reproduce), pfcwd/test_pfcwd_warm_reboot.py might fail if the test selects many storming ports in the platform like Arista7260. The reason is that if the test starts multiple fake storming threads, there might be a race condition when one thread tries to fetch storm_stop_defer from the test object's pfc_wd dictionary, which could be modified by those latter ports in their calls of setup_port_params, the thread will raise a KeyError, exit and leave the port in STORM DETECTED status. The test will fail because it cannot find any syslog entry generated because the port is already stormed when it tries to storm the port after warm reboot.

How did you do it?

1. Passing storm_start_defer and storm_stop_defer directly to the
   thread target function instead of retrieving from the test object's
   pfc_wd dictionary, which might cause race condition if
   setup_port_params of those latter ports clean up pfc_wd and the
   thread tries to fetch storm_stop_defer from pfc_wd.

2. Introduce thread class InterruptableThread to grant the exception
   awareness to the main thread caller when joins the thread. Use it in the
   thread creation in test_pfcwd_warm_reboot.py to make the test aware of
   any anomalies in storm threads.

3. Introduce join_all to timeout join a list of threads. Use it to
   wait for all storm threads to finish and alert if any threads don't exit
   as expected.

4. There is a chance the async storm thread make Ansible calls when the
   DUT SSH service is dead or Redis isn't started. Introducing a
   threading.Event object DUTACTIVE to make the storm thread wait till
   the DUT is up and active after warm reboot.

5. Make indentation all aligned with four-spaces.

How did you verify/test it?

Test over Arista7260 with t0-116.

pfcwd/test_pfcwd_warm_reboot.py::TestPfcwdWb::test_pfcwd_wb[no_storm] PASSED                                                                                                                                                                                           [ 33%]
pfcwd/test_pfcwd_warm_reboot.py::TestPfcwdWb::test_pfcwd_wb[storm] PASSED                                                                                                                                                                                              [ 66%]
pfcwd/test_pfcwd_warm_reboot.py::TestPfcwdWb::test_pfcwd_wb[async_storm] PASSED                                                                                                                                                                                        [100%]

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation