Disable Cisco VOQ WD for dualtor QOS, improve PG test debuggability by rbpittman · Pull Request #20502 · sonic-net/sonic-mgmt

rbpittman · 2025-09-03T21:13:32Z

Description of PR

Summary:

Disable VOQ WD during test_tunnel_qos_remap.py
Commonize some aspects of the VOQ WD disabling from QOS SAI to avoid major code duplication.
Improve performance of test for cisco by reducing unnecessary 8-second per loop wait time to 1 second. (Test passed with 0.5 seconds as well, since this is a SAI bypass operation the updated stat should be near-instant).
Improve debuggability of PG tunnel decap test by logging all failures and summarizing a report at the end.
Fix up test_voq_watchdog.py test with new commonization. Rename parametrization to avoid shadowing.

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Validated:

test_tunnel_qos_remap.py all tests pass on master branch for Cisco-8000 dualtor-AA.
test_qos_sai.py no regressions for Cisco-8000 dualtor-AA.
Fix and validate test_voq_watchdog.py

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

…ility and speed of the tunnel PG map test.

mssonicbld · 2025-09-03T21:13:39Z

/azp run

azure-pipelines · 2025-09-03T21:13:53Z

Azure Pipelines successfully started running 1 pipeline(s).

tests/qos/qos_sai_base.py

mssonicbld · 2025-09-04T13:56:24Z

/azp run

azure-pipelines · 2025-09-04T13:56:40Z

Azure Pipelines successfully started running 1 pipeline(s).

kevinskwang · 2025-09-05T01:11:07Z

@rbpittman can I understand the voq watchdog will impact the watermark counter function? if that is the case, how we can get pass in test_qos_sai?

rbpittman · 2025-09-05T02:52:45Z

No it doesn't impact watermark function.
test_qos_sai already disables voq_watchdog, hence this is adding the same disable to the tunnel tests.
The voq watchdog activates when queues are blocked for a long time, thus it needs to be disabled in long-running TX_DISABLE tests. In this case, the PG watermark would stop going up due to the VOQ watchdog flushing queues.

mssonicbld · 2025-09-08T06:50:35Z

@rbpittman PR conflicts with 202411 branch

kevinskwang · 2025-09-08T06:53:35Z

@rbpittman could you create PR to fix the conflicts on 202411?

rbpittman · 2025-09-08T20:42:34Z

Hi @kevinskwang
There's some significant conflicts, most notably missing #17937 from 202411, which disables VOQ WD for QOS SAI. That PR should at least be merged first.

kevinskwang · 2025-09-09T09:06:47Z

Hi @kevinskwang There's some significant conflicts, most notably missing #17937 from 202411, which disables VOQ WD for QOS SAI. That PR should at least be merged first.

@rbpittman 17937 is cherry-picked to 202411 today.

rbpittman · 2025-09-09T15:28:39Z

Hi @kevinskwang
cc @zhixzhu @rraghav-cisco

There's quite a bit more missing code here to support this diff, the question is what features we actually want in the 202411 branch.

We can either:

Try to merge more of the missing code, for example

T2:snappi_tests:Add a fixture to disable voq watchdog, use it in pfc scripts. #18564
New test case for OQ watchdog #18937
Consolidate voq watchdog testcases to single testcase #19076
This is a lot of delta, even including new test cases. Plus this list is likely not complete.

I can write a custom version of this PR for 202411, however this will cause all the above PRs to also require custom patches. Depends whether these PRs are needed in 202411.

Both options are difficult, what is our prioritization for 202411 maintenance? If we care more about just fixing it for pass-rates I can write a custom patch so test_tunnel can pass, but this would assume we don't want the other PRs.

zhixzhu · 2025-09-09T16:58:14Z

Hi @kevinskwang cc @zhixzhu @rraghav-cisco

There's quite a bit more missing code here to support this diff, the question is what features we actually want in the 202411 branch.

We can either:
1. Try to merge more of the missing code, for example


* [T2:snappi_tests:Add a fixture to disable voq watchdog, use it in pfc scripts. #18564](https://github.com/sonic-net/sonic-mgmt/pull/18564)

* [New test case for OQ watchdog #18937](https://github.com/sonic-net/sonic-mgmt/pull/18937)

* [Consolidate voq watchdog testcases to single testcase #19076](https://github.com/sonic-net/sonic-mgmt/pull/19076)
  This is a lot of delta, even including new test cases. Plus this list is likely not complete.


2. I can write a custom version of this PR for 202411, however this will cause all the above PRs to also require custom patches. Depends whether these PRs are needed in 202411.
Both options are difficult, what is our prioritization for 202411 maintenance? If we care more about just fixing it for pass-rates I can write a custom patch so test_tunnel can pass, but this would assume we don't want the other PRs.

Yes, these PRs are required in 202411 branch. T0/T1 also enabled voq watchdog and oq watchdog.

rbpittman · 2025-09-09T17:16:39Z

Then this PR should be considered blocked until at minimum those 3 PRs are merged, in the correct order, and with any missing dependency gaps filled in as that's not necessarily a complete list.

kevinskwang · 2025-09-10T02:36:38Z

@zhixzhu those 3 PRs all have conflicts, could you fix the confilct by first for 202411?

zhixzhu · 2025-09-11T00:08:01Z

@zhixzhu those 3 PRs all have conflicts, could you fix the confilct by first for 202411?

Double commit New test case for OQ watchdog #18937 to 202411: #20610

After the above one merged, I will check Consolidate voq watchdog testcases to single testcase #19076, it depends on the above one.

zhixzhu · 2025-09-15T18:03:39Z

Double commit Consolidate voq watchdog testcases to single testcase #19076 to 202411: #20660

…onic-net#20502) * Disable Cisco VOQ watchdog for dualtor tunnel tests. Improve debuggability and speed of the tunnel PG map test. * Fix VOQ watchdog test.

zhixzhu · 2025-09-17T22:03:35Z

@kevinskwang #20660 is waiting for your review and merge.

kevinskwang · 2025-09-24T08:02:13Z

@zhixzhu Merged

rbpittman · 2025-09-24T17:33:45Z

Based on latest conflicts, looks like #19821 is the last double commit missing.

…onic-net#20502) * Disable Cisco VOQ watchdog for dualtor tunnel tests. Improve debuggability and speed of the tunnel PG map test. * Fix VOQ watchdog test.

rbpittman · 2025-09-30T14:00:19Z

@kevinskwang Filed #20870 for the double commit to 202411. Had some import header conflicts due to the recent "noqa" to "noqa:" conversion. No other feature conflicts.

…onic-net#20502) * Disable Cisco VOQ watchdog for dualtor tunnel tests. Improve debuggability and speed of the tunnel PG map test. * Fix VOQ watchdog test.

rbpittman · 2025-09-30T14:13:42Z

Also filed #20871 for 202505 branch (which had the same set of noqa conflicts).

…20502) (#20870) Double commit #20502 Disable VOQ WD during test_tunnel_qos_remap.py Commonize some aspects of the VOQ WD disabling from QOS SAI to avoid major code duplication. Improve performance of test for cisco by reducing unnecessary 8-second per loop wait time to 1 second. (Test passed with 0.5 seconds as well, since this is a SAI bypass operation the updated stat should be near-instant). Improve debuggability of PG tunnel decap test by logging all failures and summarizing a report at the end. Fix up test_voq_watchdog.py test with new commonization. Rename parametrization to avoid shadowing.

…onic-net#20502) * Disable Cisco VOQ watchdog for dualtor tunnel tests. Improve debuggability and speed of the tunnel PG map test. * Fix VOQ watchdog test.

…20502) (#20871) Summary: Double commit #20502 Disable VOQ WD during test_tunnel_qos_remap.py Commonize some aspects of the VOQ WD disabling from QOS SAI to avoid major code duplication. Improve performance of test for cisco by reducing unnecessary 8-second per loop wait time to 1 second. (Test passed with 0.5 seconds as well, since this is a SAI bypass operation the updated stat should be near-instant). Improve debuggability of PG tunnel decap test by logging all failures and summarizing a report at the end. Fix up test_voq_watchdog.py test with new commonization. Rename parametrization to avoid shadowing.

…onic-net#20502) * Disable Cisco VOQ watchdog for dualtor tunnel tests. Improve debuggability and speed of the tunnel PG map test. * Fix VOQ watchdog test. Signed-off-by: opcoder0 <[email protected]>

…onic-net#20502) * Disable Cisco VOQ watchdog for dualtor tunnel tests. Improve debuggability and speed of the tunnel PG map test. * Fix VOQ watchdog test. Signed-off-by: Guy Shemesh <[email protected]>

…onic-net#20502) * Disable Cisco VOQ watchdog for dualtor tunnel tests. Improve debuggability and speed of the tunnel PG map test. * Fix VOQ watchdog test. Signed-off-by: Aharon Malkin <[email protected]>

…onic-net#20502) * Disable Cisco VOQ watchdog for dualtor tunnel tests. Improve debuggability and speed of the tunnel PG map test. * Fix VOQ watchdog test. Signed-off-by: Guy Shemesh <[email protected]>

…onic-net#20502) * Disable Cisco VOQ watchdog for dualtor tunnel tests. Improve debuggability and speed of the tunnel PG map test. * Fix VOQ watchdog test.

…onic-net#20502) * Disable Cisco VOQ watchdog for dualtor tunnel tests. Improve debuggability and speed of the tunnel PG map test. * Fix VOQ watchdog test. Signed-off-by: Guy Shemesh <[email protected]>

…onic-net#20502) * Disable Cisco VOQ watchdog for dualtor tunnel tests. Improve debuggability and speed of the tunnel PG map test. * Fix VOQ watchdog test. Signed-off-by: Lakshmi Yarramaneni <[email protected]>

…onic-net#20502) * Disable Cisco VOQ watchdog for dualtor tunnel tests. Improve debuggability and speed of the tunnel PG map test. * Fix VOQ watchdog test. Signed-off-by: Yael Tzur <[email protected]>

…atically (sonic-net#20502) #### Why I did it src/sonic-snmpagent ``` * 7580ae2 - (HEAD -> 202311, origin/202311) Fix SNMP output having fewer unicast queues than expected (sonic-net#330) (19 hours ago) [Justin Wong] ``` #### How I did it #### How to verify it #### Description for the changelog

…onic-net#20502) * Disable Cisco VOQ watchdog for dualtor tunnel tests. Improve debuggability and speed of the tunnel PG map test. * Fix VOQ watchdog test.

Disable Cisco VOQ watchdog for dualtor tunnel tests. Improve debuggab…

1084f7f

…ility and speed of the tunnel PG map test.

zhixzhu reviewed Sep 3, 2025

View reviewed changes

tests/qos/qos_sai_base.py Show resolved Hide resolved

Fix VOQ watchdog test.

07a062b

zhixzhu approved these changes Sep 4, 2025

View reviewed changes

kevinskwang approved these changes Sep 8, 2025

View reviewed changes

kevinskwang merged commit d61736c into sonic-net:master Sep 8, 2025
16 checks passed

kevinskwang added Request for 202411 branch Approved for 202411 branch labels Sep 8, 2025

mssonicbld added the Cherry Pick Conflict_202411 label Sep 8, 2025

rbpittman deleted the disable_voq_wd_tunnel_tests_master branch September 8, 2025 20:20

rbpittman mentioned this pull request Sep 30, 2025

[202411] Disable Cisco VOQ WD for dualtor QOS, improve PG test debuggability #20870

Merged

11 tasks

rbpittman mentioned this pull request Sep 30, 2025

[202505] Disable Cisco VOQ WD for dualtor QOS, improve PG test debuggability #20871

Merged

11 tasks

Conversation

rbpittman commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

mssonicbld commented Sep 3, 2025

Uh oh!

azure-pipelines bot commented Sep 3, 2025

Uh oh!

Uh oh!

mssonicbld commented Sep 4, 2025

Uh oh!

azure-pipelines bot commented Sep 4, 2025

Uh oh!

kevinskwang commented Sep 5, 2025

Uh oh!

rbpittman commented Sep 5, 2025

Uh oh!

Uh oh!

mssonicbld commented Sep 8, 2025

Uh oh!

kevinskwang commented Sep 8, 2025

Uh oh!

rbpittman commented Sep 8, 2025

Uh oh!

kevinskwang commented Sep 9, 2025

Uh oh!

rbpittman commented Sep 9, 2025

Uh oh!

zhixzhu commented Sep 9, 2025

Uh oh!

rbpittman commented Sep 9, 2025

Uh oh!

kevinskwang commented Sep 10, 2025

Uh oh!

zhixzhu commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhixzhu commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhixzhu commented Sep 17, 2025

Uh oh!

kevinskwang commented Sep 24, 2025

Uh oh!

rbpittman commented Sep 24, 2025

Uh oh!

rbpittman commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rbpittman commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rbpittman commented Sep 3, 2025 •

edited

Loading

zhixzhu commented Sep 11, 2025 •

edited

Loading

zhixzhu commented Sep 15, 2025 •

edited

Loading

rbpittman commented Sep 30, 2025 •

edited

Loading

rbpittman commented Sep 30, 2025 •

edited

Loading