Enhance connection graph and devutils for PDU host vars by wangxin · Pull Request #7782 · sonic-net/sonic-mgmt

wangxin · 2023-03-18T02:01:22Z

Description of PR

Summary:
Fixes # (issue)

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

201911
202012
202205

Approach

What is the motivation for this PR?

Each DUT could be connected to multiple PDU hosts. The current implementation assumes that all the PDU hosts connected to same DUT shares same set of host variables. In case the PDU hosts are different, this would not work.

How did you do it?

This change enhanced the devutils, pdu_manager and connection graph modules to use host vars specific to the PDU hosts while getting pdu manager object for DUTs.

Add 'Hostname' field for the collected device info in connection graph.
Pass PDU information of all related PDU hosts down to pdu_manager_factory
Use host vars of current PDU host while creating pdu manager object.

How did you verify/test it?

Run devutils to get PDU status.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Each DUT could be connected to multiple PDU hosts. The current implementation assumes that all the PDU hosts connected to same DUT shares same set of host variables. In case the PDU hosts are different, this would not work. This change enhanced the devutils, pdu_manager and connection graph modules to use host vars specific to the PDU hosts while getting pdu manager object for DUTs. Signed-off-by: Xin Wang <[email protected]>

mssonicbld · 2023-03-18T02:02:26Z

The pre-commit check detected issues in the files touched by this pull request.
The detected issues may be old or new. For new issues, please try to fix them.

For old issues, it is not mandatory to fix them because they were not caused by this change. It is unfair to blame
author of this pull request. But if you can take extra effort to fix the old issues as well, that would be great!

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 ansible/devutils:7:1: E402 module level import not at top of file ansible/devutils:8:1: E402 module level import not at top of file ansible/devutils:9:1: E402 module level import not at top of file ansible/devutils:10:1: E402 module level import not at top of file ansible/devutils:11:1: E402 module level import not at top of file ansible/devutils:13:1: E402 module level import not at top of file ansible/devutils:14:1: E402 module level import not at top of file ansible/devutils:15:1: E402 module level import not at top of file ansible/devutils:16:1: E402 module level import not at top of file ansible/devutils:18:1: E402 module level import not at top of file ansible/devutils:20:1: E402 module level import not at top of file ... [truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

yejianquan

LGTM

nhe-NV · 2023-03-27T07:15:17Z

HI @wangxin
After take this PR into our regression I meet the issue when get the pdu_controller.
def _build_pdu_manager_from_graph(pduman, dut_hostname, conn_graph_facts, pdu_vars):
logger.info('Creating pdu manager from graph information')
pdu_devs = conn_graph_facts['device_pdu_info']
pdu_links = conn_graph_facts['device_pdu_links']
pdu_info = _merge_dev_link(pdu_devs, pdu_links)
if dut_hostname not in pdu_info or not pdu_info[dut_hostname]:
# No PDU information in graph
logger.info('PDU informatin for {} is not found in graph'.format(dut_hostname))
return False

    for psu_name, psu_peer in list(pdu_info[dut_hostname].items()):

      pduman.add_controller(psu_name, psu_peer, pdu_vars[psu_peer['Hostname']])

E KeyError: u'pdu-10-208-0-158'

I see for the pdu_vars, the value is {'snmp_rocommunity': 'public', 'snmp_rwcommunity': 'private'}
do we need to update some xml file to make the test pass?

Thanks
Nana

wangxin · 2023-03-28T01:11:33Z

Yes, I noticed that. I'll submit a fix soon. Thanks for reporting this!

nhe-NV · 2023-03-28T01:12:58Z

Thanks! please share the PR once it is ready.

PR sonic-net#7782 updated the pdu_manager_factory interface. Only devutils using this function was updated accordingly. Fixtures like pdu_controller using this function also need to be updated. This fixture updated the pdu_controller and get_pdu_controller fixtures accordingly to pass correct PDU variables to the pdu_manager_factory function. This change only can fix the issue under python2. Under python3, the pysnmp package version 4.4.12 is much higher than the 4.2.5 package used in python2. The newer package has compatibility issues. The current snmp_pdu_controllers.py script needs to be updated for python3. This issue will be addressed in other PRs. Signed-off-by: Xin Wang <[email protected]>

wangxin · 2023-03-28T06:02:48Z

@nhe-NV Fixed in #7876

nhe-NV · 2023-03-28T10:12:42Z

@nhe-NV Fixed in #7876

Thanks, Will take this fix and test

What is the motivation for this PR? PR #7782 updated the pdu_manager_factory interface. Only devutils using this function was updated accordingly. Fixtures like pdu_controller using this function also need to be updated. How did you do it? This PR updated the pdu_controller and get_pdu_controller fixtures accordingly to pass correct PDU variables to the pdu_manager_factory function. This change only can fix the issue under python2. Under python3, the pysnmp package version 4.4.12 is much higher than the 4.2.5 package used in python2. The newer package has compatibility issues. The current snmp_pdu_controllers.py script needs to be updated for python3. That issue is fixed in #7887 Signed-off-by: Xin Wang <[email protected]>

* Fix "logging not defined" issue of some snappi tests (sonic-net#7855) What is the motivation for this PR? The logging module is used by some files under tests.snappi. However, the module is not imported in the files. This caused error like below while collecting the snappi tests: E NameError: name 'logging' is not defined How did you do it? The fix is to import logging in those files. Signed-off-by: Xin Wang <[email protected]> * [Python3 migration]Fix oid issue in snmp_pdu_controllers.py sonic-net#7887 What is the motivation for this PR? In Python2, oid.prettyPrint() prints 1.3.6.1.4.1.9.1.283 In Python3, oid.prettyPrint() prints SNMPv2-SMI::enterprises.9.1.283 This difference breaks port_label_dict in _build_outlet_maps() in snmp_pdu_controllers.py How did you do it? Change oid.prettyPrint() to str(oid). This will make code work both in Python2 and Python3. How did you verify/test it? PDU testcase can run manually both in Python2 and Python3. * Fix pdu KeyError for pdu_controller fixture (sonic-net#7876) What is the motivation for this PR? PR sonic-net#7782 updated the pdu_manager_factory interface. Only devutils using this function was updated accordingly. Fixtures like pdu_controller using this function also need to be updated. How did you do it? This PR updated the pdu_controller and get_pdu_controller fixtures accordingly to pass correct PDU variables to the pdu_manager_factory function. This change only can fix the issue under python2. Under python3, the pysnmp package version 4.4.12 is much higher than the 4.2.5 package used in python2. The newer package has compatibility issues. The current snmp_pdu_controllers.py script needs to be updated for python3. That issue is fixed in sonic-net#7887 Signed-off-by: Xin Wang <[email protected]> * Make test_plan.py compatible between python2 and python3 (sonic-net#7886) * Make test_plan.py compatible between python2 and python3 Approach What is the motivation for this PR? Make test_plan.py compatible between python2 and python3 co-authorized by: [email protected] * RDMA pfc global pause multi dut test * Refine testbedv2 template output (sonic-net#7895) * Refine testbedv2 template output Approach What is the motivation for this PR? Refine testbedv2 template output Co-authorized by: [email protected] * fix test_route_perf packet verify issue (sonic-net#7877) * fix route perf failure caused by missing some port-channel members * Skip warm-reboot cases for T2. (sonic-net#7784) --------- Signed-off-by: Xin Wang <[email protected]> Co-authored-by: Xin Wang <[email protected]> Co-authored-by: ShiyanWangMS <[email protected]> Co-authored-by: Ye Jianquan <[email protected]> Co-authored-by: jcaiMR <[email protected]> Co-authored-by: rraghav-cisco <[email protected]>

What is the motivation for this PR? Each DUT could be connected to multiple PDU hosts. The current implementation assumes that all the PDU hosts connected to same DUT shares same set of host variables. In case the PDU hosts are different, this would not work. How did you do it? This change enhanced the devutils, pdu_manager and connection graph modules to use host vars specific to the PDU hosts while getting pdu manager object for DUTs. * Add 'Hostname' field for the collected device info in connection graph. * Pass PDU information of all related PDU hosts down to pdu_manager_factory * Use host vars of current PDU host while creating pdu manager object. How did you verify/test it? Run devutils to get PDU status. Signed-off-by: Xin Wang <[email protected]>

What is the motivation for this PR? PR #7782 updated the pdu_manager_factory interface. Only devutils using this function was updated accordingly. Fixtures like pdu_controller using this function also need to be updated. How did you do it? This PR updated the pdu_controller and get_pdu_controller fixtures accordingly to pass correct PDU variables to the pdu_manager_factory function. This change only can fix the issue under python2. Under python3, the pysnmp package version 4.4.12 is much higher than the 4.2.5 package used in python2. The newer package has compatibility issues. The current snmp_pdu_controllers.py script needs to be updated for python3. That issue is fixed in #7887 Signed-off-by: Xin Wang <[email protected]>

What is the motivation for this PR? Each DUT could be connected to multiple PDU hosts. The current implementation assumes that all the PDU hosts connected to same DUT shares same set of host variables. In case the PDU hosts are different, this would not work. How did you do it? This change enhanced the devutils, pdu_manager and connection graph modules to use host vars specific to the PDU hosts while getting pdu manager object for DUTs. * Add 'Hostname' field for the collected device info in connection graph. * Pass PDU information of all related PDU hosts down to pdu_manager_factory * Use host vars of current PDU host while creating pdu manager object. How did you verify/test it? Run devutils to get PDU status. Signed-off-by: Xin Wang <[email protected]>

What is the motivation for this PR? PR #7782 updated the pdu_manager_factory interface. Only devutils using this function was updated accordingly. Fixtures like pdu_controller using this function also need to be updated. How did you do it? This PR updated the pdu_controller and get_pdu_controller fixtures accordingly to pass correct PDU variables to the pdu_manager_factory function. This change only can fix the issue under python2. Under python3, the pysnmp package version 4.4.12 is much higher than the 4.2.5 package used in python2. The newer package has compatibility issues. The current snmp_pdu_controllers.py script needs to be updated for python3. That issue is fixed in #7887 Signed-off-by: Xin Wang <[email protected]>

…frastructure (#7849) * Added Multi Dut Multi asic Support * PreCommit Check * Update common_helpers.py * Rdma multiasic global pause (#9) * Fix "logging not defined" issue of some snappi tests (#7855) What is the motivation for this PR? The logging module is used by some files under tests.snappi. However, the module is not imported in the files. This caused error like below while collecting the snappi tests: E NameError: name 'logging' is not defined How did you do it? The fix is to import logging in those files. Signed-off-by: Xin Wang <[email protected]> * [Python3 migration]Fix oid issue in snmp_pdu_controllers.py #7887 What is the motivation for this PR? In Python2, oid.prettyPrint() prints 1.3.6.1.4.1.9.1.283 In Python3, oid.prettyPrint() prints SNMPv2-SMI::enterprises.9.1.283 This difference breaks port_label_dict in _build_outlet_maps() in snmp_pdu_controllers.py How did you do it? Change oid.prettyPrint() to str(oid). This will make code work both in Python2 and Python3. How did you verify/test it? PDU testcase can run manually both in Python2 and Python3. * Fix pdu KeyError for pdu_controller fixture (#7876) What is the motivation for this PR? PR #7782 updated the pdu_manager_factory interface. Only devutils using this function was updated accordingly. Fixtures like pdu_controller using this function also need to be updated. How did you do it? This PR updated the pdu_controller and get_pdu_controller fixtures accordingly to pass correct PDU variables to the pdu_manager_factory function. This change only can fix the issue under python2. Under python3, the pysnmp package version 4.4.12 is much higher than the 4.2.5 package used in python2. The newer package has compatibility issues. The current snmp_pdu_controllers.py script needs to be updated for python3. That issue is fixed in #7887 Signed-off-by: Xin Wang <[email protected]> * Make test_plan.py compatible between python2 and python3 (#7886) * Make test_plan.py compatible between python2 and python3 Approach What is the motivation for this PR? Make test_plan.py compatible between python2 and python3 co-authorized by: [email protected] * RDMA pfc global pause multi dut test * Refine testbedv2 template output (#7895) * Refine testbedv2 template output Approach What is the motivation for this PR? Refine testbedv2 template output Co-authorized by: [email protected] * fix test_route_perf packet verify issue (#7877) * fix route perf failure caused by missing some port-channel members * Skip warm-reboot cases for T2. (#7784) --------- Signed-off-by: Xin Wang <[email protected]> Co-authored-by: Xin Wang <[email protected]> Co-authored-by: ShiyanWangMS <[email protected]> Co-authored-by: Ye Jianquan <[email protected]> Co-authored-by: jcaiMR <[email protected]> Co-authored-by: rraghav-cisco <[email protected]> * removing 2nd asic value * review feedback changes * removing global pause case from infra changes * removing qos fixture from this branch * namespace to asic_value * resolving comments Signed-off-by: selldinesh <[email protected]> * resolving comments Signed-off-by: selldinesh <[email protected]> * typo fix * removing mlnx only * adding qos fixtures file back * pre commit fix * pre commit fix * changing 'None' to None * resolving precommit errors * resolving comments * adding a short description about this file * review comment fixes * removing quotes * review changes * review changes * review changes * review changes --------- Signed-off-by: Xin Wang <[email protected]> Signed-off-by: selldinesh <[email protected]> Co-authored-by: vkuma82 <[email protected]> Co-authored-by: Xin Wang <[email protected]> Co-authored-by: ShiyanWangMS <[email protected]> Co-authored-by: Ye Jianquan <[email protected]> Co-authored-by: jcaiMR <[email protected]> Co-authored-by: rraghav-cisco <[email protected]> Co-authored-by: Vinod Kumar <[email protected]> Co-authored-by: selldinesh <[email protected]>

…frastructure (sonic-net#7849) * Added Multi Dut Multi asic Support * PreCommit Check * Update common_helpers.py * Rdma multiasic global pause (sonic-net#9) * Fix "logging not defined" issue of some snappi tests (sonic-net#7855) What is the motivation for this PR? The logging module is used by some files under tests.snappi. However, the module is not imported in the files. This caused error like below while collecting the snappi tests: E NameError: name 'logging' is not defined How did you do it? The fix is to import logging in those files. Signed-off-by: Xin Wang <[email protected]> * [Python3 migration]Fix oid issue in snmp_pdu_controllers.py sonic-net#7887 What is the motivation for this PR? In Python2, oid.prettyPrint() prints 1.3.6.1.4.1.9.1.283 In Python3, oid.prettyPrint() prints SNMPv2-SMI::enterprises.9.1.283 This difference breaks port_label_dict in _build_outlet_maps() in snmp_pdu_controllers.py How did you do it? Change oid.prettyPrint() to str(oid). This will make code work both in Python2 and Python3. How did you verify/test it? PDU testcase can run manually both in Python2 and Python3. * Fix pdu KeyError for pdu_controller fixture (sonic-net#7876) What is the motivation for this PR? PR sonic-net#7782 updated the pdu_manager_factory interface. Only devutils using this function was updated accordingly. Fixtures like pdu_controller using this function also need to be updated. How did you do it? This PR updated the pdu_controller and get_pdu_controller fixtures accordingly to pass correct PDU variables to the pdu_manager_factory function. This change only can fix the issue under python2. Under python3, the pysnmp package version 4.4.12 is much higher than the 4.2.5 package used in python2. The newer package has compatibility issues. The current snmp_pdu_controllers.py script needs to be updated for python3. That issue is fixed in sonic-net#7887 Signed-off-by: Xin Wang <[email protected]> * Make test_plan.py compatible between python2 and python3 (sonic-net#7886) * Make test_plan.py compatible between python2 and python3 Approach What is the motivation for this PR? Make test_plan.py compatible between python2 and python3 co-authorized by: [email protected] * RDMA pfc global pause multi dut test * Refine testbedv2 template output (sonic-net#7895) * Refine testbedv2 template output Approach What is the motivation for this PR? Refine testbedv2 template output Co-authorized by: [email protected] * fix test_route_perf packet verify issue (sonic-net#7877) * fix route perf failure caused by missing some port-channel members * Skip warm-reboot cases for T2. (sonic-net#7784) --------- Signed-off-by: Xin Wang <[email protected]> Co-authored-by: Xin Wang <[email protected]> Co-authored-by: ShiyanWangMS <[email protected]> Co-authored-by: Ye Jianquan <[email protected]> Co-authored-by: jcaiMR <[email protected]> Co-authored-by: rraghav-cisco <[email protected]> * removing 2nd asic value * review feedback changes * removing global pause case from infra changes * removing qos fixture from this branch * namespace to asic_value * resolving comments Signed-off-by: selldinesh <[email protected]> * resolving comments Signed-off-by: selldinesh <[email protected]> * typo fix * removing mlnx only * adding qos fixtures file back * pre commit fix * pre commit fix * changing 'None' to None * resolving precommit errors * resolving comments * adding a short description about this file * review comment fixes * removing quotes * review changes * review changes * review changes * review changes --------- Signed-off-by: Xin Wang <[email protected]> Signed-off-by: selldinesh <[email protected]> Co-authored-by: vkuma82 <[email protected]> Co-authored-by: Xin Wang <[email protected]> Co-authored-by: ShiyanWangMS <[email protected]> Co-authored-by: Ye Jianquan <[email protected]> Co-authored-by: jcaiMR <[email protected]> Co-authored-by: rraghav-cisco <[email protected]> Co-authored-by: Vinod Kumar <[email protected]> Co-authored-by: selldinesh <[email protected]>

wangxin requested a review from bingwang-ms March 18, 2023 02:01

wangxin requested a review from yejianquan March 20, 2023 07:05

yejianquan approved these changes Mar 20, 2023

View reviewed changes

wangxin merged commit 94ea97b into sonic-net:master Mar 20, 2023

wangxin mentioned this pull request Mar 28, 2023

Fix pdu KeyError for pdu_controller fixture #7876

Merged

6 tasks

wangxin added Request for 202205 branch Request for 202012 branch labels Apr 27, 2023

wangxin added the Included in 202012 branch label Apr 27, 2023

wangxin added the Included in 202205 branch label Apr 27, 2023

wangxin deleted the conng-pdu branch July 12, 2023 02:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhance connection graph and devutils for PDU host vars#7782

Enhance connection graph and devutils for PDU host vars#7782
wangxin merged 1 commit intosonic-net:masterfrom
wangxin:conng-pdu

wangxin commented Mar 18, 2023 •

edited

Loading

Uh oh!

mssonicbld commented Mar 18, 2023

Uh oh!

yejianquan left a comment

Uh oh!

nhe-NV commented Mar 27, 2023

Uh oh!

wangxin commented Mar 28, 2023

Uh oh!

nhe-NV commented Mar 28, 2023

Uh oh!

wangxin commented Mar 28, 2023

Uh oh!

nhe-NV commented Mar 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

wangxin commented Mar 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

mssonicbld commented Mar 18, 2023

Uh oh!

yejianquan left a comment

Choose a reason for hiding this comment

Uh oh!

nhe-NV commented Mar 27, 2023

Uh oh!

wangxin commented Mar 28, 2023

Uh oh!

nhe-NV commented Mar 28, 2023

Uh oh!

wangxin commented Mar 28, 2023

Uh oh!

nhe-NV commented Mar 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wangxin commented Mar 18, 2023 •

edited

Loading