Skip to content

fix the everflow ipv6 failure on dualtor A-A#10228

Merged
kevinskwang merged 6 commits intosonic-net:masterfrom
yanjundeng:everflow6_a_a_fix
Mar 20, 2024
Merged

fix the everflow ipv6 failure on dualtor A-A#10228
kevinskwang merged 6 commits intosonic-net:masterfrom
yanjundeng:everflow6_a_a_fix

Conversation

@yanjundeng
Copy link
Copy Markdown
Contributor

@yanjundeng yanjundeng commented Oct 4, 2023

Description of PR

After debug and investigation, this failure of "test_everflow_ipv6" on dualtor A-A including two issues:

  1. the mux status is standby on the selected DUT.
  2. the test pkts are not sent to selected DUT, which has everflow rule config.

The rootcause of 1 is due to BGP shutdown. In the script, it tries to shutdown all BGP during setup, which makes T1 not reachable from the TOR. Then the DUT moves itself into standby mode. That's from the design of mux state machine.

The rootcause of 2 is similiar to the following issue:
#8158

We need fall back to active-standby mode to test this scenario for
active-active ports.

Summary:
Fixes # (9983)

Type of change

  • Bug fix
  • Testbed and Framework(new/improvement)
  • Test case(new/improvement)

Back port request

  • 201911
  • 202012
  • 202205
  • 202305

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Verified this fix on A-A, A-S and single T0.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

@mssonicbld
Copy link
Copy Markdown
Collaborator

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing tests/everflow/everflow_test_utilities.py
Fixing tests/everflow/test_everflow_ipv6.py

fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/everflow/test_everflow_ipv6.py:21:10: F821 undefined name 'logging'
tests/everflow/test_everflow_ipv6.py:23:1: E302 expected 2 blank lines, found 1

flake8...............................................(no files to check)Skipped
...
[truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

  1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
    the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
    docker container.
  2. Ensure that the pre-commit package is installed:
sudo pip install pre-commit
  1. Go to repository root folder
  2. Install the pre-commit hooks:
pre-commit install
  1. Use pre-commit to check staged file:
pre-commit
  1. Alternatively, you can check committed files using:
pre-commit run --from-ref <commit_id> --to-ref <commit_id>

@mssonicbld
Copy link
Copy Markdown
Collaborator

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing tests/everflow/everflow_test_utilities.py
Fixing tests/everflow/test_everflow_ipv6.py

fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/everflow/test_everflow_ipv6.py:21:10: F821 undefined name 'logging'
tests/everflow/test_everflow_ipv6.py:23:1: E302 expected 2 blank lines, found 1

flake8...............................................(no files to check)Skipped
...
[truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

  1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
    the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
    docker container.
  2. Ensure that the pre-commit package is installed:
sudo pip install pre-commit
  1. Go to repository root folder
  2. Install the pre-commit hooks:
pre-commit install
  1. Use pre-commit to check staged file:
pre-commit
  1. Alternatively, you can check committed files using:
pre-commit run --from-ref <commit_id> --to-ref <commit_id>

@yanjundeng
Copy link
Copy Markdown
Contributor Author

@kevinskwang pls assign one feature owner to help review this PR.

Comment thread tests/everflow/test_everflow_ipv6.py
@svelamal
Copy link
Copy Markdown
Contributor

Please add backport request to 202305 branch too

@XuChen-MSFT
Copy link
Copy Markdown
Contributor

observed below failure:


==================================== ERRORS ====================================
_______________ ERROR collecting everflow/test_everflow_ipv6.py ________________
/usr/local/lib/python3.8/dist-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
        cls        = <class '_pytest.runner.CallInfo'>
        duration   = 0.0033242040080949664
        excinfo    = <ExceptionInfo NameError("name 'logging' is not defined") tblen=16>
        func       = <function pytest_make_collect_report.<locals>.<lambda> at 0x7f801c844ee0>
        precise_start = 154328.703899066
        precise_stop = 154328.70722327
        reraise    = None
        result     = None
        start      = 1706037003.8554313
        stop       = 1706037003.8587563
        when       = 'collect'
/usr/local/lib/python3.8/dist-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
        collector  = <Module test_everflow_ipv6.py>
/usr/local/lib/python3.8/dist-packages/_pytest/python.py:531: in collect
    self._inject_setup_module_fixture()
        __class__  = <class '_pytest.python.Module'>
        self       = <Module test_everflow_ipv6.py>
/usr/local/lib/python3.8/dist-packages/_pytest/python.py:545: in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
        has_nose   = True
        self       = <Module test_everflow_ipv6.py>
/usr/local/lib/python3.8/dist-packages/_pytest/python.py:310: in obj
    self._obj = obj = self._getobj()
        obj        = None
        self       = <Module test_everflow_ipv6.py>
/usr/local/lib/python3.8/dist-packages/_pytest/python.py:528: in _getobj
    return self._importtestmodule()
        self       = <Module test_everflow_ipv6.py>
/usr/local/lib/python3.8/dist-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
        importmode = 'prepend'
        self       = <Module test_everflow_ipv6.py>
/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:565: in import_path
    importlib.import_module(module_name)
        mode       = <ImportMode.prepend: 'prepend'>
        module_name = 'tests.everflow.test_everflow_ipv6'
        names      = ['tests', 'everflow', 'test_everflow_ipv6']
        p          = PosixPath('/var/src/sonic-mgmt_vms28-dual-t0-8102_6551d780a6069aa8dc7c1c2e/tests/everflow/test_everflow_ipv6.py')
        path       = PosixPath('/var/src/sonic-mgmt_vms28-dual-t0-8102_6551d780a6069aa8dc7c1c2e/tests/everflow/test_everflow_ipv6.py')
        pkg_path   = PosixPath('/var/src/sonic-mgmt_vms28-dual-t0-8102_6551d780a6069aa8dc7c1c2e/tests')
        pkg_root   = PosixPath('/var/src/sonic-mgmt_vms28-dual-t0-8102_6551d780a6069aa8dc7c1c2e')
        root       = PosixPath('/var/src/sonic-mgmt_vms28-dual-t0-8102_6551d780a6069aa8dc7c1c2e/tests')
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
        level      = 0
        name       = 'tests.everflow.test_everflow_ipv6'
        package    = None
<frozen importlib._bootstrap>:1014: in _gcd_import
    ???
        level      = 0
        name       = 'tests.everflow.test_everflow_ipv6'
        package    = None
<frozen importlib._bootstrap>:991: in _find_and_load
    ???
        import_    = <function _gcd_import at 0x7f80222ba430>
        module     = <object object at 0x7f8022290060>
        name       = 'tests.everflow.test_everflow_ipv6'
<frozen importlib._bootstrap>:975: in _find_and_load_unlocked
    ???
        import_    = <function _gcd_import at 0x7f80222ba430>
        name       = 'tests.everflow.test_everflow_ipv6'
        parent     = 'tests.everflow'
        parent_module = <module 'tests.everflow' from '/var/src/sonic-mgmt_vms28-dual-t0-8102_6551d780a6069aa8dc7c1c2e/tests/everflow/__init__.py'>
        path       = ['/var/src/sonic-mgmt_vms28-dual-t0-8102_6551d780a6069aa8dc7c1c2e/tests/everflow']
        spec       = ModuleSpec(name='tests.everflow.test_everflow_ipv6', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7...7add90>, origin='/var/src/sonic-mgmt_vms28-dual-t0-8102_6551d780a6069aa8dc7c1c2e/tests/everflow/test_everflow_ipv6.py')
<frozen importlib._bootstrap>:671: in _load_unlocked
    ???
        module     = <module 'tests.everflow.test_everflow_ipv6' from '/var/src/sonic-mgmt_vms28-dual-t0-8102_6551d780a6069aa8dc7c1c2e/tests/everflow/test_everflow_ipv6.py'>
        spec       = ModuleSpec(name='tests.everflow.test_everflow_ipv6', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7...7add90>, origin='/var/src/sonic-mgmt_vms28-dual-t0-8102_6551d780a6069aa8dc7c1c2e/tests/everflow/test_everflow_ipv6.py')
<frozen importlib._bootstrap_external>:848: in exec_module
    ???
        code       = <code object <module> at 0x7f801c64eea0, file "/var/src/sonic-mgmt_vms28-dual-t0-8102_6551d780a6069aa8dc7c1c2e/tests/everflow/test_everflow_ipv6.py", line 1>
        module     = <module 'tests.everflow.test_everflow_ipv6' from '/var/src/sonic-mgmt_vms28-dual-t0-8102_6551d780a6069aa8dc7c1c2e/tests/everflow/test_everflow_ipv6.py'>
        self       = <_frozen_importlib_external.SourceFileLoader object at 0x7f801c7add90>
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
    ???
        args       = (<code object <module> at 0x7f801c64eea0, file "/var/src/sonic-mgmt_vms28-dual-t0-8102_6551d780a6069aa8dc7c1c2e/tests/...verflowTest'>, 'DOWN_STREAM': 'downstream', 'EVERFLOW_V6_RULES': 'ipv6_test_rules.yaml', 'UP_STREAM': 'upstream', ...})
        f          = <built-in function exec>
        kwds       = {}
everflow/test_everflow_ipv6.py:21: in <module>
    logger = logging.getLogger(__name__)
E   NameError: name 'logging' is not defined
        BaseEverflowTest = <class 'tests.everflow.everflow_test_utilities.BaseEverflowTest'>
        DOWN_STREAM = 'downstream'
        EVERFLOW_V6_RULES = 'ipv6_test_rules.yaml'
        UP_STREAM  = 'upstream'
        __builtins__ = <builtins>
        __cached__ = '/var/src/sonic-mgmt_vms28-dual-t0-8102_6551d780a6069aa8dc7c1c2e/tests/everflow/__pycache__/test_everflow_ipv6.cpython-38.pyc'
        __doc__    = 'Test cases to support the Everflow IPv6 Mirroring feature in SONiC.'
        __file__   = '/var/src/sonic-mgmt_vms28-dual-t0-8102_6551d780a6069aa8dc7c1c2e/tests/everflow/test_everflow_ipv6.py'
        __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f801c7add90>
        __name__   = 'tests.everflow.test_everflow_ipv6'
        __package__ = 'tests.everflow'
        __spec__   = ModuleSpec(name='tests.everflow.test_everflow_ipv6', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7...7add90>, origin='/var/src/sonic-mgmt_vms28-dual-t0-8102_6551d780a6069aa8dc7c1c2e/tests/everflow/test_everflow_ipv6.py')
        active_active_ports = <function active_active_ports at 0x7f801df279d0>
        config_active_active_dualtor_active_standby = <function config_active_active_dualtor_active_standby at 0x7f801d48c700>
        everflow_utils = <module 'tests.everflow.everflow_test_utilities' from '/var/src/sonic-mgmt_vms28-dual-t0-8102_6551d780a6069aa8dc7c1c2e/tests/everflow/everflow_test_utilities.py'>
        pytest     = <module 'pytest' from '/usr/local/lib/python3.8/dist-packages/pytest/__init__.py'>
        pytestmark = [MarkDecorator(mark=Mark(name='topology', args=('t0', 't1', 't2', 'm0'), kwargs={}))]
        setup_info = <function setup_info at 0x7f801c7c0c10>
        testutils  = <module 'ptf.testutils' from '/usr/local/lib/python3.8/dist-packages/ptf/testutils.py'>
        time       = <module 'time' (built-in)>
        toggle_all_simulator_ports_to_rand_selected_tor = <function toggle_all_simulator_ports_to_rand_selected_tor at 0x7f801d50d3a0>
        validate_active_active_dualtor_setup = <function validate_active_active_dualtor_setup at 0x7f801d48c5e0>
=============================== warnings summary ===============================

@mssonicbld
Copy link
Copy Markdown
Collaborator

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing tests/everflow/everflow_test_utilities.py
Fixing tests/everflow/test_everflow_ipv6.py

fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/everflow/test_everflow_ipv6.py:24:1: E302 expected 2 blank lines, found 1

flake8...............................................(no files to check)Skipped
check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

  1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
    the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
    docker container.
  2. Ensure that the pre-commit package is installed:
sudo pip install pre-commit
  1. Go to repository root folder
  2. Install the pre-commit hooks:
pre-commit install
  1. Use pre-commit to check staged file:
pre-commit
  1. Alternatively, you can check committed files using:
pre-commit run --from-ref <commit_id> --to-ref <commit_id>

@mssonicbld
Copy link
Copy Markdown
Collaborator

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing tests/everflow/everflow_test_utilities.py
Fixing tests/everflow/test_everflow_ipv6.py

fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Passed
flake8...............................................(no files to check)Skipped
check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

  1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
    the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
    docker container.
  2. Ensure that the pre-commit package is installed:
sudo pip install pre-commit
  1. Go to repository root folder
  2. Install the pre-commit hooks:
pre-commit install
  1. Use pre-commit to check staged file:
pre-commit
  1. Alternatively, you can check committed files using:
pre-commit run --from-ref <commit_id> --to-ref <commit_id>

@Aravind-Subbaroyan
Copy link
Copy Markdown

@XuChen-MSFT - The comment is addressed. Please help to review.

Copy link
Copy Markdown
Contributor

@XuChen-MSFT XuChen-MSFT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kevinskwang
Copy link
Copy Markdown
Contributor

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@mssonicbld
Copy link
Copy Markdown
Collaborator

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing tests/everflow/everflow_test_utilities.py
Fixing tests/everflow/test_everflow_ipv6.py

fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Passed
flake8...............................................(no files to check)Skipped
check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

  1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
    the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
    docker container.
  2. Ensure that the pre-commit package is installed:
sudo pip install pre-commit
  1. Go to repository root folder
  2. Install the pre-commit hooks:
pre-commit install
  1. Use pre-commit to check staged file:
pre-commit
  1. Alternatively, you can check committed files using:
pre-commit run --from-ref <commit_id> --to-ref <commit_id>

bpar9
bpar9 previously approved these changes Mar 7, 2024
@jamesan47
Copy link
Copy Markdown

Hi @kevinskwang , @XuChen-MSFT , the PR is approved. Could you help with the Azure.sonic-mgmt failures? Not certain if these are due to the code changes. Could you help with the merge?

@kevinskwang
Copy link
Copy Markdown
Contributor

@yanjundeng pls fix the pr test failure

@mssonicbld
Copy link
Copy Markdown
Collaborator

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing tests/everflow/test_everflow_ipv6.py

fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Passed
flake8...............................................(no files to check)Skipped
check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

  1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
    the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
    docker container.
  2. Ensure that the pre-commit package is installed:
sudo pip install pre-commit
  1. Go to repository root folder
  2. Install the pre-commit hooks:
pre-commit install
  1. Use pre-commit to check staged file:
pre-commit
  1. Alternatively, you can check committed files using:
pre-commit run --from-ref <commit_id> --to-ref <commit_id>

bpar9
bpar9 previously approved these changes Mar 15, 2024
@yanjundeng
Copy link
Copy Markdown
Contributor Author

@kevinskwang resolved. Pls help review and merge.

@kevinskwang kevinskwang merged commit 21401f2 into sonic-net:master Mar 20, 2024
mssonicbld pushed a commit to mssonicbld/sonic-mgmt that referenced this pull request Mar 20, 2024
* fix the everflow ipv6 failure on dualtor A-A

* change comment

* add logging

* add one line

* remove spaces

* remove white space
mssonicbld pushed a commit to mssonicbld/sonic-mgmt that referenced this pull request Mar 20, 2024
* fix the everflow ipv6 failure on dualtor A-A

* change comment

* add logging

* add one line

* remove spaces

* remove white space
@mssonicbld
Copy link
Copy Markdown
Collaborator

Cherry-pick PR to 202305: #12060

@mssonicbld
Copy link
Copy Markdown
Collaborator

Cherry-pick PR to 202311: #12061

mssonicbld pushed a commit that referenced this pull request Mar 20, 2024
* fix the everflow ipv6 failure on dualtor A-A

* change comment

* add logging

* add one line

* remove spaces

* remove white space
mssonicbld pushed a commit that referenced this pull request Mar 20, 2024
* fix the everflow ipv6 failure on dualtor A-A

* change comment

* add logging

* add one line

* remove spaces

* remove white space
bingwang-ms added a commit that referenced this pull request Mar 25, 2024
StormLiangMS pushed a commit that referenced this pull request Mar 26, 2024
mssonicbld pushed a commit to mssonicbld/sonic-mgmt that referenced this pull request Mar 26, 2024
mssonicbld pushed a commit to mssonicbld/sonic-mgmt that referenced this pull request Mar 26, 2024
mssonicbld pushed a commit that referenced this pull request Mar 26, 2024
StormLiangMS pushed a commit that referenced this pull request Mar 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants