Wait for ASIC_DB confirmation after disabling LAG members by yxieca · Pull Request #22849 · sonic-net/sonic-mgmt

yxieca · 2026-03-10T17:12:05Z

Description of PR

Summary:
After swssconfig disables LAG members, the command returns before orchagent/syncd finishes applying the configuration to ASIC_DB. This race causes intermittent test failures in test_lag_member_forwarding_packets because traffic verification runs before hardware actually disables the LAG members (~280ms gap observed).

Added a wait_until poll on ASIC_DB to confirm all LAG members have SAI_LAG_MEMBER_ATTR_EGRESS_DISABLE=true before proceeding with traffic tests.

Fixes #17095

Type of change

Back port request

Approach

What is the motivation for this PR?

swssconfig returns immediately after writing to APP_DB, but orchagent has not yet processed the request and applied it via syncd to ASIC_DB/hardware. The test sends traffic immediately after swssconfig returns, hitting a race window where LAG members are still active. This causes test_lag_member_forwarding_packets to fail intermittently on hardware platforms.

How did you do it?

Added a wait_until(10, 0.5, 0, ...) poll after swssconfig that checks ASIC_DB for SAI_LAG_MEMBER_ATTR_EGRESS_DISABLE=true on all LAG member entries. This is the definitive signal that orchagent→syncd has fully applied the disable to hardware. The 0.5s poll interval with 10s timeout is generous — real-world logs show the gap is ~280ms.

How did you verify/test it?

Code review of the race condition timing from issue logs
flake8 lint pass (max-line-length=120)
The fix uses the same wait_until pattern used extensively throughout sonic-mgmt

Any platform specific information?

The VS (virtual switch) platform stores EGRESS_DISABLE in ASIC_DB but does not enforce it in the kernel dataplane — the existing VS skip block runs after this new wait, so VS tests still skip traffic verification as before.

Supported testbed topology if it's a new test case?

N/A (bug fix)

Documentation

N/A

After swssconfig disables LAG members, the command returns before orchagent/syncd finishes applying the configuration to ASIC_DB. This race condition causes subsequent traffic verification to fail intermittently because packets are still forwarded through the LAG members that haven't been disabled yet in hardware. Add a wait_until poll on ASIC_DB to confirm all LAG members have SAI_LAG_MEMBER_ATTR_EGRESS_DISABLE set to true before proceeding with traffic tests. Fixes sonic-net#17095 Signed-off-by: Ying Xie <[email protected]>

yxieca · 2026-03-10T17:12:10Z

This PR was raised by an AI agent on behalf of Ying Xie.

mssonicbld · 2026-03-10T17:12:13Z

/azp run

azure-pipelines · 2026-03-10T17:12:30Z

Azure Pipelines successfully started running 1 pipeline(s).

VS SAI does not populate SAI_LAG_MEMBER_ATTR_EGRESS_DISABLE in ASIC_DB, causing the wait_until check to timeout. Move the VS early-return before the ASIC_DB poll since VS doesn't enforce LAG member disable in the dataplane anyway. Signed-off-by: Ying Xie <[email protected]>

mssonicbld · 2026-03-10T18:48:26Z

/azp run

azure-pipelines · 2026-03-10T18:48:42Z

Azure Pipelines successfully started running 1 pipeline(s).

yxieca · 2026-03-10T20:04:30Z

Local KVM Test Results (commit `0a01830`)

Test: pc/test_lag_member_forwarding.py on KVM T0 (vms-kvm-t0, vlab-01)

1 passed in 239.26s (0:03:59)

Fix in this commit: Moved VS early-return before the ASIC_DB wait — VS SAI does not populate SAI_LAG_MEMBER_ATTR_EGRESS_DISABLE in ASIC_DB, so the wait_until check was timing out. On real hardware, the ASIC_DB poll confirms config is applied before traffic verification.

The previous check collected all LAG members across all LAGs and verified EGRESS_DISABLE=true on all of them. This would always fail when other LAGs exist that were not disabled. Fix: look up the SAI OID for the specific PortChannel under test via COUNTERS_LAG_NAME_MAP, then filter LAG_MEMBER entries to only those belonging to that LAG. Also move the ASIC_DB check before the VS skip so it runs on VS too (VS SAI does populate ASIC_DB, it just doesn't enforce disable in the dataplane). Signed-off-by: Ying Xie <[email protected]>

mssonicbld · 2026-03-12T02:00:58Z

/azp run

azure-pipelines · 2026-03-12T02:01:14Z

Azure Pipelines successfully started running 1 pipeline(s).

The fallback checked all LAG members across all LAGs, which is the bug we are fixing. Fail explicitly instead. Signed-off-by: Ying Xie <[email protected]>

mssonicbld · 2026-03-12T02:05:33Z

/azp run

azure-pipelines · 2026-03-12T02:05:49Z

Azure Pipelines successfully started running 1 pipeline(s).

Instead of blindly picking the first two PortChannels, iterate and find a pair where all BGP neighbors (v4 and v6) are established. Skip the test if two suitable PortChannels cannot be found. This avoids failures on testbeds where some PortChannels have permanently idle IPv6 sessions. Signed-off-by: Ying Xie <[email protected]>

mssonicbld · 2026-03-12T02:19:31Z

/azp run

azure-pipelines · 2026-03-12T02:19:57Z

Azure Pipelines successfully started running 1 pipeline(s).

COUNTERS_LAG_NAME_MAP may not have entries for all PortChannels on converged-peer testbeds. Instead, look up the SAI OIDs of the member ports via COUNTERS_PORT_NAME_MAP and match them against SAI_LAG_MEMBER_ATTR_PORT_ID in ASIC_DB. This works regardless of whether the LAG itself has a counter entry. Signed-off-by: Ying Xie <[email protected]>

mssonicbld · 2026-03-12T02:30:25Z

/azp run

azure-pipelines · 2026-03-12T02:30:38Z

Azure Pipelines successfully started running 1 pipeline(s).

yxieca-admin · 2026-03-18T04:13:07Z

/azp run

azure-pipelines · 2026-03-18T04:13:22Z

Azure Pipelines successfully started running 1 pipeline(s).

yxieca · 2026-04-01T15:22:03Z

High-Level Summary of Changes

Problem (issue #17095):
test_lag_member_forwarding_packets fails intermittently on hardware. Root cause: swssconfig returns immediately after writing to APP_DB, but orchagent/syncd has not yet disabled the LAG members in ASIC_DB (~280ms gap). The test sends traffic during that race window, so disabled LAG members are still forwarding.

Evolution across 6 commits:

Initial fix — Added a wait_until poll on ASIC_DB checking SAI_LAG_MEMBER_ATTR_EGRESS_DISABLE=true before sending traffic
Move VS skip earlier — VS SAI does not populate the EGRESS_DISABLE attribute, so moved the VS platform skip before the ASIC_DB wait (otherwise it would timeout on VS)
Scope the check to the tested LAG only — Original check was looking at ALL LAG members in ASIC_DB. Fixed to only verify members belonging to the specific PortChannel under test
Remove broken fallback — If the LAG OID was not found, there was a silent fallback that would skip the check. Changed to fail explicitly instead of hiding the problem
Select PortChannels with established BGP — Pick test candidates that actually have all BGP neighbors up, avoiding flaky selection of half-up LAGs
Use port OIDs for member lookup — Instead of querying by LAG OID (which required an extra lookup step), directly match ASIC_DB LAG member entries using the port OIDs. Simpler and more reliable.

TL;DR: Started with a straightforward ASIC_DB wait, then iteratively tightened the scope (right LAG, right members), hardened error handling (no silent fallback), improved test robustness (only pick healthy LAGs), and simplified the ASIC_DB query path.

nhe-NV · 2026-04-02T01:29:10Z

@yxieca the issue also happen on the 202511 branch, should it also include in the 202511 branch, right?

lolyu

✅ Approved

ASIC_DB wait after swssconfig is the right fix for the race condition. PortChannel selection with BGP state check is a good improvement. LGTM 🚀

github-actions bot requested review from ZhaohuiS, saiarcot895 and yejianquan March 10, 2026 17:12

yxieca-admin requested a review from nhe-NV March 10, 2026 17:12

yxieca-admin mentioned this pull request Mar 10, 2026

Xfail test_lag_member_forwarding.py due to github issue https://githu… #22091

Merged

12 tasks

weiguo-nvidia mentioned this pull request Mar 12, 2026

[Test Gap][LAG][t0-64] Applying lag members configuration is not awaited #17095

Closed

Remove broken fallback: fail if LAG OID not found

5b6be65

The fallback checked all LAG members across all LAGs, which is the bug we are fixing. Fail explicitly instead. Signed-off-by: Ying Xie <[email protected]>

lolyu self-requested a review April 2, 2026 01:16

lolyu approved these changes Apr 3, 2026

View reviewed changes

yxieca merged commit c216233 into sonic-net:master Apr 3, 2026
16 checks passed

yxieca deleted the fix/lag-member-await-config branch April 3, 2026 15:23

Conversation

yxieca commented Mar 10, 2026

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

yxieca commented Mar 10, 2026

Uh oh!

mssonicbld commented Mar 10, 2026

Uh oh!

azure-pipelines bot commented Mar 10, 2026

Uh oh!

mssonicbld commented Mar 10, 2026

Uh oh!

azure-pipelines bot commented Mar 10, 2026

Uh oh!

yxieca commented Mar 10, 2026

Local KVM Test Results (commit 0a01830)

Uh oh!

mssonicbld commented Mar 12, 2026

Uh oh!

azure-pipelines bot commented Mar 12, 2026

Uh oh!

mssonicbld commented Mar 12, 2026

Uh oh!

azure-pipelines bot commented Mar 12, 2026

Uh oh!

mssonicbld commented Mar 12, 2026

Uh oh!

azure-pipelines bot commented Mar 12, 2026

Uh oh!

mssonicbld commented Mar 12, 2026

Uh oh!

azure-pipelines bot commented Mar 12, 2026

Uh oh!

yxieca-admin commented Mar 18, 2026

Uh oh!

azure-pipelines bot commented Mar 18, 2026

Uh oh!

yxieca commented Apr 1, 2026

High-Level Summary of Changes

Uh oh!

nhe-NV commented Apr 2, 2026

Uh oh!

lolyu left a comment

Choose a reason for hiding this comment

✅ Approved

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Local KVM Test Results (commit `0a01830`)