-
Notifications
You must be signed in to change notification settings - Fork 1.8k
[Soaking Test] In long term test with DMAC mobility, the memory runs out. #4357
Description
Description
Steps to reproduce the issue:
- Port 1 and Port 2 are members of Vlan2, and Port 1 and Port 2 are connected to the tester respectively.
- Send the same traffic with 2k DMAC to the two ports.
Describe the results you received:
Memory continues to grow until memory runs out.
Describe the results you expected:
Memory shouldn't run out.
Additional information you deem important (e.g. issue happens only occasionally):
**Output of `show version`:**
```
SONiC Software Version: SONiC.201811_rj.0-85995e0-20200307.004601
```
From a dump out of memory information, the cause is that the SYNCD put the message of MAC address changes in the message queue, and start a thread to retrieve messages from the queue to updates Redis database. While Redis is busy, which leads to slow update Redis database. A backlog of messages in the message queue causes memory usage increasement. Shutdown the interface or stop the stream, and memory usage will slowly decrease to normal level.
Solution: This problem can be solved by neutralizing MAC addition and aging messages.