Skip to content

NAT DVS tests fail consistently #1199

@daall

Description

@daall

It looks like some dependencies might be missing from the docker virtual switch:

Feb 20 22:16:00.273533 f2be27c28a0c INFO #supervisord: natmgrd sh: 1: /sbin/iptables: not found
Feb 20 22:18:56.293810 f2be27c28a0c INFO #supervisord: natmgrd sh: 1: conntrack: not found

Even after installing these dependencies, the tests are still failing and I'm seeing this error throughout:

Feb 20 22:28:02.795680 f2be27c28a0c NOTICE #natmgrd: :- sigterm_handler: Got SIGTERM
Feb 20 22:28:02.800283 f2be27c28a0c NOTICE #natmgrd: :- sigterm_handler: Sending notification to orchagent to cleanup NAT entries in REDIS/ASIC
Feb 20 22:25:23.602449 f2be27c28a0c INFO #supervisord: intfmgrd RTNETLINK answers: File exists
Feb 20 22:28:02.800302 f2be27c28a0c INFO #supervisord: natmgrd conntrack v1.4.4 (conntrack-tools): connection tracking table has been emptied.
Feb 20 22:28:02.800707 f2be27c28a0c NOTICE #orchagent: :- doTask: Received RedisDB and ASIC  cleanup notification on NAT docker stop
Feb 20 22:28:10.472606 f2be27c28a0c INFO #supervisord 2020-02-20 22:28:02,793 WARN received SIGTERM indicating exit request
Feb 20 22:28:10.472858 f2be27c28a0c INFO #supervisord 2020-02-20 22:28:02,794 INFO waiting for natmgrd, neighsyncd, vrfmgrd, natsyncd, zebra, teamsyncd, rsyslogd, teammgrd, portsyncd, orchagent, vxlanmgrd, intfmgrd, syncd, buffermgrd, portmgrd, staticd, vlanmgrd, redis-server, fpmsyncd, sflowmgrd, nbrmgrd to die
Feb 20 22:28:10.472866 f2be27c28a0c INFO #supervisord 2020-02-20 22:28:02,795 INFO stopped: natsyncd (terminated by SIGTERM)
Feb 20 22:28:10.473872 f2be27c28a0c INFO #supervisord 2020-02-20 22:28:05,807 INFO waiting for natmgrd, neighsyncd, vrfmgrd, zebra, teamsyncd, rsyslogd, teammgrd, portsyncd, orchagent, vxlanmgrd, intfmgrd, syncd, buffermgrd, portmgrd, staticd, vlanmgrd, redis-server, fpmsyncd, sflowmgrd, nbrmgrd to die
Feb 20 22:28:10.473895 f2be27c28a0c INFO #supervisord 2020-02-20 22:28:08,813 INFO waiting for natmgrd, neighsyncd, vrfmgrd, zebra, teamsyncd, rsyslogd, teammgrd, portsyncd, orchagent, vxlanmgrd, intfmgrd, syncd, buffermgrd, portmgrd, staticd, vlanmgrd, redis-server, fpmsyncd, sflowmgrd, nbrmgrd to die
Feb 20 22:28:17.420409 f2be27c28a0c INFO #liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="141" x-info="http://www.rsyslog.com"] start
Feb 20 22:28:17.420616 f2be27c28a0c ERR #liblogging-stdlog: Could not open output pipe '/dev/xconsole':: No such file or directory [v8.24.0 try http://www.rsyslog.com/e/2039 ]
Feb 20 22:28:17.420885 f2be27c28a0c INFO #supervisord: rsyslogd rsyslogd: Could not open output pipe '/dev/xconsole':: No such file or directory [v8.24.0 try http://www.rsyslog.com/e/2039 ]
Feb 20 22:28:17.420995 f2be27c28a0c WARNING #liblogging-stdlog: action 'action 11' suspended, next retry is Thu Feb 20 22:28:47 2020 [v8.24.0 try http://www.rsyslog.com/e/2007 ]
Feb 20 22:28:17.420456 f2be27c28a0c INFO #supervisord 2020-02-20 22:28:11,821 INFO waiting for natmgrd, neighsyncd, vrfmgrd, zebra, teamsyncd, rsyslogd, teammgrd, portsyncd, orchagent, vxlanmgrd, intfmgrd, syncd, buffermgrd, portmgrd, staticd, vlanmgrd, redis-server, fpmsyncd, sflowmgrd, nbrmgrd to die
Feb 20 22:28:17.420462 f2be27c28a0c INFO #supervisord 2020-02-20 22:28:12,823 WARN killing 'natmgrd' (676) with SIGKILL
Feb 20 22:28:17.420577 f2be27c28a0c INFO #supervisord 2020-02-20 22:28:12,825 INFO stopped: natmgrd (terminated by SIGKILL)

Tests were based off latest master build: https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-all/218/

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions