Update syncd stop script to collect saisdkdump during orch abort by vivekrnv · Pull Request #38 · vivekrnv/sonic-buildimage

vivekrnv · 2022-10-28T21:16:52Z

Signed-off-by: Vivek Reddy Karri [email protected]

Why I did it

Update the syncd stop flow to collect saisdkdump and save it to /var/log/orch_abrt_status/ location during a SAI programming failure by orchagent.
1. Limit the number of such archived to 10
2. Create a tmp file as a sync mechanism with auto-techsupport process
Update the swss stop flow to remove PortInitDone. This is done so as to provide other daemons a reliable way of checking if switch_init is successful after services are deemed started. (Eg: Before running saisdkdump, switch should be initialized. techsupport will use this info to determine that)

How I did it

How to verify it


root@r-bulldog-03:/home/admin# sonic-db-cli STATE_DB SET ORCH_ABRT_STATUS 1

root@r-bulldog-03:/home/admin# systemctl restart swss

Oct 28 21:10:52.892667 r-bulldog-03 INFO syncd.sh[26913]: sai_sdk_dump_1666991448.tar.gz collected before taking stopping syncd

root@r-bulldog-03:/home/admin# tar -tvf /var/log/orch_abrt_status/sai_sdk_dump_1666991448.tar.gz
drwxr-xr-x root/root         0 2022-10-28 21:10 ./
-rw-r--r-- root/root   4883498 2022-10-28 21:10 ./sdkdump_ext_cr_001-28_10_2022-21_10_50-980950.udmp
-rw-r--r-- root/root   3964382 2022-10-28 21:10 ./sai_sdk_dump_10_28_2022_09_10_PM
-rw-r--r-- root/root       288 2022-10-28 21:10 ./sdkdump_ext_meta_001-28_10_2022-21_10_50-407583
-rw-r--r-- root/root       288 2022-10-28 21:10 ./sdkdump_ext_meta_001-28_10_2022-21_10_50-980950
-rw-r--r-- root/root   2656360 2022-10-28 21:10 ./sai_sdk_dump_10_28_2022_09_10_PM.json
-rw-r--r-- root/root   4883498 2022-10-28 21:10 ./sdkdump_ext_cr_001-28_10_2022-21_10_50-407583.udmp
-rw-r--r-- root/root   4883498 2022-10-28 21:10 ./sdkdump_ext_cr_001-28_10_2022-21_10_51-573167.udmp
-rw-r--r-- root/root       288 2022-10-28 21:10 ./sdkdump_ext_meta_001-28_10_2022-21_10_51-573167

Which release branch to backport (provide reason below if selected)

Description for the changelog

Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Signed-off-by: Vivek Reddy Karri <[email protected]>

dgsudharsan · 2022-11-24T02:22:57Z

    /usr/bin/docker exec syncd$DEV rm -f /tmp/${sai_dump_filename_epoch}.tar
    /usr/bin/docker exec syncd$DEV mkdir -p ${TMP_DMP_DIR}
-    /usr/bin/docker exec syncd$DEV saisdkdump -f ${TMP_DMP_DIR}/${sai_dump_filename} > /dev/null
+    timeout 1m bash -c "/usr/bin/docker exec syncd$DEV saisdkdump -f ${TMP_DMP_DIR}/${sai_dump_filename} > /dev/null"


1 minute seems to be a huge timeout. Can you check how long saisdkdump takes in each platform and come up with 2x of max time taken. Ideally saisdkdump should finish within 5-6 seconds. If that's the scenario we can add timeout of 15 seconds

This is worst case scenario right. And we can't predict how long will the saisdkdump of other vendors would take. Anyways, i'll set it to 30s. Seems reasonable?

Updated the timeout to 30s

…to orch_abort

Signed-off-by: Vivek Reddy Karri <[email protected]>

…lly (sonic-net#639) #### Why I did it src/sonic-swss ``` * 4baf54f - (HEAD -> 202412, origin/202412) SRv6: add dscp_mode configuration for MySID entry (#38) (6 hours ago) [mssonicbld] * ff491ba - [SRv6] Add support for SRv6 VPN (#37) (9 hours ago) [mssonicbld] ``` #### How I did it #### How to verify it #### Description for the changelog

… automatically (sonic-net#702) #### Why I did it src/sonic-platform-common ``` * c8eac22 - (HEAD -> 202412, origin/202412) [code sync] Merge code from sonic-net/sonic-platform-common:202411 to 202412 (#38) (21 hours ago) [mssonicbld] ``` #### How I did it #### How to verify it #### Description for the changelog

…tomatically (sonic-net#899) #### Why I did it src/sonic-linux-kernel ``` * b1aeb41 - (HEAD -> 202412, origin/HEAD, origin/202412) [code sync] Merge code from sonic-net/sonic-linux-kernel:202411 to 202412 (#38) (20 hours ago) [mssonicbld] ``` #### How I did it #### How to verify it #### Description for the changelog

…omatically (sonic-net#928) #### Why I did it src/sonic-swss-common ``` * 0087183 - (HEAD -> 202412, origin/HEAD, origin/202412) Merge pull request #38 from mssonicbld/sonicbld/202412-merge (35 hours ago) [mssonicbld] * e32b71e - Merge branch '202411' of https://github.com/sonic-net/sonic-swss-common into 202412 (2 days ago) [Sonic Automation] * 3bc4141 - [FC] remove FLEX_COUNTER_DELAY_STATUS_FIELD (sonic-net#982) (3 weeks ago) [mssonicbld] ``` #### How I did it #### How to verify it #### Description for the changelog

…tically (sonic-net#937) #### Why I did it src/sonic-sairedis ``` * 8930167 - (HEAD -> 202412, origin/HEAD, origin/202412) [FC] Fix the update failure in switch debug counters (#38) (3 hours ago) [mssonicbld] ``` #### How I did it #### How to verify it #### Description for the changelog

…sonic-net#22193) #### Why I did it src/dhcpmon ``` * 749c7e5 - (HEAD -> master, origin/master, origin/HEAD) Update DB separator for per-interface counter (#38) (23 hours ago) [Yaqiang Zhu] ``` #### How I did it #### How to verify it #### Description for the changelog

…tically (sonic-net#22803) #### Why I did it src/sonic-dash-api ``` * 573485d - (HEAD -> master, origin/master, origin/HEAD) Update pipeline to use Bookworm and Ubuntu 24.04 (#38) (6 days ago) [Saikrishna Arcot] ``` #### How I did it #### How to verify it #### Description for the changelog

update syncd stop script to collect saisdkdump during orch abort

c443ecf

Signed-off-by: Vivek Reddy Karri <[email protected]>

vivekrnv marked this pull request as ready for review October 28, 2022 21:21

vivekrnv added 7 commits November 14, 2022 23:07

scripts updated to new design

3075768

Signed-off-by: Vivek Reddy Karri <[email protected]>

Minor updates

60b04aa

Signed-off-by: Vivek Reddy Karri <[email protected]>

Minor update

67cea83

Signed-off-by: Vivek Reddy Karri <[email protected]>

create the tmp file for all plaforms

b0258b4

Signed-off-by: Vivek Reddy Karri <[email protected]>

Log updated

046f171

Signed-off-by: Vivek Reddy Karri <[email protected]>

removed sleep

dcf3298

Signed-off-by: Vivek Reddy Karri <[email protected]>

Move cleaning after the copy

bbf0f56

Signed-off-by: Vivek Reddy Karri <[email protected]>

dgsudharsan requested changes Nov 22, 2022

View reviewed changes

Comment thread files/scripts/syncd.sh

dgsudharsan requested changes Nov 22, 2022

View reviewed changes

Comment thread files/scripts/syncd.sh

Add 1m timeout on invoking saisdkdump

e0eaf6c

Signed-off-by: Vivek Reddy Karri <[email protected]>

vivekrnv requested a review from dgsudharsan November 23, 2022 22:20

dgsudharsan requested changes Nov 24, 2022

View reviewed changes

vivekrnv added 2 commits November 29, 2022 01:06

Merge branch 'master' of https://github.com/Azure/sonic-buildimage in…

4c36ba5

…to orch_abort

reduced timeout to 30s

05431b7

Signed-off-by: Vivek Reddy Karri <[email protected]>

vivekrnv requested a review from dgsudharsan December 1, 2022 02:56

Limit Number of dumps to 3

0a5733d

Signed-off-by: Vivek Reddy Karri <[email protected]>

dgsudharsan approved these changes Dec 2, 2022

View reviewed changes

vivekrnv closed this Dec 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update syncd stop script to collect saisdkdump during orch abort#38

Update syncd stop script to collect saisdkdump during orch abort#38
vivekrnv wants to merge 12 commits intomasterfrom
orch_abort

vivekrnv commented Oct 28, 2022 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

dgsudharsan Nov 24, 2022

Uh oh!

vivekrnv Nov 24, 2022

Uh oh!

vivekrnv Dec 1, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

vivekrnv commented Oct 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Uh oh!

Uh oh!

Uh oh!

dgsudharsan Nov 24, 2022

Choose a reason for hiding this comment

Uh oh!

vivekrnv Nov 24, 2022

Choose a reason for hiding this comment

Uh oh!

vivekrnv Dec 1, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vivekrnv commented Oct 28, 2022 •

edited

Loading