Fix issue: pmon services's restart count is not cleared during config reload by stephenxs · Pull Request #4314 · sonic-net/sonic-utilities

stephenxs · 2026-02-27T06:43:06Z

What I did

Currently, when "config reload" is executed, services' restart count are cleared to avoid reaching restart limit. This is done by listing all services using command systemctl list-dependencies --plain .target. However, this doesn't include pmon service, neither all other services that don't have WantedBy=sonic.target, which means pmon's start count is not cleared.

How I did it

Sometimes pmon fails to restart due to reaching start limit (3 times in 1200 seconds). The pmon service can be started by featured, syncd during config reload. Before multi-ASIC, pmon depends on syncd. The dependency is removed after multi-ASIC, which means pmon can restart immediately triggered by sonic.target which is once more restarting. As a result the pmon service is more likely to reach the restart limit.

How to verify it

Clear restart count also for services that have reverse dependency on sonic.target.

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

… reload What I did Currently, when "config reload" is executed, services' restart count are cleared to avoid reaching restart limit. This is done by listing all services using command systemctl list-dependencies --plain .target. However, this doesn't include pmon service, neither all other services that don't have WantedBy=sonic.target, which means pmon's start count is not cleared. After multi-ASIC PRs are merged, there is a high probability that pmon fails to restart due to reaching start limit (3 times in 1200 seconds). The pmon service can be started by featured, syncd during config reload. Before multi-ASIC, pmon depends on syncd. The dependency is removed after multi-ASIC, which means pmon can restart immediately triggered by sonic.target which is once more restarting. As a result the pmon service is more likely to reach the restart limit. How I did it Clear restart count also for services that have reverse dependency on sonic.target. How to verify it Previous command output (if the output of a command-line utility has changed) New command output (if the output of a command-line utility has changed) Signed-off-by: Stephen Sun <[email protected]>

mssonicbld · 2026-02-27T06:43:13Z

/azp run

azure-pipelines · 2026-02-27T06:43:24Z

Azure Pipelines successfully started running 1 pipeline(s).

stephenxs · 2026-03-03T02:20:55Z

/azpw run

mssonicbld · 2026-03-03T02:20:57Z

/AzurePipelines run

azure-pipelines · 2026-03-03T02:21:07Z

Azure Pipelines successfully started running 1 pipeline(s).

StormLiangMS · 2026-03-06T13:44:32Z

Cross-reference: Complementary fix in sonic-buildimage

Related PR: sonic-net/sonic-buildimage#25932 — adds the missing [Install] WantedBy=sonic.target section to 9 container service templates (pmon, lldp, gnmi, snmp, telemetry, otel, sflow, bmp, mgmt-framework).

Root cause analysis

After deeper investigation, we found the start-limit-hit issue has three contributing factors:

Missing [Install] section (fixed by sonic-buildimage#25932): After the systemd-sonic-generator rework (PR Trixie base image upgrade sonic-buildimage#23340), the generator only creates sonic.target.wants/ symlinks based on [Install] targets. These 9 services have BindsTo=sonic.target but no [Install], so they are no longer listed as sonic.target dependencies.
_reset_failed_services() misses these services (fixed by this PR): It iterates systemctl list-dependencies --plain sonic.target which no longer includes these services. Rate limit counters are never reset between config reloads.
featured daemon issues redundant systemctl start calls: On 202511, enable_feature() checks if unit_file_state == 'enabled': continue — but UnitFileState is static (no [Install]), so the check fails. Featured runs systemctl enable (fails silently due to the raise_exception=False Trixie change) then proceeds to systemctl start — adding an extra start attempt on every config reload. This is why pmon exceeds StartLimitBurst=3 after multiple reloads.

Why 202505 doesn't have this issue

On 202505, featured's enable_feature() uses raise_exception=True for the enable command → systemctl enable fails for static services → exception caught → systemctl start is never reached. On 202511, raise_exception=False causes the failure to be silently ignored → systemctl start proceeds → extra start attempts accumulate.

How the two PRs complement each other

Fix	What it addresses
sonic-buildimage#25932	Adds `[Install] WantedBy=sonic.target` → services become proper `sonic.target` dependencies, `UnitFileState` becomes `enabled`, featured skips already-enabled services
This PR (sonic-utilities#4314)	Adds `--reverse` to `_get_sonic_services()` → `_reset_failed_services` covers `BindsTo` services even without `[Install]`

Both PRs are valid fixes. sonic-buildimage#25932 addresses all 3 factors, while this PR provides defense-in-depth for _reset_failed_services. Ideally both should merge.

Tracked by: sonic-net/sonic-buildimage#25931

cc @stephenxs @saiarcot895

StormLiangMS

LGTM

StormLiangMS · 2026-03-06T13:55:42Z

hi @qiluo-msft could you help to review and merge?

Skip test_load_minigraph_with_golden_config when issue #25931 is open. This test performs 4 consecutive config reloads which causes pmon to hit start-limit-hit due to missing sonic.target.wants/ symlinks after the systemd-sonic-generator rework (sonic-net/sonic-buildimage#23340). The test leaves pmon in a bad state (start-limit-hit), which can affect subsequent tests in the nightly run. Fix PRs: - sonic-net/sonic-buildimage#25932 (add [Install] to service templates) - sonic-net/sonic-utilities#4314 (fix _reset_failed_services) The skip will auto-resolve when sonic-net/sonic-buildimage#25931 is closed. Signed-off-by: Storm Liang <[email protected]>

saiarcot895 · 2026-03-06T15:51:08Z

config/main.py

+def _get_sonic_services(reverse=False):
    cmd = ['systemctl', 'list-dependencies', '--plain', 'sonic.target']
+    if reverse:
+        cmd.append('--reverse')


I would suggest also adding --type=service. when adding --reverse. Otherwise, this will also include multi-user.target and graphical.target, and while there probably wouldn't be any negative effects from resetting the failed state of the units, it might be better to restrict it to SONiC services.

Skip test_load_minigraph_with_golden_config when issue #25931 is open. This test performs 4 consecutive config reloads which causes pmon to hit start-limit-hit due to missing sonic.target.wants/ symlinks after the systemd-sonic-generator rework (sonic-net/sonic-buildimage#23340). The test leaves pmon in a bad state (start-limit-hit), which can affect subsequent tests in the nightly run. Fix PRs: - sonic-net/sonic-buildimage#25932 (add [Install] to service templates) - sonic-net/sonic-utilities#4314 (fix _reset_failed_services) The skip will auto-resolve when sonic-net/sonic-buildimage#25931 is closed. Signed-off-by: Storm Liang <[email protected]>

StormLiangMS · 2026-03-07T12:27:53Z

hi @stephenxs could you check comments from Saikrishna? And file a separate PR to 202511 branch? Thanks for the fix. Fixes: sonic-net/sonic-buildimage#25931

…c-net#22775) Skip test_load_minigraph_with_golden_config when issue #25931 is open. This test performs 4 consecutive config reloads which causes pmon to hit start-limit-hit due to missing sonic.target.wants/ symlinks after the systemd-sonic-generator rework (sonic-net/sonic-buildimage#23340). The test leaves pmon in a bad state (start-limit-hit), which can affect subsequent tests in the nightly run. Fix PRs: - sonic-net/sonic-buildimage#25932 (add [Install] to service templates) - sonic-net/sonic-utilities#4314 (fix _reset_failed_services) The skip will auto-resolve when sonic-net/sonic-buildimage#25931 is closed. Signed-off-by: Storm Liang <[email protected]> Signed-off-by: mssonicbld <[email protected]>

Signed-off-by: Stephen Sun <[email protected]>

mssonicbld · 2026-03-09T05:17:50Z

/azp run

azure-pipelines · 2026-03-09T05:18:00Z

Azure Pipelines successfully started running 1 pipeline(s).

liat-grozovik · 2026-03-09T07:38:59Z

As feedback handled and PR now has the additional request, i assume we can merge it.
@stephenxs can you please provide the PR for 202511 and relate to this one?

… reload (sonic-net#4314) - What I did Currently, when "config reload" is executed, services' restart count are cleared to avoid reaching restart limit. This is done by listing all services using command systemctl list-dependencies --plain .target. However, this doesn't include pmon service, neither all other services that don't have WantedBy=sonic.target, which means pmon's start count is not cleared. - How I did it Sometimes pmon fails to restart due to reaching start limit (3 times in 1200 seconds). The pmon service can be started by featured, syncd during config reload. Before multi-ASIC, pmon depends on syncd. The dependency is removed after multi-ASIC, which means pmon can restart immediately triggered by sonic.target which is once more restarting. As a result the pmon service is more likely to reach the restart limit. - How to verify it Clear restart count also for services that have reverse dependency on sonic.target. Signed-off-by: Stephen Sun <[email protected]>

@stephenxs

…ng config reload (sonic-net#4314) This is the same fix as PR sonic-net#4336 by @stephenxs, with corrected test assertion counts. During config reload, _reset_failed_services() now also resets services that have a reverse dependency (BindsTo) on sonic.target, such as pmon. This ensures pmon's restart count is properly cleared and prevents start-limit-hit failures. Changes: - _get_sonic_services() accepts reverse=False parameter to query reverse deps - _reset_failed_services() unions forward and reverse dependencies - Test mock updates with correct call_count assertions (19/15/15 vs original 16/12/12) The original PR sonic-net#4336 had an off-by-one error in test counts because the first mock was updated to return featured.timer as an additional forward dep, adding one more reset-failed call that wasn't accounted for. Signed-off-by: Storm Liang <[email protected]> Co-authored-by: Copilot <[email protected]>

Cherry-pick of PR sonic-net#22775 to 202511, rebased on latest 202511 to include the duplicate key fix from PR sonic-net#22796. The test performs multiple config reloads that cause pmon start-limit-hit due to missing sonic.target symlinks after systemd-sonic-generator rework. Fix PRs: sonic-net/sonic-buildimage#25932, sonic-net/sonic-utilities#4314 Tracking issue: sonic-net/sonic-buildimage#25931 Signed-off-by: Storm Liang <[email protected]> Co-authored-by: Copilot <[email protected]>

…ng config reload (#4314) (#4336) * Fix issue: pmon services's restart count is not cleared during config reload (#4314) - What I did Currently, when "config reload" is executed, services' restart count are cleared to avoid reaching restart limit. This is done by listing all services using command systemctl list-dependencies --plain .target. However, this doesn't include pmon service, neither all other services that don't have WantedBy=sonic.target, which means pmon's start count is not cleared. - How I did it Sometimes pmon fails to restart due to reaching start limit (3 times in 1200 seconds). The pmon service can be started by featured, syncd during config reload. Before multi-ASIC, pmon depends on syncd. The dependency is removed after multi-ASIC, which means pmon can restart immediately triggered by sonic.target which is once more restarting. As a result the pmon service is more likely to reach the restart limit. - How to verify it Clear restart count also for services that have reverse dependency on sonic.target. Signed-off-by: Stephen Sun <[email protected]> * Fix UT failure Signed-off-by: Stephen Sun <[email protected]> --------- Signed-off-by: Stephen Sun <[email protected]>

…fig (#22830) Cherry-pick of PR #22775 to 202511, rebased on latest 202511 to include the duplicate key fix from PR #22796. The test performs multiple config reloads that cause pmon start-limit-hit due to missing sonic.target symlinks after systemd-sonic-generator rework. Fix PRs: sonic-net/sonic-buildimage#25932, sonic-net/sonic-utilities#4314 Tracking issue: sonic-net/sonic-buildimage#25931 Co-authored-by: Copilot <[email protected]>

…c-net#22775) Skip test_load_minigraph_with_golden_config when issue #25931 is open. This test performs 4 consecutive config reloads which causes pmon to hit start-limit-hit due to missing sonic.target.wants/ symlinks after the systemd-sonic-generator rework (sonic-net/sonic-buildimage#23340). The test leaves pmon in a bad state (start-limit-hit), which can affect subsequent tests in the nightly run. Fix PRs: - sonic-net/sonic-buildimage#25932 (add [Install] to service templates) - sonic-net/sonic-utilities#4314 (fix _reset_failed_services) The skip will auto-resolve when sonic-net/sonic-buildimage#25931 is closed. Signed-off-by: Storm Liang <[email protected]>

…c-net#22775) Skip test_load_minigraph_with_golden_config when issue #25931 is open. This test performs 4 consecutive config reloads which causes pmon to hit start-limit-hit due to missing sonic.target.wants/ symlinks after the systemd-sonic-generator rework (sonic-net/sonic-buildimage#23340). The test leaves pmon in a bad state (start-limit-hit), which can affect subsequent tests in the nightly run. Fix PRs: - sonic-net/sonic-buildimage#25932 (add [Install] to service templates) - sonic-net/sonic-utilities#4314 (fix _reset_failed_services) The skip will auto-resolve when sonic-net/sonic-buildimage#25931 is closed. Signed-off-by: Storm Liang <[email protected]> Signed-off-by: Mihut Aronovici <[email protected]>

…g config reload (sonic-net#4314)" This reverts commit 4d0cc93.

…c-net#22775) Skip test_load_minigraph_with_golden_config when issue #25931 is open. This test performs 4 consecutive config reloads which causes pmon to hit start-limit-hit due to missing sonic.target.wants/ symlinks after the systemd-sonic-generator rework (sonic-net/sonic-buildimage#23340). The test leaves pmon in a bad state (start-limit-hit), which can affect subsequent tests in the nightly run. Fix PRs: - sonic-net/sonic-buildimage#25932 (add [Install] to service templates) - sonic-net/sonic-utilities#4314 (fix _reset_failed_services) The skip will auto-resolve when sonic-net/sonic-buildimage#25931 is closed. Signed-off-by: Storm Liang <[email protected]> Signed-off-by: selldinesh <[email protected]>

…c-net#22775) Skip test_load_minigraph_with_golden_config when issue #25931 is open. This test performs 4 consecutive config reloads which causes pmon to hit start-limit-hit due to missing sonic.target.wants/ symlinks after the systemd-sonic-generator rework (sonic-net/sonic-buildimage#23340). The test leaves pmon in a bad state (start-limit-hit), which can affect subsequent tests in the nightly run. Fix PRs: - sonic-net/sonic-buildimage#25932 (add [Install] to service templates) - sonic-net/sonic-utilities#4314 (fix _reset_failed_services) The skip will auto-resolve when sonic-net/sonic-buildimage#25931 is closed. Signed-off-by: Storm Liang <[email protected]> Signed-off-by: Abhishek <[email protected]>

saiarcot895 · 2026-03-18T15:39:57Z

@stephenxs Can you cherry-pick this PR for 202511 branch? It seems there are cherry-pick conflicts.

…c-net#22775) Skip test_load_minigraph_with_golden_config when issue #25931 is open. This test performs 4 consecutive config reloads which causes pmon to hit start-limit-hit due to missing sonic.target.wants/ symlinks after the systemd-sonic-generator rework (sonic-net/sonic-buildimage#23340). The test leaves pmon in a bad state (start-limit-hit), which can affect subsequent tests in the nightly run. Fix PRs: - sonic-net/sonic-buildimage#25932 (add [Install] to service templates) - sonic-net/sonic-utilities#4314 (fix _reset_failed_services) The skip will auto-resolve when sonic-net/sonic-buildimage#25931 is closed. Signed-off-by: Storm Liang <[email protected]> Signed-off-by: Venkata Gouri Rajesh Etla <[email protected]>

…c-net#22775) Skip test_load_minigraph_with_golden_config when issue #25931 is open. This test performs 4 consecutive config reloads which causes pmon to hit start-limit-hit due to missing sonic.target.wants/ symlinks after the systemd-sonic-generator rework (sonic-net/sonic-buildimage#23340). The test leaves pmon in a bad state (start-limit-hit), which can affect subsequent tests in the nightly run. Fix PRs: - sonic-net/sonic-buildimage#25932 (add [Install] to service templates) - sonic-net/sonic-utilities#4314 (fix _reset_failed_services) The skip will auto-resolve when sonic-net/sonic-buildimage#25931 is closed. Signed-off-by: Storm Liang <[email protected]>

…c-net#22775) Skip test_load_minigraph_with_golden_config when issue #25931 is open. This test performs 4 consecutive config reloads which causes pmon to hit start-limit-hit due to missing sonic.target.wants/ symlinks after the systemd-sonic-generator rework (sonic-net/sonic-buildimage#23340). The test leaves pmon in a bad state (start-limit-hit), which can affect subsequent tests in the nightly run. Fix PRs: - sonic-net/sonic-buildimage#25932 (add [Install] to service templates) - sonic-net/sonic-utilities#4314 (fix _reset_failed_services) The skip will auto-resolve when sonic-net/sonic-buildimage#25931 is closed. Signed-off-by: Storm Liang <[email protected]> Signed-off-by: selldinesh <[email protected]>

…c-net#22775) Skip test_load_minigraph_with_golden_config when issue #25931 is open. This test performs 4 consecutive config reloads which causes pmon to hit start-limit-hit due to missing sonic.target.wants/ symlinks after the systemd-sonic-generator rework (sonic-net/sonic-buildimage#23340). The test leaves pmon in a bad state (start-limit-hit), which can affect subsequent tests in the nightly run. Fix PRs: - sonic-net/sonic-buildimage#25932 (add [Install] to service templates) - sonic-net/sonic-utilities#4314 (fix _reset_failed_services) The skip will auto-resolve when sonic-net/sonic-buildimage#25931 is closed. Signed-off-by: Storm Liang <[email protected]>

stephenxs marked this pull request as draft February 27, 2026 06:44

stephenxs marked this pull request as ready for review March 2, 2026 07:16

keboliu approved these changes Mar 3, 2026

View reviewed changes

StormLiangMS mentioned this pull request Mar 6, 2026

Add missing [Install] section to container service templates sonic-net/sonic-buildimage#25932

Closed

2 tasks

StormLiangMS previously approved these changes Mar 6, 2026

View reviewed changes

StormLiangMS added the Request for 202511 Branch label Mar 6, 2026

mssonicbld added the Cherry Pick Conflict_202511 label Mar 6, 2026

StormLiangMS requested a review from qiluo-msft March 6, 2026 13:55

StormLiangMS mentioned this pull request Mar 6, 2026

Add conditional skip for test_load_minigraph_with_golden_config sonic-net/sonic-mgmt#22775

Merged

6 tasks

saiarcot895 reviewed Mar 6, 2026

View reviewed changes

StormLiangMS mentioned this pull request Mar 7, 2026

Fix featured to skip enable/start for static services sonic-net/sonic-host-services#353

Closed

mssonicbld mentioned this pull request Mar 7, 2026

[action] [PR:22775] Add conditional skip for test_load_minigraph_with_golden_config sonic-net/sonic-mgmt#22793

Merged

6 tasks

Update for review comments

172f108

Signed-off-by: Stephen Sun <[email protected]>

stephenxs dismissed StormLiangMS’s stale review via 172f108 March 9, 2026 05:17

liat-grozovik approved these changes Mar 9, 2026

View reviewed changes

liat-grozovik merged commit 4d0cc93 into sonic-net:master Mar 9, 2026
9 checks passed

stephenxs deleted the fix-pmon-restart-count-not-clear branch March 9, 2026 07:49

stephenxs mentioned this pull request Mar 9, 2026

[202511] Fix issue: pmon services's restart count is not cleared during config reload (#4314) #4336

Merged

StormLiangMS mentioned this pull request Mar 10, 2026

[202511] Fix issue: pmon services restart count is not cleared during config reload (#4314) #4341

Closed

StormLiangMS mentioned this pull request Mar 10, 2026

[202511] Add conditional skip for test_load_minigraph_with_golden_config sonic-net/sonic-mgmt#22830

Merged

zili11720 added a commit to zili11720/sonic-utilities that referenced this pull request Mar 11, 2026

Revert "Fix issue: pmon services's restart count is not cleared durin…

7b3c354

…g config reload (sonic-net#4314)" This reverts commit 4d0cc93.

Conversation

stephenxs commented Feb 27, 2026

What I did

How I did it

How to verify it

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

Uh oh!

mssonicbld commented Feb 27, 2026

Uh oh!

azure-pipelines bot commented Feb 27, 2026

Uh oh!

stephenxs commented Mar 3, 2026

Uh oh!

mssonicbld commented Mar 3, 2026

Uh oh!

azure-pipelines bot commented Mar 3, 2026

Uh oh!

StormLiangMS commented Mar 6, 2026

Cross-reference: Complementary fix in sonic-buildimage

Root cause analysis

Why 202505 doesn't have this issue

How the two PRs complement each other

Uh oh!

StormLiangMS left a comment

Choose a reason for hiding this comment

Uh oh!

StormLiangMS commented Mar 6, 2026

Uh oh!

saiarcot895 Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

stephenxs Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

StormLiangMS commented Mar 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mssonicbld commented Mar 9, 2026

Uh oh!

azure-pipelines bot commented Mar 9, 2026

Uh oh!

liat-grozovik commented Mar 9, 2026

Uh oh!

Uh oh!

saiarcot895 commented Mar 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

StormLiangMS commented Mar 7, 2026 •

edited

Loading