This PR aims to fix #6219

Please add 'Resolves: #6219' line to the commit message.

I'd argue that the StartLimit settings introduced alongside it should accomplish the goal of avoiding an infinite loop in most scenarios:

     StartLimitIntervalSec=50s
     StartLimitBurst=5

IIUC, this limits the rate and allows boot to proceed, but it doesn't prevent infinite loop.

I would suggest to make sure 'sssd' exit due to internal watchdog has unique exit code and then use RestartForceExitStatus for this code.

IIUC, this limits the rate and allows boot to proceed, but it doesn't prevent infinite loop.

Well it should stop an infinite loop when sssd manages to exit 5 times in the configured 50 seconds.
If it's too slow, for example because of a slow LDAP server, I guess you could still end up in an infinite loop, but the adjusted restart policy does not change anything about that, so I'd say it's something to be looked at separately.

If it's too slow, for example because of a slow LDAP server

Broken DNS (unresponsive server), for example.

All the complains in #6219 are about restart after internal watchdog.
That's why I think handling this case explicitly is safer.

All the complains in #6219 are about restart after internal watchdog.
That's why I think handling this case explicitly is safer.

I see where you come from but what exit code would you define in RestartForceExitStatus ?
lemme explain what i mean...

In the Issue #6219 people see was terminated by own WATCHDOG which IIUC would correspond to Exit Code 70?
see SSS_WATCHDOG_EXIT_CODE constant and the was terminated by own WATCHDOG message here

However the exit code that systemd gets differs from case to case.

Some examples, unfortunately not a lot of people have shown their journalctl -u sssd output :/

1. In the other PR systemd: Restart=on-failure #8408 you can see exit 1:

Jan 26 17:53:06 ### sssd[1537]: Child [921930] ('###':'###') was terminated by own WATCHDOG. Consult corresponding logs to figure out the reason.
Jan 26 17:53:36 ### sssd_nss[921939]: Starting up
Jan 26 17:53:38 ### sssd[1537]: Exiting the SSSD. Could not restart critical service [nss].
Jan 26 17:53:38 ### sssd_be[921937]: Starting up
Jan 26 17:53:59 ### sssd_be[921937]: Shutting down (status = 0)
Jan 26 17:54:03 ### sssd_pac[921910]: Shutting down (status = 0)
Jan 26 17:54:08 ### systemd[1]: sssd.service: Main process exited, code=exited, status=1/FAILURE

2. This Person does not state the exact code but nonzero:
   sssd don't restart properly after being killed by watchdog #6219 (comment)

3. I myself have encountered exit 6 (and others which I cant find anymore):

Jan 09 00:04:08  sssd[3797703]: Child [2807075] ('ldap':'%BE_ldap') was terminated by own WATCHDOG. Consult corresponding logs to figure out the reason.
Jan 09 00:04:08  sssd[3797703]: Exiting the SSSD. Could not restart critical service [ldap].
Jan 09 00:04:08  sssd_pam[3797706]: Shutting down (status = 0)
Jan 09 00:04:08  sssd_nss[3797705]: Shutting down (status = 0)
Jan 09 00:04:08  systemd[1]: sssd.service: Main process exited, code=killed, status=6/ABRT

So ultimately there are many different scenarios like high cpu load, high io load, someone said it started after he saw oom messages etc... which result in different non-zero exit codes.

From my perspective as a sysadmin I would want SSSD to be restarted in any of them and for the unrecoverable ones we have a backoff mechanism

     StartLimitIntervalSec=50s
     StartLimitBurst=5

@Squiddim
@alexey-tikhonov proposes to modify SSSD's code to always return a well-known return code when terminated by the watchdog, and then use this return code in the RestartForceExitStatus.

I think this might work.

@alexey-tikhonov proposes to modify SSSD's code to always return a well-known return code when terminated by the watchdog, and then use this return code in the RestartForceExitStatus.

Right, that's what I meant (though required code modification and testing might be not so trivial).

But good point made is that this wouldn't cover OOM case, that also needs automatic restart.

If it's too slow, for example because of a slow LDAP server

Broken DNS (unresponsive server), for example.

But in this case sssd starts in offline mode, it is not terminated., or is it?

All the complains in #6219 are about restart after internal watchdog. That's why I think handling this case explicitly is safer.

How about going the other way around and use on-failure and RestartPreventExitStatus= to prevent restarts when SSSD cannot really start (misconfiguration)... but then again these cases would be covered by the burst configuration.

If it's too slow, for example because of a slow LDAP server

Broken DNS (unresponsive server), for example.

But in this case sssd starts in offline mode, it is not terminated., or is it?

Main SSSD process ('monitor') has hard coded timeout of 7 seconds for starting up all 'sssd_be' processes.
If 'sssd_be' can't resolve own hostname, it doesn't report to 'monitor' it started and thus gets terminated.

The pull request was accepted by @alexey-tikhonov with the following PR CI status:

🟢 CodeQL (success)
🟢 osh-diff-scan:fedora-rawhide-x86_64:upstream (success)
🟢 rpm-build:centos-stream-10-x86_64:upstream (success)
🟢 rpm-build:fedora-42-x86_64:upstream (success)
🟢 rpm-build:fedora-43-x86_64:upstream (success)
🟢 rpm-build:fedora-rawhide-x86_64:upstream (success)
🟢 Analyze (target) / cppcheck (success)
🟢 Build / freebsd (success)
🟢 Build / make-distcheck (success)
➖ ci / intgcheck (skipped)
🔴 ci / prepare (failure)
➖ ci / system (skipped)
➖ Coverity scan / coverity (skipped)
🟢 Static code analysis / codeql (success)
🟢 Static code analysis / pre-commit (success)
🟢 Static code analysis / python-system-tests (success)

There are unsuccessful or unfinished checks. Make sure that the failures are not related to this pull request before merging.

We tried this out on Rocky9.7, but unfortunately it doesn't seem to fix #6219 for us (we're having the same problem as this comment).

it doesn't solve the problem

We tried this out on Rocky9.7, but unfortunately it doesn't seem to fix #6219 for us (we're having the same problem as this comment).

it doesn't solve the problem

Unfortunately, those comments are useless without logs and in the closed PR.