Describe the bug
When our production system server is rebooted, the autograding shipper & workers do not restart automatically. (This may be a misconfiguration on our system.). This reboot happens infrequently -- usually when there is a campus power or network outage -- and we forget to check & restart the autograding.
Expected behavior
Describe the bug
When our production system server is rebooted, the autograding shipper & workers do not restart automatically. (This may be a misconfiguration on our system.). This reboot happens infrequently -- usually when there is a campus power or network outage -- and we forget to check & restart the autograding.
Expected behavior
We've just added a cron script to detect and restart the websockets. Let's do the same with the autograding.
[Feature:System] Repair Submitty Services #11566
let's update the system administrator documentation for both websockets and autograding to document how they are restarted automatically, how to disable that cron job (if specific testing is to be done), and how to check the logs about these restarts, and set up automatic notifications about these logs.