[Feature:System] Repair Submitty Services by jeffrey-cordero · Pull Request #11566 · Submitty/Submitty

jeffrey-cordero · 2025-03-26T14:48:55Z

Please check if the PR fulfills these requirements:

Tests for the changes have been added/updated (if possible)
Documentation has been updated/added if relevant
Screenshots are attached to Github PR if visual/UI changes were made

What is the current behavior?

Submitty services, such as the WebSocket server, are not restarted regularly.

What is the new behavior?

A new cron job has been added to restart any services that are not currently active.

Other information?

You must run submitty_install to ensure the cron-related files are updated and the system migration runs for the new logging directory utilized by the new cron job that runs every hour. You can run the following commands to test a broken WebSocket server scenario.

$submitty_install_bin
$sudo systemctl stop submitty_websocket_server
$su submitty_daemon
$sudo /usr/local/submitty/sbin/repair_services.sh
Restarting submitty_websocket_server
...
$ sudo systemctl status submitty_websocket_server
● submitty_websocket_server.service - Submitty WebSocket Server
     Loaded: loaded (/etc/systemd/system/submitty_websocket_server.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2025-03-26 10:40:07 EDT; 4min 39s ago
       Docs: http://submitty.org
   Main PID: 2339 (php)
      Tasks: 1 (limit: 2274)
     Memory: 16.3M
        CPU: 34ms
     CGroup: /system.slice/submitty_websocket_server.service
             └─2339 php /usr/local/submitty/site/socket/index.php

The cron job was also confirmed to work after re-installing Submitty, submitty_install, where all logs are stored in /var/log/services.

Restarting submitty_websocket_server

○ submitty_websocket_server.service - Submitty WebSocket Server
     Loaded: loaded (/etc/systemd/system/submitty_websocket_server.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Tue 2025-04-08 16:43:41 EDT; 3s ago
       Docs: http://submitty.org
    Process: 22580 ExecStart=/usr/bin/env php /usr/local/submitty/site/socket/index.php (code=killed, signal=TERM)
   Main PID: 22580 (code=killed, signal=TERM)
        CPU: 41ms

Apr 08 16:42:34 ubuntu systemd[1]: Started Submitty WebSocket Server.
Apr 08 16:43:41 ubuntu systemd[1]: Stopping Submitty WebSocket Server...
Apr 08 16:43:41 ubuntu systemd[1]: submitty_websocket_server.service: Deactivated successfully.
Apr 08 16:43:41 ubuntu systemd[1]: Stopped Submitty WebSocket Server.

----------------------------------------

Fixes #10450

codecov · 2025-03-26T14:50:29Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 22.07%. Comparing base (2521492) to head (074f4e1).
Report is 40 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11566      +/-   ##
============================================
+ Coverage     22.06%   22.07%   +0.01%     
- Complexity     8888     8906      +18     
============================================
  Files           246      247       +1     
  Lines         32039    32108      +69     
  Branches         79       79              
============================================
+ Hits           7069     7089      +20     
- Misses        24894    24943      +49     
  Partials         76       76

Flag	Coverage Δ
autograder	`21.60% <ø> (ø)`
js	`26.85% <ø> (ø)`
migrator	`100.00% <ø> (ø)`
php	`19.06% <ø> (-0.03%)`	⬇️
python_submitty_utils	`80.08% <ø> (ø)`
submitty_daemon_jobs	`88.88% <ø> (-2.06%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

lavalleeale · 2025-04-08T15:30:58Z

+        status=$(sudo systemctl status "${service}")
+        if ! echo "${status}" | grep -q 'Active: active'; then


Suggested change

status=$(sudo systemctl status "${service}")

if ! echo "${status}" | grep -q 'Active: active'; then

if ! sudo systemctl is-active --quiet "${service}"; then

Manually greping the output seems flaky, systemctl is-active will return the result we want directly as an error code

lavalleeale

I suggested the one change that I think this PR needs before it is ready to be approved, although I wonder if we want to add more logging to this since it might keep us unaware of a frequently crashing service.

lavalleeale

Looks good to me now, hopefully it will never have to do anything, but I tested and when I manually stop a service it returns within the minute.

bmcutler · 2025-04-11T18:31:33Z

 # Update OS Info and Service Health every hour
 0 * * * * submitty_daemon   /usr/local/submitty/sbin/update_worker_sysinfo.sh UpdateSystemInfo
+# Run the repair_services.sh script every minute
+* * * * * submitty_daemon   sudo /usr/local/submitty/sbin/repair_services.sh


Let's back this off to once an hour. We don't need to check it quite so frequently.
We should update the worker health message to once an hour as well.

I don't want to burden the server with things we expect to (essentially) never need.

martig7

I was able to confrim that when the websocket service is manually stopped it restarts on the minute of the hour given by submitty_crontab. While testing this I noticed that reconnecting to the websocket server while viewing the poll histogram causes some unexpected behavior on the histogram, but that's far outside of scope, just making a note of it here to go back to later.

lavalleeale

Installed and can confirm stopped services get restarted at the top of every hour, with appropiate logs added to /var/log/services

bmcutler

thanks for the PR!

Updates to the System Customization, Automated Grading, and Websockets / System & Debugging documentation pages have been implemented to document how services are restarted automatically via the cron job introduced within [#11566](Submitty/Submitty#11566), how to disable this hourly script, and how to set automatic notifications for service failure outputs. This makes progress on [#11622](Submitty/Submitty#11622).

### Please check if the PR fulfills these requirements: * [ ] Tests for the changes have been added/updated (if possible) * [ ] Documentation has been updated/added if relevant * [ ] Screenshots are attached to Github PR if visual/UI changes were made ### What is the current behavior?  The existing repair services cron job, originally introduced in #11566, only restarts local auto-grading shippers/worker daemons via `systemctl`, but this doesn't restart remote machine auto-grading components, which can be crucial for production environments. ### What is the new behavior? Now, when the local autograding shipper status is not active, the cron job will invoke the script to restart all local and remote autograding shippers/workers. ```bash root@ubuntu:/usr/local/submitty# sudo sbin/repair_services.sh 2025-07-16:08:42:35: Autograding Shipper Failure detected within the Autograding Shipper shipper daemon is inactive shipper daemon is inactive worker daemon is inactive shipper daemon is active worker daemon is active Stopping the local shipper daemon... Stopping all local worker daemon... Stopping all remote worker daemons... Delaying 5 seconds to allow the system to stabilize... Starting the local shipper daemon... Starting the local worker daemon... Starting all worker daemons... Delaying 5 seconds to allow the system to stablize... Verifying all worker daemons... Finished! ○ submitty_autograding_shipper.service - Submitty Autograding Shipper Loaded: loaded (/etc/systemd/system/submitty_autograding_shipper.service; enabled; vendor preset: enabled) Active: inactive (dead) since Wed 2025-07-16 08:42:02 EDT; 21s ago Docs: http://submitty.org Process: 122666 ExecStart=/usr/local/submitty/autograder/submitty_autograding_shipper.py (code=killed, signal=TERM) Main PID: 122666 (code=killed, signal=TERM) CPU: 3min 41.173s Jul 16 08:42:01 ubuntu submitty_autograding_shipper.py[122666]: JOB 682 True 2025-07-15 14:27:09-0400 /var/local/submitty/to_be_graded_queue/f25__testing__locked_homework__schaeb__1 Jul 16 08:42:01 ubuntu submitty_autograding_shipper.py[122666]: JOB 683 True 2025-07-15 14:27:09-0400 /var/local/submitty/to_be_graded_queue/f25__testing__locked_homework__schusa__1 Jul 16 08:42:01 ubuntu submitty_autograding_shipper.py[122666]: JOB 684 True 2025-07-15 14:27:09-0400 /var/local/submitty/to_be_graded_queue/f25__testing__locked_homework__sengea__1 Jul 16 08:42:01 ubuntu submitty_autograding_shipper.py[122666]: JOB 685 True 2025-07-15 14:27:09-0400 /var/local/submitty/to_be_graded_queue/f25__testing__locked_homework__sengea__2 Jul 16 08:42:01 ubuntu submitty_autograding_shipper.py[122666]: JOB 686 True 2025-07-15 14:27:09-0400 /var/local/submitty/to_be_graded_queue/f25__testing__locked_homework__sengea__3 Jul 16 08:42:01 ubuntu submitty_autograding_shipper.py[122666]: JOB 687 True 2025-07-15 14:27:09-0400 /var/local/submitty/to_be_graded_queue/f25__testing__locked_homework__ta__1 Jul 16 08:42:02 ubuntu systemd[1]: Stopping Submitty Autograding Shipper... Jul 16 08:42:02 ubuntu systemd[1]: submitty_autograding_shipper.service: Deactivated successfully. Jul 16 08:42:02 ubuntu systemd[1]: Stopped Submitty Autograding Shipper. Jul 16 08:42:02 ubuntu systemd[1]: submitty_autograding_shipper.service: Consumed 3min 41.173s CPU time. ---------------------------------------- ``` Additionally, the cron job script has been refactored to include timestamps in messages and provide clearer service identifiers. ```bash 2025-07-16:08:46:51: WebSocket Server Failure detected within the WebSocket Server ○ submitty_websocket_server.service - Submitty WebSocket Server Loaded: loaded (/etc/systemd/system/submitty_websocket_server.service; enabled; vendor preset: enabled) Active: inactive (dead) since Wed 2025-07-16 08:46:47 EDT; 3s ago Docs: http://submitty.org Process: 106708 ExecStart=/usr/bin/env php /usr/local/submitty/site/socket/index.php (code=killed, signal=TERM) Main PID: 106708 (code=killed, signal=TERM) CPU: 49ms Jul 16 08:36:34 ubuntu systemd[1]: Started Submitty WebSocket Server. Jul 16 08:46:47 ubuntu systemd[1]: Stopping Submitty WebSocket Server... Jul 16 08:46:47 ubuntu systemd[1]: submitty_websocket_server.service: Deactivated successfully. Jul 16 08:46:47 ubuntu systemd[1]: Stopped Submitty WebSocket Server. ---------------------------------------- ``` ### Other information?   To test, run `submitty_install` for the migration, run `sudo systemctl stop submitty_autograding_shipper`, and finally run `sudo sbin/repair_services.sh` to get the output above. This makes progress on #11622.

Restart services on an hourly basis

b4da35d

github-project-automation Bot added this to Submitty Development Mar 26, 2025

github-project-automation Bot moved this to Seeking Reviewer in Submitty Development Mar 26, 2025

jeffrey-cordero and others added 3 commits March 26, 2025 10:55

Bash Lint Fix

70d4b01

Actual Bash Lint Fix

76073f6

Merge branch 'main' into restart-submitty-services

45853b8

jeffrey-cordero moved this from Seeking Reviewer to Work in Progress in Submitty Development Mar 28, 2025

jeffrey-cordero added 2 commits March 28, 2025 16:50

New cron file

c9c005f

Updated cron instructions

64392d3

jeffrey-cordero marked this pull request as draft March 28, 2025 21:46

jeffrey-cordero added 2 commits April 5, 2025 14:17

Updated cron setup

d5b8ff7

Automatic CRON invocation confirmation

72da4ba

jeffrey-cordero moved this from Work in Progress to Seeking Reviewer in Submitty Development Apr 7, 2025

jeffrey-cordero marked this pull request as ready for review April 7, 2025 18:23

lavalleeale reviewed Apr 8, 2025

View reviewed changes

lavalleeale requested changes Apr 8, 2025

View reviewed changes

github-project-automation Bot moved this from Seeking Reviewer to Work in Progress in Submitty Development Apr 8, 2025

Improved status and logging measures

6240c15

jeffrey-cordero requested a review from lavalleeale April 8, 2025 21:05

automateprojectmangement Bot moved this from Work in Progress to In Review in Submitty Development Apr 8, 2025

jeffrey-cordero added 2 commits April 8, 2025 17:09

Shell Linting

c425204

Migration for logging directory

fc62e2b

jeffrey-cordero changed the title ~~[Feature:System] Restart Submitty Services~~ [Feature:System] Repair Submitty Services Apr 10, 2025

lavalleeale approved these changes Apr 11, 2025

View reviewed changes

github-project-automation Bot moved this from In Review to Awaiting Maintainer Review in Submitty Development Apr 11, 2025

bmcutler requested changes Apr 11, 2025

View reviewed changes

github-project-automation Bot moved this from Awaiting Maintainer Review to Work in Progress in Submitty Development Apr 11, 2025

Pivot to one hour cron job

074f4e1

martig7 approved these changes Apr 14, 2025

View reviewed changes

zacharymnp moved this from Work in Progress to Awaiting Maintainer Review in Submitty Development Apr 15, 2025

lavalleeale approved these changes Apr 15, 2025

View reviewed changes

williamjallen requested a review from bmcutler April 25, 2025 19:44

automateprojectmangement Bot moved this from Awaiting Maintainer Review to In Review in Submitty Development Apr 25, 2025

bmcutler approved these changes Apr 25, 2025

View reviewed changes

github-project-automation Bot moved this from In Review to Awaiting Maintainer Review in Submitty Development Apr 25, 2025

bmcutler merged commit 5e86105 into main Apr 25, 2025

bmcutler deleted the restart-submitty-services branch April 25, 2025 19:50

bmcutler mentioned this pull request May 15, 2025

add cron job to check that autograding is running #11622

Open

2 tasks

This was referenced Jun 4, 2025

[Feature:System] Restart Local/Remote Autograding Daemons #11707

Merged

[Documentation:System] Repair Services Cron Job Documentation Submitty/submitty.github.io#679

Merged

bmcutler mentioned this pull request Apr 15, 2026

Repair Services needs debugging #12787

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Feature:System] Repair Submitty Services#11566

[Feature:System] Repair Submitty Services#11566
bmcutler merged 12 commits into
mainfrom
restart-submitty-services

jeffrey-cordero commented Mar 26, 2025 •

edited

Loading

Uh oh!

codecov Bot commented Mar 26, 2025 •

edited

Loading

Uh oh!

lavalleeale Apr 8, 2025 •

edited

Loading

Uh oh!

lavalleeale left a comment

Uh oh!

lavalleeale left a comment

Uh oh!

bmcutler Apr 11, 2025

Uh oh!

martig7 left a comment

Uh oh!

lavalleeale left a comment

Uh oh!

bmcutler left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		status=$(sudo systemctl status "${service}")
		if ! echo "${status}" \| grep -q 'Active: active'; then

	status=$(sudo systemctl status "${service}")
	if ! echo "${status}" \| grep -q 'Active: active'; then
	if ! sudo systemctl is-active --quiet "${service}"; then

Uh oh!

Conversation

jeffrey-cordero commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Please check if the PR fulfills these requirements:

What is the current behavior?

What is the new behavior?

Other information?

Uh oh!

codecov Bot commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

lavalleeale Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lavalleeale left a comment

Choose a reason for hiding this comment

Uh oh!

lavalleeale left a comment

Choose a reason for hiding this comment

Uh oh!

bmcutler Apr 11, 2025

Choose a reason for hiding this comment

Uh oh!

martig7 left a comment

Choose a reason for hiding this comment

Uh oh!

lavalleeale left a comment

Choose a reason for hiding this comment

Uh oh!

bmcutler left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jeffrey-cordero commented Mar 26, 2025 •

edited

Loading

codecov Bot commented Mar 26, 2025 •

edited

Loading

lavalleeale Apr 8, 2025 •

edited

Loading