[Feature:System] Restart Local/Remote Autograding Daemons#11707
Merged
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #11707 +/- ##
============================================
- Coverage 20.70% 20.67% -0.03%
- Complexity 9194 9214 +20
============================================
Files 265 265
Lines 35220 35276 +56
Branches 458 456 -2
============================================
+ Hits 7292 7295 +3
- Misses 27474 27529 +55
+ Partials 454 452 -2
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
github-actions
Bot
added
the
Abandoned PR - Needs New Owner
jeffrey-cordero
removed
the
Abandoned PR - Needs New Owner
github-actions
Bot
added
the
Abandoned PR - Needs New Owner
bmcutler
requested changes
Jul 10, 2025
bmcutler
left a comment
bmcutler
left a comment
Member
There was a problem hiding this comment.
Production log before installing this PR:
As I noted, we are failing to restart the nullsmtpd that doesn't exist on production:
But also, there are lots of blank lines -- why?
And let's add a timestamp to these restarts.
github-project-automation
Bot
moved this from Seeking Reviewer
to Work in Progress
in Submitty Development
automateprojectmangement
Bot
moved this from Work in Progress
to In Review
in Submitty Development
jeffrey-cordero
removed
the
Abandoned PR - Needs New Owner
bmcutler
reviewed
Jul 15, 2025
| echo -e "----------------------------------------" | sudo tee -a "${log_file}" | ||
| } | ||
|
|
||
| repair_autograding() { |
Member
There was a problem hiding this comment.
Rather than duplicate logic... I would rather just call the existing script:
sbin/restart_shipper_and_all_workers.py
If any problems are detected with the shippers or workers on the primary machine.
This halts all all the shippers on the primary machine AND the workers on the primary machine AND the workers processes on worker machines AND brings up all the shippers & workers in the proper order. If we change anything in the future about how these work, we don't want to have to (remember to) make the update in 2 places.
bmcutler
requested changes
Jul 15, 2025
bmcutler
left a comment
bmcutler
left a comment
Member
There was a problem hiding this comment.
thanks for making the other updates... but we've got (unneccesary?) code duplication...
github-project-automation
Bot
moved this from In Review
to Work in Progress
in Submitty Development
Member
|
Sorry if I've misread the code, but I'm worried about automating this reset if something goes wrong with a remote worker. |
bmcutler
reviewed
Jul 17, 2025
| } | ||
|
|
||
| repair_autograding() { | ||
| local status_script="/usr/local/submitty/sbin/shipper_utils/systemctl_wrapper.py" |
Member
There was a problem hiding this comment.
do you still need this? since we are only checking the primary machine, i don't think we should need this script... right? can't it look like the other systemctl status checks below?
Contributor
Author
There was a problem hiding this comment.
It can, but I forgot to convert back to the direct systemctl call instead of calling the wrapper script during the refactor.
bmcutler
approved these changes
Jul 18, 2025
github-project-automation
Bot
moved this from In Review
to Awaiting Maintainer Review
in Submitty Development
williamschen23
pushed a commit
that referenced
this pull request
Jul 18, 2025
### Please check if the PR fulfills these requirements: * [ ] Tests for the changes have been added/updated (if possible) * [ ] Documentation has been updated/added if relevant * [ ] Screenshots are attached to Github PR if visual/UI changes were made ### What is the current behavior? <!-- List issue if it fixes/closes/implements one using the "Fixes #<number>" or "Closes #<number>" syntax --> The existing repair services cron job, originally introduced in #11566, only restarts local auto-grading shippers/worker daemons via `systemctl`, but this doesn't restart remote machine auto-grading components, which can be crucial for production environments. ### What is the new behavior? Now, when the local autograding shipper status is not active, the cron job will invoke the script to restart all local and remote autograding shippers/workers. ```bash root@ubuntu:/usr/local/submitty# sudo sbin/repair_services.sh 2025-07-16:08:42:35: Autograding Shipper Failure detected within the Autograding Shipper shipper daemon is inactive shipper daemon is inactive worker daemon is inactive shipper daemon is active worker daemon is active Stopping the local shipper daemon... Stopping all local worker daemon... Stopping all remote worker daemons... Delaying 5 seconds to allow the system to stabilize... Starting the local shipper daemon... Starting the local worker daemon... Starting all worker daemons... Delaying 5 seconds to allow the system to stablize... Verifying all worker daemons... Finished! ○ submitty_autograding_shipper.service - Submitty Autograding Shipper Loaded: loaded (/etc/systemd/system/submitty_autograding_shipper.service; enabled; vendor preset: enabled) Active: inactive (dead) since Wed 2025-07-16 08:42:02 EDT; 21s ago Docs: http://submitty.org Process: 122666 ExecStart=/usr/local/submitty/autograder/submitty_autograding_shipper.py (code=killed, signal=TERM) Main PID: 122666 (code=killed, signal=TERM) CPU: 3min 41.173s Jul 16 08:42:01 ubuntu submitty_autograding_shipper.py[122666]: JOB 682 True 2025-07-15 14:27:09-0400 /var/local/submitty/to_be_graded_queue/f25__testing__locked_homework__schaeb__1 Jul 16 08:42:01 ubuntu submitty_autograding_shipper.py[122666]: JOB 683 True 2025-07-15 14:27:09-0400 /var/local/submitty/to_be_graded_queue/f25__testing__locked_homework__schusa__1 Jul 16 08:42:01 ubuntu submitty_autograding_shipper.py[122666]: JOB 684 True 2025-07-15 14:27:09-0400 /var/local/submitty/to_be_graded_queue/f25__testing__locked_homework__sengea__1 Jul 16 08:42:01 ubuntu submitty_autograding_shipper.py[122666]: JOB 685 True 2025-07-15 14:27:09-0400 /var/local/submitty/to_be_graded_queue/f25__testing__locked_homework__sengea__2 Jul 16 08:42:01 ubuntu submitty_autograding_shipper.py[122666]: JOB 686 True 2025-07-15 14:27:09-0400 /var/local/submitty/to_be_graded_queue/f25__testing__locked_homework__sengea__3 Jul 16 08:42:01 ubuntu submitty_autograding_shipper.py[122666]: JOB 687 True 2025-07-15 14:27:09-0400 /var/local/submitty/to_be_graded_queue/f25__testing__locked_homework__ta__1 Jul 16 08:42:02 ubuntu systemd[1]: Stopping Submitty Autograding Shipper... Jul 16 08:42:02 ubuntu systemd[1]: submitty_autograding_shipper.service: Deactivated successfully. Jul 16 08:42:02 ubuntu systemd[1]: Stopped Submitty Autograding Shipper. Jul 16 08:42:02 ubuntu systemd[1]: submitty_autograding_shipper.service: Consumed 3min 41.173s CPU time. ---------------------------------------- ``` Additionally, the cron job script has been refactored to include timestamps in messages and provide clearer service identifiers. ```bash 2025-07-16:08:46:51: WebSocket Server Failure detected within the WebSocket Server ○ submitty_websocket_server.service - Submitty WebSocket Server Loaded: loaded (/etc/systemd/system/submitty_websocket_server.service; enabled; vendor preset: enabled) Active: inactive (dead) since Wed 2025-07-16 08:46:47 EDT; 3s ago Docs: http://submitty.org Process: 106708 ExecStart=/usr/bin/env php /usr/local/submitty/site/socket/index.php (code=killed, signal=TERM) Main PID: 106708 (code=killed, signal=TERM) CPU: 49ms Jul 16 08:36:34 ubuntu systemd[1]: Started Submitty WebSocket Server. Jul 16 08:46:47 ubuntu systemd[1]: Stopping Submitty WebSocket Server... Jul 16 08:46:47 ubuntu systemd[1]: submitty_websocket_server.service: Deactivated successfully. Jul 16 08:46:47 ubuntu systemd[1]: Stopped Submitty WebSocket Server. ---------------------------------------- ``` ### Other information? <!-- Is this a breaking change? --> <!-- How did you test --> To test, run `submitty_install` for the migration, run `sudo systemctl stop submitty_autograding_shipper`, and finally run `sudo sbin/repair_services.sh` to get the output above. This makes progress on #11622.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please check if the PR fulfills these requirements:
What is the current behavior?
The existing repair services cron job, originally introduced in #11566, only restarts local auto-grading shippers/worker daemons via
systemctl, but this doesn't restart remote machine auto-grading components, which can be crucial for production environments.What is the new behavior?
Now, when the local autograding shipper status is not active, the cron job will invoke the script to restart all local and remote autograding shippers/workers.
Additionally, the cron job script has been refactored to include timestamps in messages and provide clearer service identifiers.
2025-07-16:08:46:51: WebSocket Server Failure detected within the WebSocket Server ○ submitty_websocket_server.service - Submitty WebSocket Server Loaded: loaded (/etc/systemd/system/submitty_websocket_server.service; enabled; vendor preset: enabled) Active: inactive (dead) since Wed 2025-07-16 08:46:47 EDT; 3s ago Docs: http://submitty.org Process: 106708 ExecStart=/usr/bin/env php /usr/local/submitty/site/socket/index.php (code=killed, signal=TERM) Main PID: 106708 (code=killed, signal=TERM) CPU: 49ms Jul 16 08:36:34 ubuntu systemd[1]: Started Submitty WebSocket Server. Jul 16 08:46:47 ubuntu systemd[1]: Stopping Submitty WebSocket Server... Jul 16 08:46:47 ubuntu systemd[1]: submitty_websocket_server.service: Deactivated successfully. Jul 16 08:46:47 ubuntu systemd[1]: Stopped Submitty WebSocket Server. ----------------------------------------Other information?
To test, run
submitty_installfor the migration, runsudo systemctl stop submitty_autograding_shipper, and finally runsudo sbin/repair_services.shto get the output above. This makes progress on #11622.