Describe the bug
The repair services cronjob introduced in PRs #11566 and #11707 has been broken since the initial installation.
We have had it commented out on our main server for many months.
The repair services cron job runs once an hour at the top of the hour.
I believe the problem is that if there is an moderately compute intensive autograding job that started before the top of the hour and is running through the change to the next hour the job will be corrupted/ignored/not recorded. (Sorry I don't remember what the problem was exactly).
Anyway, autograding breaks -- not just the grading of that student, but it ends up disabling all or part of the autograding shipper/worker daemons. Ironically the repair services job causes a huge problem rather than searches for and repairs problems.
Expected behavior
I am going to commit a PR that comments out repair services. (PR #12788)
This issue is to remind us to debug the repair services script and restore the cron job.
Describe the bug
The repair services cronjob introduced in PRs #11566 and #11707 has been broken since the initial installation.
We have had it commented out on our main server for many months.
The repair services cron job runs once an hour at the top of the hour.
I believe the problem is that if there is an moderately compute intensive autograding job that started before the top of the hour and is running through the change to the next hour the job will be corrupted/ignored/not recorded. (Sorry I don't remember what the problem was exactly).
Anyway, autograding breaks -- not just the grading of that student, but it ends up disabling all or part of the autograding shipper/worker daemons. Ironically the repair services job causes a huge problem rather than searches for and repairs problems.
Expected behavior
I am going to commit a PR that comments out repair services. (PR #12788)
This issue is to remind us to debug the repair services script and restore the cron job.