[Feature:System] Validate autograding_workers.json schema#12559
Merged
Conversation
Add JSON schema validation for autograding_workers.json in load_autograding_workers_json(). Misconfigured worker entries (missing required fields, wrong types, unknown keys) now produce a clear error at shipper startup rather than a cryptic failure when the first job is submitted. jsonschema is already a pinned project dependency. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #12559 +/- ##
=========================================
Coverage 21.66% 21.66%
Complexity 9638 9638
=========================================
Files 268 268
Lines 36233 36233
Branches 486 486
=========================================
Hits 7849 7849
Misses 27902 27902
Partials 482 482
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Tests cover: valid config, missing required fields (num_autograding_workers, enabled), wrong types, num_autograding_workers below minimum, empty capabilities list, unknown/misspelled keys, and multiple worker entries. Each test is self-contained using tempfile and unittest.mock.MagicMock, with no dependency on global state or test ordering. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
github-actions
Bot
added
the
Abandoned PR - Needs New Owner
dandrecollins07-ctrl
approved these changes
Jun 4, 2026
dandrecollins07-ctrl
left a comment
dandrecollins07-ctrl
left a comment
Contributor
There was a problem hiding this comment.
On this PR branch I tested invalid configs by removing 'num_autograding_workers', changing it to the wrong type and misspelling 'capabilities'. In each case the shipper failed at startup and logged a clear validation error.
After restoring the config, the shipper started normally again. Approving.
github-project-automation
Bot
moved this from Seeking Reviewer
to Awaiting Maintainer Review
in Submitty Development
github-project-automation
Bot
moved this from Awaiting Maintainer Review
to Done
in Submitty Development
bmcutler
added a commit
that referenced
this pull request
Jul 10, 2026
### Why is this Change Important & Necessary? The autograding workers schema introduced in PR #12559 is missing the display_environment_variable field. This causes the autograding shipper to fail. ### What is the New Behavior? The field is added to the schema, and the shipper should now work. --------- Co-authored-by: Barb Cutler <Barb Cutler>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why is this Change Important & Necessary?
A typo or missing field in `autograding_workers.json` currently causes a cryptic `KeyError` or `TypeError` deep inside the shipper when the first job is submitted — not at startup. This makes misconfiguration difficult to diagnose, especially for administrators adding or reconfiguring a worker machine for the first time.
What is the New Behavior?
`load_autograding_workers_json()` now validates the parsed JSON against a schema immediately after loading. If the file is invalid, the shipper exits with a clear, actionable error at startup. For example:
The schema enforces that every worker entry has: `address` (string), `username` (string), `num_autograding_workers` (integer ≥ 1), `enabled` (boolean), and `capabilities` (non-empty array of strings). `jsonschema` is already a pinned dependency (`jsonschema==4.26.0` in `.setup/pip/system_requirements.txt`), so no new dependency is introduced.
What steps should a reviewer take to reproduce or test the bug or new feature?
Automated Testing & Documentation
8 unit tests added to `autograder/tests/test_submitty_autograding_shipper.py` covering:
Tests use `tempfile` and `unittest.mock.MagicMock` — each is self-contained with no dependency on global state or test ordering. No documentation changes required.
Other information
Not a breaking change for correctly configured installations. No migrations required. No security concerns.
This change was written by Claude Code (claude-sonnet-4-6).