Skip to content

[Refactor:System] Declutter workers#12815

Merged
bmcutler merged 22 commits into
mainfrom
declutter-workers
Jun 25, 2026
Merged

[Refactor:System] Declutter workers#12815
bmcutler merged 22 commits into
mainfrom
declutter-workers

Conversation

@hyeenaa

@hyeenaa hyeenaa commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

Why is this Change Important & Necessary?

Might fix(?) #10918 -> unnecessary tools running on workers machines

What is the New Behavior?

The install scripts have been slightly changed to exclude some unnecessary background tools that might run after system installation. These can be stopped on installation or avoided entirely when booting up a vagrant worker.

The packages that will no longer be installed are: apache2, php-fpm8.2, psql, and node.

a 'worker' argument is passed into an installation script to maintain the context of the installation to avoid installing services that won't help workers.

What steps should a reviewer take to reproduce or test the bug or new feature?

if you have any vagrant workers, make sure to first run vagrant workers destroy

To test this PR:

  1. vagrant workers generate
  2. vagrant workers up (if you are on Linux or Windows and using the prebuilt VM, it is necessary to prepend FROM_SCRATCH=1, ex FROM_SCRATCH=1 vagrant workers up
  3. vagrant workers ssh <name> (whatever your worker's name is, most likely worker-1)
  4. Check to see that the packages were not installed: (if everything is working correctly, it should just say 'command not found')
  • apache2 -v
  • php-fpm8.2 -v
  • psql -V
  • node -v
  1. Additionally, test that the worker still functions for autograding.
  • in the main VM, run refresh_vagrant_workers, then run restart_shipper_and_all_workers.py from /usr/local/submitty/sbin.
  • The worker VM daemon should be successfully restarted.
  • Add some submissions to the autograding queue on Submitty and check to see that the worker VM is grading them successfully.

Automated Testing & Documentation

I might add tests for workers in the future to make sure no future updates break their generation.

Other information

roye2: I did additionally add a small change to fix the parsing of the vagrant workers commands. test vagrant workers halt , vagrant workers ssh , vagrant workers destroy, etc. It would be great if you can verify that these commands are broken on main and work on the declutter-workers branch.

@github-project-automation github-project-automation Bot moved this to Seeking Reviewer in Submitty Development Apr 29, 2026
@automateprojectmangement automateprojectmangement Bot moved this from Seeking Reviewer to Work in Progress in Submitty Development Apr 29, 2026
@hyeenaa hyeenaa changed the title Declutter workers [System:Refactor] Declutter workers Apr 29, 2026
@codecov

codecov Bot commented Apr 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 21.40%. Comparing base (75511e4) to head (becdc44).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##               main   #12815   +/-   ##
=========================================
  Coverage     21.40%   21.40%           
  Complexity     9944     9944           
=========================================
  Files           268      268           
  Lines         37124    37124           
  Branches        494      494           
=========================================
  Hits           7948     7948           
  Misses        28686    28686           
  Partials        490      490           
Flag Coverage Δ
autograder 21.16% <ø> (ø)
js 2.01% <ø> (ø)
migrator 100.00% <ø> (ø)
php 20.44% <ø> (ø)
python_submitty_utils 80.08% <ø> (ø)
submitty_daemon_jobs 91.13% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@hyeenaa hyeenaa changed the title [System:Refactor] Declutter workers [Refactor:System] Declutter workers Apr 29, 2026
@github-actions github-actions Bot added the Abandoned PR - Needs New Owner No activity on PR for more than 2 weeks -- seeking new owner to complete label May 14, 2026
@dagemcn dagemcn self-assigned this Jun 11, 2026
@dagemcn dagemcn removed the Abandoned PR - Needs New Owner No activity on PR for more than 2 weeks -- seeking new owner to complete label Jun 11, 2026
@dagemcn
dagemcn marked this pull request as ready for review June 11, 2026 19:28
@automateprojectmangement automateprojectmangement Bot moved this from Work in Progress to Seeking Reviewer in Submitty Development Jun 11, 2026
@dagemcn dagemcn removed their assignment Jun 23, 2026
@dandrecollins07-ctrl dandrecollins07-ctrl self-assigned this Jun 24, 2026

@dandrecollins07-ctrl dandrecollins07-ctrl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested on Windows. Generated and started a worker VM, confirmed vagrant workers halt, up, and ssh worked correctly, refreshed worker configuration, restarted shipper/workers, and successfully autograded a submission. Submission was graded on worker-1 as expected. Approving.

Image

@github-project-automation github-project-automation Bot moved this from Seeking Reviewer to Awaiting Maintainer Review in Submitty Development Jun 24, 2026

@dagemcn dagemcn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested on ARM M4, workers seem to be working. I was able to verify that the packages were not installed and the worker can still connect to the VM. Autograding was also working, although the autograding did not immediately switch over from primary to worker-1. I was also able to verify that the commands were working on the branch as well as testing on main where the commands were not working. Looks good.

@bmcutler
bmcutler merged commit 3dcf453 into main Jun 25, 2026
27 checks passed
@bmcutler
bmcutler deleted the declutter-workers branch June 25, 2026 20:44
@github-project-automation github-project-automation Bot moved this from Awaiting Maintainer Review to Done in Submitty Development Jun 25, 2026
jndlansh pushed a commit to jndlansh/Submitty that referenced this pull request Jul 1, 2026
<!-- ** Please remove all comment blocks in the description before
submitting this PR. ** -->


<!-- NOTE: Please ensure your title and description align with Submitty
conventions (see
https://submitty.org/developer/getting_started/make_a_pull_request for
more details).
Each title has a prefix, and a limit of 40 chars. A description template
has been
provided and must be completed in full. Please also ensure that all CI
tests
are passing. Pull requests that do not meet these requirements are
ineligible for
review and may be closed. -->

### Why is this Change Important & Necessary?
<!-- Include any GitHub issue that is fixed/closed using "Fixes
#<number>" or "Closes #<number>" syntax.
Alternately write "Partially addresses #<number>" or "Related to
#<number>" as appropriate. -->

Might fix(?) Submitty#10918 -> unnecessary tools running on workers machines

### What is the New Behavior?
<!-- Include before & after screenshots/videos if the user interface has
changed. -->

The install scripts have been slightly changed to exclude some
unnecessary background tools that might run after system installation.
These can be stopped on installation or avoided entirely when booting up
a vagrant worker.

The packages that will no longer be installed are: apache2, php-fpm8.2,
psql, and node.

a 'worker' argument is passed into an installation script to maintain
the context of the installation to avoid installing services that won't
help workers.

### What steps should a reviewer take to reproduce or test the bug or
new feature?

if you have any vagrant workers, make sure to first run `vagrant workers
destroy`

To test this PR:
1.  `vagrant workers generate`
2. `vagrant workers up` (if you are on Linux or Windows and using the
prebuilt VM, it is necessary to prepend FROM_SCRATCH=1, ex
`FROM_SCRATCH=1 vagrant workers up`
3. `vagrant workers ssh <name>` (whatever your worker's name is, most
likely worker-1)
4. Check to see that the packages were not installed: (if everything is
working correctly, it should just say 'command not found')
- `apache2 -v`
- `php-fpm8.2 -v`
- `psql -V`
- `node -v`
5. Additionally, test that the worker still functions for autograding.
- in the main VM, run `refresh_vagrant_workers`, then run
restart_shipper_and_all_workers.py from /usr/local/submitty/sbin.
- The worker VM daemon should be successfully restarted. 
- Add some submissions to the autograding queue on Submitty and check to
see that the worker VM is grading them successfully.

### Automated Testing & Documentation
<!-- Is this feature sufficiently tested by unit tests and end-to-end
tests?
If this PR does not add/update the necessary automated tests, write a
new GitHub issue and link it below.
Is this feature sufficiently documented on submitty.org?
Link related PRs or new GitHub issue to update documentation. -->

I might add tests for workers in the future to make sure no future
updates break their generation.


### Other information
<!-- Is this a breaking change?  
Does this PR include migrations to update existing installations?  
Are there security concerns with this PR? -->
roye2: I did additionally add a small change to fix the parsing of the
vagrant workers commands. test vagrant workers halt <name>, vagrant
workers ssh <name>, vagrant workers destroy, etc. It would be great if
you can verify that these commands are broken on main and work on the
declutter-workers branch.

---------

Co-authored-by: dagemcn <[email protected]>
Co-authored-by: dagemcn <[email protected]>
Co-authored-by: D'Andre Collins <[email protected]>
Co-authored-by: Emma Roy <[email protected]>
prestoncarman added a commit that referenced this pull request Jul 1, 2026
* main: (78 commits)
  [Bugfix:TAGrading] Auto-open single file (#12625)
  [Feature:InstructorUI] Add sortable columns to manage students (#12957)
  [Dependency] Bump php-ds/php-ds from 1.7.0 to 2.0.1 in /site (#12827)
  [Refactor:TAGrading] Events Infrastructure & StatusBanner (#12941)
  [Dependency] Bump sqlalchemy from 2.0.48 to 2.0.51 in /.setup/pip (#12884)
  [Bugfix:Developer] trust hashicorp for vagrant install (#12968)
  [Feature:TAGrading] Backend for submission clustering (#12886)
  [Bugfix:InstructorUI] Missing Subsection in the Edit User form (#12955)
  [Bugfix:Submission] Warning Banners Accessibility Fix (#12956)
  [Bugfix:System] Preserve signup fields (#12939)
  [Bugfix:InstructorUI] Fix Csv Download of Subsections (#12954)
  [Bugfix:InstructorUI] Fix Manage Students Toggle Columns (#12953)
  [Feature:TAGrading] Persistent Auto-Open (#12931)
  [Bugfix:Submission] Dot File Upload (#12924)
  [Refactor:System] Declutter workers (#12815)
  [Bugfix:Developer] save/restore autograding_containers.json (#12952)
  [Bugfix:InstructorUI] Create Course Validation (#12949)
  [Testing:Notifications] Fix grade_inquiries.spec.js (#12950)
  [UI/UX:System] Add fullscreen button vue component (#12936)
  [Bugfix:TAGrading] Student Name Resize (#12946)
  ...

# Conflicts:
#	.pylintrc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

5 participants