Skip to content

[Refactor:Autograding] Deprecate submitty_count#12609

Draft
tijaruS wants to merge 18 commits into
Submitty:mainfrom
tijaruS:fix/deprecate-submitty-count
Draft

[Refactor:Autograding] Deprecate submitty_count#12609
tijaruS wants to merge 18 commits into
Submitty:mainfrom
tijaruS:fix/deprecate-submitty-count

Conversation

@tijaruS

@tijaruS tijaruS commented Mar 19, 2026

Copy link
Copy Markdown
Contributor

Why is this Change Important & Necessary?

Fixes #11784

This change is necessary to fully deprecate submitty_count from AnalysisTools as it has been replaced by the more modern Tree-sitter-based submitty_count_ts. The old tool is being phased out, and this PR cleans up the remaining infrastructure and examples.

Related to: https://submitty.org/instructor/autograding/static_analysis/count
Alongside this PR there are 3 other companion PRs in other Submitty repos.
DockerImages - PR#75 - merged
DockerImagesRPI - PR#41 - merged
Tutorial - PR#56 - merged

What is the New Behavior?

submitty_count is no longer in the allowed autograding commands by default.
The autograder (TestCase.cpp/TestCase.h) no longer contains specific logic or resource limit adjustments for submitty_count.
All submitty_count commands in the python_multipart_static_analysis example and its associated tests have been updated to use the submitty_count_ts syntax.
Submitty count is also no longer installed on the autograding-default docker image, the docker image for csci1200, and is no longer used in the tutorial examples.

What steps should a reviewer take to reproduce or test the bug or new feature?

  1. Checkout all PRs including this PR and the 3 companion PRs (no longer required, they are merged)
  2. Do a vagrant destroy and vagrant up with from scratch
  3. Make sure that Tutorial examples 4, 6, and 7 all still build and autograde correctly
  4. Make sure that python_multipart_static_analysis still builds and autogrades correctly.
  5. Verify that submitty_count is no longer listed in /usr/local/submitty/config/allowed_autograding_commands_default.json after running an installation.

Also refer to the testing instructions in the 3 companion PRs. There will be instructions how to test the changes they made.

Automated Testing & Documentation

  • Documentation for submitty_count_ts is already available on submitty.org but the page should be updated.

Other Information

THIS IS A BREAKING CHANGE - any custom gradeables still manually using submitty_count will need their config.json updated to use submitty_count_ts.
There are no migrations.
No security concerns.

@codecov

codecov Bot commented Mar 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 21.10%. Comparing base (c8a359a) to head (9069792).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##               main   #12609   +/-   ##
=========================================
  Coverage     21.10%   21.10%           
  Complexity    10115    10115           
=========================================
  Files           282      282           
  Lines         38265    38265           
  Branches        596      596           
=========================================
  Hits           8074     8074           
  Misses        29609    29609           
  Partials        582      582           
Flag Coverage Δ
autograder 21.16% <ø> (ø)
js 1.73% <ø> (ø)
migrator 100.00% <ø> (ø)
php 20.18% <ø> (ø)
python_submitty_utils 79.83% <ø> (ø)
submitty_daemon_jobs 91.13% <ø> (ø)
vue 100.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@tijaruS tijaruS changed the title [Cleanup:Autograding] Deprecate submitty_count [Refactor:Autograding] Deprecate submitty_count in favor of submitty_count_ts Mar 24, 2026
@tijaruS tijaruS changed the title [Refactor:Autograding] Deprecate submitty_count in favor of submitty_count_ts [Refactor:Autograding] Deprecate submitty_count Mar 24, 2026
@tijaruS

tijaruS commented Mar 31, 2026

Copy link
Copy Markdown
Contributor Author

Hello can anyone review this pr? @williamjallen

@tijaruS tijaruS changed the title [Refactor:Autograding] Deprecate submitty_count [Refactor:Autograding] Deprecate submitty_count Apr 6, 2026
@github-actions github-actions Bot added the Abandoned PR - Needs New Owner No activity on PR for more than 2 weeks -- seeking new owner to complete label Apr 21, 2026

@GarvitKhandelwal31 GarvitKhandelwal31 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tijaruS Please fix the CI/ Integration workflow, before I can start with the review.

@automateprojectmangement automateprojectmangement Bot moved this from Seeking Reviewer to Work in Progress in Submitty Development Jun 25, 2026
bmcutler pushed a commit to Submitty/DockerImages that referenced this pull request Jul 13, 2026
### What is the current behavior?
Fixes Submitty
[Issue#11784](Submitty/Submitty#11784) - This
one of the companion PRs to Submitty
[PR#12609](Submitty/Submitty#12609).

### What is the new behavior?
submitty_count is longer installed in preparation for deprecation.

### Other information?
<!-- Is this a breaking change? -->
This is a breaking change. Any autograding that uses the old
submitty_count instead of submitty_count_ts will no longer work.
<!-- How did you test -->
1. Follow the instructions on [this
page](https://submitty.org/instructor/autograding/docker_images) up to
step 6 with the Dockerfile modified in this PR. (you may be able to
reuse the one I used `dagemcn/dockerimages:default-test` adding it
through docker UI)
2. After you have added the new docker image with the DockerUI, update
the config of any gradeable with `submitty_count` with
```
"autograding_method": "docker",
"container_options": {
    "container_image": "IMAGE_NAME"
},
``` 
where `IMAGE_NAME` is the name of the image, for example,
`dagemcn/dockerimages:default-test` if you reused mine.
3. Create a gradeable using submitty_count, (Tutorial examples 04, 06,
and 07 all utilize submitty_count)
4. Test that submitty_count will not work in the config and that
submitty_count_ts successfully runs
5. Also test that regular autograding works successfully on the image

Attached is a zip of a sample solution and gradeable config with both
submitty_count and submitty_count_ts commands present for testing.

[example.zip](https://github.com/user-attachments/files/29638384/example.zip)
bmcutler pushed a commit to Submitty/DockerImagesRPI that referenced this pull request Jul 13, 2026
### What is the current behavior?
Fixes Submitty
[Issue#11784](Submitty/Submitty#11784) - This
one of the companion PRs to Submitty
[PR#12609](Submitty/Submitty#12609).

### What is the new behavior?
submitty_count is longer installed in preparation for deprecation.

### Other information?
<!-- Is this a breaking change? -->
This is a breaking change. Any autograding that uses the old
submitty_count instead of submitty_count_ts will no longer work.
<!-- How did you test -->
1. Follow the instructions on [this
page](https://submitty.org/instructor/autograding/docker_images) up to
step 6 with the Dockerfile modified in this PR. (you may be able to
reuse the one I used `dagemcn/dockerimages:csci1200-test` adding it
through docker UI)
2. After you have added the new docker image with the DockerUI, update
the config of any gradeable with `submitty_count` with
```
"autograding_method": "docker",
"container_options": {
    "container_image": "IMAGE_NAME"
},
``` 
where `IMAGE_NAME` is the name of the image, for example,
`dagemcn/dockerimages:csci1200-test` if you reused mine.
3. Create a gradeable using submitty_count, (Tutorial examples 04, 06,
and 07 all utilize submitty_count)
4. Test that submitty_count will not work in the config and that
submitty_count_ts successfully runs
5. Also test that regular autograding works successfully on the image

Attached is a zip of a sample solution and gradeable config with both
submitty_count and submitty_count_ts commands present for testing.

[example.zip](https://github.com/user-attachments/files/29638384/example.zip)
bmcutler pushed a commit to Submitty/Tutorial that referenced this pull request Jul 13, 2026
### What is the current behavior?
Fixes Submitty
[Issue#11784](Submitty/Submitty#11784) - This
one of the companion PRs to Submitty
[PR#12609](Submitty/Submitty#12609).
The tutorial courses 04 - Python Static Analysis, 06 - Loop Types, and
07 - Loop Depth all use `submitty_count` in their autograding configs.
This command is being deprecated, so we should update it.

### What is the new behavior?
`submitty_count` has been removed from `04 - Python Static Analysis` and
`06 - Loop Types` and replaced with `submitty_count_ts` in preparation
for deprecation. For `07 - Loop Depth`, there was no equivalent
functionality present in `submitty_count_ts` so the testcase is entirely
commented out.

### Other information?
<!-- Is this a breaking change? -->
This specific PR is not a breaking change. However, the other PRs
accompanying this one are breaking PRs.
<!-- How did you test -->
Tested that all homework configs still build and their accompanying
still submissions work as intended.

@roye2 roye2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies for the long screenshot:

Image

When I tested this with the solutions, I found that it was not working as intended, the errors can be seen in the screenshot. I am not sure how submitty_count_ts works but I'm wondering if it's being used incorrectly here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Abandoned PR - Needs New Owner No activity on PR for more than 2 weeks -- seeking new owner to complete

Projects

Status: Work in Progress

Development

Successfully merging this pull request may close these issues.

Deprecated submitty count

5 participants