[Feature:System] Add autograding testcases to database#12660
Conversation
automateprojectmangement
Bot
moved this from Seeking Reviewer
to Work in Progress
in Submitty Development
dagemcn
changed the title
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #12660 +/- ##
============================================
- Coverage 21.50% 21.49% -0.01%
Complexity 9896 9896
============================================
Files 268 268
Lines 36968 36992 +24
Branches 495 495
============================================
+ Hits 7951 7953 +2
- Misses 28526 28548 +22
Partials 491 491
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
…tty/Submitty into full_autograding_testcases fixed python linting again
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
NicholasCiuica
left a comment
NicholasCiuica
left a comment
There was a problem hiding this comment.
I tested this PR, see below the images for my method. I also did a quick code review, and all your additions look reasonable. My only concern is that currently, the following configurations on autograders are not considered major changes and do not delete stale testcase data, although it's possible they should:
- amount of time the submission can work before termination
- number of processes the submission is allowed to launch
- the list of allowed system calls that can be run by the compilation testcase/the whole test suite
- various details about the tests that are not stored in the database, like title, command, and all validation parameters
See below an image of the autograder config and my changes which were not considered major.
Method:
I ran submitty_install, logged in as instructor, rebuilt the a regular and a team autograded homework, and ran SELECT * FROM autograding_testcase in the SQL Toolbox to see each autograded test and the submission limit for this gradeable successfully added to the database for each gradeable rebuilt. The gradeable_id, testcase_id, testcase_order, hidden and extra credit booleans, and points_possible values for each autograder were correctly set.
I also regraded multiple autograded submissions within each rebuilt gradeable and saw that their user_id, team_id, g_version, and points_earned were all successfully added to the autograding_testcase_data table. I regraded them again and saw they were still in the database.
I made my own autograder gradeable, made a submission, and rebuilt it with no change and then a minor change (changing gradeable name) and saw that the testcase data for these gradeable's submissions was not deleted from the database, as expected. Then I rebuilt the same gradeable with a major change (changing the number of points for a testcase) and saw the stale testcase data removed from the database, as expected.
I regraded submissions on non-rebuilt gradeables and the autograding testcase data was not entered into the database, as expected.
|
I have made the changes requested @GarvitKhandelwal31. Let me know if you have any other changes. Also let me know if I implemented the db.begin() logic properly as I'm not sure I completely understood your request. |
GarvitKhandelwal31
left a comment
GarvitKhandelwal31
left a comment
There was a problem hiding this comment.
Everything looks good to me, I have not tested it though, but since Nick has done that, it should be good to go
github-project-automation
Bot
moved this from In Review
to Awaiting Maintainer Review
in Submitty Development
bmcutler
left a comment
bmcutler
left a comment
There was a problem hiding this comment.
new DB tables look good
quick test install was ok.
NOTE: This will not insert autograding data for new submissions to existing gradeables unless the gradeable is rebuilt first.
github-project-automation
Bot
moved this from Awaiting Maintainer Review
to Done
in Submitty Development
### Why is this Change Important & Necessary? First part of Submitty#12360 ### What is the New Behavior? Each time a homework is built, the gradeable_id, testcase_id, testcase_order, hidden and extra credit booleans, and points_possible fields will all be filled in inside the autograding_testcase table. Each time an autograded gradeable is graded, fields like user_id, team_id, g_version, and points_earned will also be populated in the autograding_testcase_data table, but only if the homework has already been built and the corresponding entries in autograding_testcase have been filled in. If a homework is rebuilt and the number of test cases, points per test case, test case name, or if the a test case is hidden or extra credit, any autograding results in the database will be deleted as they are stale. ### What steps should a reviewer take to reproduce or test the bug or new feature? 1. Run submitty_install - this will apply new migration 2. Log in as instructor 3. Go to any assignment with autograded testcases 4. Click on the pencil to edit the assignment and click the submissions/autograding tab 5. Scroll to the bottom and click 'rebuild gradeable' (alternatively create a new gradeable with autograding) 6. Then go back to the gradeables page and click on the grading button 7. Click on any student and regrade their submission OR submit a new submission 8. Go to the SQL Toolbox 9. Use `SELECT * FROM autograding_testcase;` and `SELECT * FROM autograding_testcase_data` or similar database queries in the SQL Toolbox to see the new rows that have been added to the database ### Automated Testing & Documentation Cypress tests to be added. Manual testing done with team and non-team gradeables, rebuilding with minor changes and major changes that should trigger a deletion, regrading a submission with and without data in the database. ### Other information This is not a breaking change. There is a migration included within this PR. There are no known security concerns with this PR. --------- Co-authored-by: JManion32 <[email protected]> Co-authored-by: Barb Cutler <[email protected]> Co-authored-by: Barb Cutler <Barb Cutler>
### Why is this Change Important & Necessary? Closes Issue #12360. ### What is the New Behavior? Prior to the change: <img width="2040" height="340" alt="image" src="https://github.com/user-attachments/assets/53eb04fc-79ab-4ee5-991c-4583b6f6ba9b" /> After the change: <img width="2274" height="1132" alt="image" src="https://github.com/user-attachments/assets/16574cd1-f769-4d09-a385-1afea1a74f59" /> ### What steps should a reviewer take to reproduce or test the bug or new feature? 1. Run `submitty_install_site` 2. Log in as instructor 3. Go to any gradeable with autograding (best to test with are `Autograder Hidden and Extra Credit` and `Autograde and TA Homework (C System Calls)`) 4. Go to the statistics and charts page 5. Click on the new tab 6. Ensure all new features work correctly, the new tab, the checkboxes, the new filters ### Automated Testing & Documentation Cypress tests should be added for this, I am working on this ### Other information This takes advantage of the previously unused autograding_testcase and autograding_testcase_data tables implemented in PR #12660. If a gradeable was built before that PR was merged, they will not have results in the new tab.
6 participants
Why is this Change Important & Necessary?
First part of #12360
What is the New Behavior?
Each time a homework is built, the gradeable_id, testcase_id, testcase_order, hidden and extra credit booleans, and points_possible fields will all be filled in inside the autograding_testcase table. Each time an autograded gradeable is graded, fields like user_id, team_id, g_version, and points_earned will also be populated in the autograding_testcase_data table, but only if the homework has already been built and the corresponding entries in autograding_testcase have been filled in. If a homework is rebuilt and the number of test cases, points per test case, test case name, or if the a test case is hidden or extra credit, any autograding results in the database will be deleted as they are stale.
What steps should a reviewer take to reproduce or test the bug or new feature?
SELECT * FROM autograding_testcase;andSELECT * FROM autograding_testcase_dataor similar database queries in the SQL Toolbox to see the new rows that have been added to the databaseAutomated Testing & Documentation
Cypress tests to be added.
Manual testing done with team and non-team gradeables, rebuilding with minor changes and major changes that should trigger a deletion, regrading a submission with and without data in the database.
Other information
This is not a breaking change.
There is a migration included within this PR.
There are no known security concerns with this PR.