What problem are you trying to solve with Submitty
Currently the breakdown of autograding test cases per student per submission are only available in the file system. Specifically in /var/local/submitty/courses/SEMESTER/COURSE/results/GRADEABLE/USERNAME/VERSION/RESULTS/grade.txt and results.json.
Instructors would benefit from a way to search and pull statistics about student performance on autograding test cases from the database.
The database currently stores the total of multiple autograding tests (hidden, nonhidden, extra credit) but not individual scores.
Describe the way you'd like to solve this problem
Let's add an additional table to the database to store these autograding details.
While we should write a migration to create this table for all existing courses, let's not attempt to fill this table with data for autograding that was completed before the update is installed.
So we will be missing data for old gradeables. That's ok.
Once we have this data populated for submissions autograded after installation, instructors will be able to collect statistics through read only access to SQL Toolbox. And we will be able to add a tab with a breakdown of autograding per problem to the "statistics & charts" button of TA grading.
Note: This data is complicated since each student can submit multiple times and will thus have multiple copies of autograding results. The 'active version' is usually the highest numbered version but not always. This will also be messy if the instructor modifies the autograding script to change the number of autograding tests, or change the max number of points awarded for one or more tests. If the instructor does not regrade all submissions for all students after these changes that this data could be inconsistent/confusing. Also, when an assignment is regraded, we'll need to make sure to remove all scores for that version from the database, not just replace the scores (in case the total number of test cases is decreased).
feature requested by @KCony
What problem are you trying to solve with Submitty
Currently the breakdown of autograding test cases per student per submission are only available in the file system. Specifically in /var/local/submitty/courses/SEMESTER/COURSE/results/GRADEABLE/USERNAME/VERSION/RESULTS/grade.txt and results.json.
Instructors would benefit from a way to search and pull statistics about student performance on autograding test cases from the database.
The database currently stores the total of multiple autograding tests (hidden, nonhidden, extra credit) but not individual scores.
Describe the way you'd like to solve this problem
Let's add an additional table to the database to store these autograding details.
While we should write a migration to create this table for all existing courses, let's not attempt to fill this table with data for autograding that was completed before the update is installed.
So we will be missing data for old gradeables. That's ok.
Once we have this data populated for submissions autograded after installation, instructors will be able to collect statistics through read only access to SQL Toolbox. And we will be able to add a tab with a breakdown of autograding per problem to the "statistics & charts" button of TA grading.
Note: This data is complicated since each student can submit multiple times and will thus have multiple copies of autograding results. The 'active version' is usually the highest numbered version but not always. This will also be messy if the instructor modifies the autograding script to change the number of autograding tests, or change the max number of points awarded for one or more tests. If the instructor does not regrade all submissions for all students after these changes that this data could be inconsistent/confusing. Also, when an assignment is regraded, we'll need to make sure to remove all scores for that version from the database, not just replace the scores (in case the total number of test cases is decreased).
feature requested by @KCony