Academia.eduAcademia.edu

Figure from:Building a Comprehensive Automated Programming Assessment System

See full PDFdownloadDownload figure
Mekterovicé et a/.: Building a Comprehensive Automated Programming Assessment System This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: D l= oe, 10.1109/ACCESS.2020.2990980, IEEE Access

Table 3 Mekterovicé et a/.: Building a Comprehensive Automated Programming Assessment System This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: D l= oe, 10.1109/ACCESS.2020.2990980, IEEE Access

Related Figures (25)

TABLE 1. Edgar's usage statistics for the top five courses We believe that automated and online assessment tools present a modern addition to the classic methods of program- ming skills assessment and that they will become a standard tool to CS teaching in the future. For instance, in our courses we typically combine several automated assessments (exams in a laboratory), several automated homework assignments, Edgar has been in use since the spring semester of 2016/2017, is still used today and it is going to be used in the foreseeable future. It has become an indispensable tool for programming courses at FER — Table 1 shows the usage statistics for top five courses with respect to the number of enrolled students.
TABLE 1. Edgar's usage statistics for the top five courses We believe that automated and online assessment tools present a modern addition to the classic methods of program- ming skills assessment and that they will become a standard tool to CS teaching in the future. For instance, in our courses we typically combine several automated assessments (exams in a laboratory), several automated homework assignments, Edgar has been in use since the spring semester of 2016/2017, is still used today and it is going to be used in the foreseeable future. It has become an indispensable tool for programming courses at FER — Table 1 shows the usage statistics for top five courses with respect to the number of enrolled students.
FIGURE 1. Minimal APAS feature set order to support heterogeneous requirements in the computer science domain. Teachers should be empowered to indepen- dently administer the entire course through the user-friendly GUI and potentially export all the data collected. In order to classify an APAS as "comprehensive" we’ve identified necessary or minimal feature set. Some of these features are often neglected or not discussed in the literature, namely the first two rows in Fig. 1. This feature set is compatible with the Abello et al’s requirements for e-assessment system ("allow the delivery of assessment activities, the recording of re- sponses, timely feedback, automatic grading, and weighted- average grade calculation") [26], Noonan’s grading system phases ("“delivery, assessment, marking, return") [27], and extends the Wang et al’s. conception of a web-based assess- ment and test analysis (WATA) system based on Triple A model (assembling, administering, appraising") [28] .
FIGURE 1. Minimal APAS feature set order to support heterogeneous requirements in the computer science domain. Teachers should be empowered to indepen- dently administer the entire course through the user-friendly GUI and potentially export all the data collected. In order to classify an APAS as "comprehensive" we’ve identified necessary or minimal feature set. Some of these features are often neglected or not discussed in the literature, namely the first two rows in Fig. 1. This feature set is compatible with the Abello et al’s requirements for e-assessment system ("allow the delivery of assessment activities, the recording of re- sponses, timely feedback, automatic grading, and weighted- average grade calculation") [26], Noonan’s grading system phases ("“delivery, assessment, marking, return") [27], and extends the Wang et al’s. conception of a web-based assess- ment and test analysis (WATA) system based on Triple A model (assembling, administering, appraising") [28] .
FIGURE 2. Student’s application logging events during an exam FALL AVUlVilsS A OLUUILTIL ULIUILILAAYYS Vil UY YVACIIL vPeCSY dit UI browser should be logged. This helps teachers conduct the exam, monitor students as they take the exam, and helps resolve any issues later, after the exam has been submitted. Surprisingly, this topic is neglected in literature; more so since it plays a very important role in the real-world ex- amination process. Edgar provides a rich set of features for exam monitoring. During the exam, the client single page application logs all events that occur to the server via AJAX calls and also sends a heartbeat to indicate whether it is alive (Fig. 2 and Fig. 3). During the exam, teachers have a dashboard with overview of students currently running an exam with basic information — personal details, IP address, image, lost focus flags and a heartbeat. Fig. 3 shows exam monitoring dashboard. We group the students according to their IP ranges (each range in a different tab) corresponding to lab computers. Lost focus flags are useful in the context of multiple-choice questions where, in our case, it is typically forbidden to lose focus from the exam window, as that might indicate student browsing the web in another window to find the solution.
FIGURE 2. Student’s application logging events during an exam FALL AVUlVilsS A OLUUILTIL ULIUILILAAYYS Vil UY YVACIIL vPeCSY dit UI browser should be logged. This helps teachers conduct the exam, monitor students as they take the exam, and helps resolve any issues later, after the exam has been submitted. Surprisingly, this topic is neglected in literature; more so since it plays a very important role in the real-world ex- amination process. Edgar provides a rich set of features for exam monitoring. During the exam, the client single page application logs all events that occur to the server via AJAX calls and also sends a heartbeat to indicate whether it is alive (Fig. 2 and Fig. 3). During the exam, teachers have a dashboard with overview of students currently running an exam with basic information — personal details, IP address, image, lost focus flags and a heartbeat. Fig. 3 shows exam monitoring dashboard. We group the students according to their IP ranges (each range in a different tab) corresponding to lab computers. Lost focus flags are useful in the context of multiple-choice questions where, in our case, it is typically forbidden to lose focus from the exam window, as that might indicate student browsing the web in another window to find the solution.
Mekterovic et a/.: Building a Comprehensive Automated Programming Assessment System FIGURE 3. Exam monitoring view, students are grouped according to IP ranges, faces blurred on purpose
Mekterovic et a/.: Building a Comprehensive Automated Programming Assessment System FIGURE 3. Exam monitoring view, students are grouped according to IP ranges, faces blurred on purpose
FIGURE 4. Exam score box plot
FIGURE 4. Exam score box plot
FIGURE 6. Student behaviour analytics markdown is used to support tables and figures and MathJax library [32] is used for I4TRXstyle mathematical formulas. If a question that was already used by a student is saved via the edit form (e.g. to fix a bug in the question) Edgar will save changes to a new question while automatically deactivating the old question so it does not to appear in future exams. This keeps the history accurate, and past students’ exams with er- roneous question will still be linked to the original question. This behaviour can be overridden via "save in place" option which does not instantiate new question that could be useful for fixing (minor) text typos. Additionally, questions can have custom tags having meanings (e.g. "requires-attention" to mark a question to be checked) and list of reviewers. In our experience it is very important to structure a strict question writing/reviewing process and to assign responsibility for the question to author and reviewer. Questions are organized into a network of nodes of arbitrary types (e.g. unit, module, tutorial, etc.) and a question belongs to one or many nodes. In practice, we’ve encountered a dilemma whether to structure the questions and nodes thematically or organizationally. It appears that organizational structure is more used, but often both are used as shown in Fig. 7 and Fig. 8. Except in rare cases when the same unit, module or tutorial is used in several courses, nodes are typically structured hierarchically in a tree. Once the questions are grouped in nodes it is possible to define an exam using those nodes. An exam can have multiple parts where one exam part corresponds to one node and additionally defines minimum and maximum questions to be generated at random from the assigned node. Building on the example in Fig. 7 we could, for instance, define an exam with three questions using two exam parts (EP):
FIGURE 6. Student behaviour analytics markdown is used to support tables and figures and MathJax library [32] is used for I4TRXstyle mathematical formulas. If a question that was already used by a student is saved via the edit form (e.g. to fix a bug in the question) Edgar will save changes to a new question while automatically deactivating the old question so it does not to appear in future exams. This keeps the history accurate, and past students’ exams with er- roneous question will still be linked to the original question. This behaviour can be overridden via "save in place" option which does not instantiate new question that could be useful for fixing (minor) text typos. Additionally, questions can have custom tags having meanings (e.g. "requires-attention" to mark a question to be checked) and list of reviewers. In our experience it is very important to structure a strict question writing/reviewing process and to assign responsibility for the question to author and reviewer. Questions are organized into a network of nodes of arbitrary types (e.g. unit, module, tutorial, etc.) and a question belongs to one or many nodes. In practice, we’ve encountered a dilemma whether to structure the questions and nodes thematically or organizationally. It appears that organizational structure is more used, but often both are used as shown in Fig. 7 and Fig. 8. Except in rare cases when the same unit, module or tutorial is used in several courses, nodes are typically structured hierarchically in a tree. Once the questions are grouped in nodes it is possible to define an exam using those nodes. An exam can have multiple parts where one exam part corresponds to one node and additionally defines minimum and maximum questions to be generated at random from the assigned node. Building on the example in Fig. 7 we could, for instance, define an exam with three questions using two exam parts (EP):
FIGURE 8. Real world node structure from the Database course. Bubble sizes reflect the number of questions. Bubble (course units) labels are not meant to be readable. FIGURE 7. Parallel node structure - organizational (gray) and thematical (white)
FIGURE 8. Real world node structure from the Database course. Bubble sizes reflect the number of questions. Bubble (course units) labels are not meant to be readable. FIGURE 7. Parallel node structure - organizational (gray) and thematical (white)
Mekterovicé et a/.: Building a Comprehensive Automated Programming Assessment System
Mekterovicé et a/.: Building a Comprehensive Automated Programming Assessment System
TABLE 2. Currently implemented question types and the associated properties time left (starting from 1 and approaching 0 as the final value). Teacher can enter a custom formula to define the final value, e.g. [5] * (—0.05 « [NV] + 1.05) «(0.7 *[Z] +0.3) where S is base score described above. A detailed example of C and SQL question evaluation is provided in the Appendix A. where PP is the penalty_percentage - the percentage of points to deduct (starting from 100%) if student missed the correct answer in the multi-choice question or failed a test- case in the code question. SQL and Mongo questions have a single test-case (i.e. recordset comparison) with PP=100%. Consequently, the former can have correctness in the [0, 1] range, while later have binary correctness — 0 or 1. Grading model defines how to calculate question score from answer correctness and is defined as a triple of real numbers: FIGURE 9. Exam definition form with N=3 questions and two exam parts with different grading models
TABLE 2. Currently implemented question types and the associated properties time left (starting from 1 and approaching 0 as the final value). Teacher can enter a custom formula to define the final value, e.g. [5] * (—0.05 « [NV] + 1.05) «(0.7 *[Z] +0.3) where S is base score described above. A detailed example of C and SQL question evaluation is provided in the Appendix A. where PP is the penalty_percentage - the percentage of points to deduct (starting from 100%) if student missed the correct answer in the multi-choice question or failed a test- case in the code question. SQL and Mongo questions have a single test-case (i.e. recordset comparison) with PP=100%. Consequently, the former can have correctness in the [0, 1] range, while later have binary correctness — 0 or 1. Grading model defines how to calculate question score from answer correctness and is defined as a triple of real numbers: FIGURE 9. Exam definition form with N=3 questions and two exam parts with different grading models
FIGURE 10. Example of an exam. The answer to the current C programming language question passed one of three test-cases (the first test-case is public)
FIGURE 10. Example of an exam. The answer to the current C programming language question passed one of three test-cases (the first test-case is public)
FIGURE 11. Captioned test-cases with detailed, programmatically generated, feedback his article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: D Pre Aaa 10.1109/ACCESS.2020.2990980, IEEE Access
FIGURE 11. Captioned test-cases with detailed, programmatically generated, feedback his article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: D Pre Aaa 10.1109/ACCESS.2020.2990980, IEEE Access
Mekterovié et a/.: Building a Comprehensive Automated Programming Assessment System FIGURE 12. Exam review. The student can see saved evaluation data and can re-run their code. The student incorrectly used the float type instead of double and failed two out of three test-cases due to differences in precision of those two types)
Mekterovié et a/.: Building a Comprehensive Automated Programming Assessment System FIGURE 12. Exam review. The student can see saved evaluation data and can re-run their code. The student incorrectly used the float type instead of double and failed two out of three test-cases due to differences in precision of those two types)
FIGURE 13. Edgar’s modular architecture (3" party components shown in gray) Mekterovic et a/.: Building a Comprehensive Automated Programming Assessment System
FIGURE 13. Edgar’s modular architecture (3" party components shown in gray) Mekterovic et a/.: Building a Comprehensive Automated Programming Assessment System
Mekterovic¢ et a/.: Building a Comprehensive Automated Programming Assessment System FIGURE 14. Performance issues on the March the 19" (execution times over 400 seconds) This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: D I me ee 10.1109/ACCESS.2020.2990980, IEEE Access
Mekterovic¢ et a/.: Building a Comprehensive Automated Programming Assessment System FIGURE 14. Performance issues on the March the 19" (execution times over 400 seconds) This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: D I me ee 10.1109/ACCESS.2020.2990980, IEEE Access
FIGURE 15. Code runner (Java) execution times by the time of day for whole semester
FIGURE 15. Code runner (Java) execution times by the time of day for whole semester
FIGURE 17. Execution times by server for the whole semester FIGURE 16. Code runner (Java) execution times by the time for 2019-05-16
FIGURE 17. Execution times by server for the whole semester FIGURE 16. Code runner (Java) execution times by the time for 2019-05-16
Mekterovic et a/.: Building a Comprehensive Automated Programming Assessment System
Mekterovic et a/.: Building a Comprehensive Automated Programming Assessment System
FIGURE 18. Distribution of execution by server for the winter semester(19/20))
FIGURE 18. Distribution of execution by server for the winter semester(19/20))
FIGURE 19. Execution by server for the winter semester(19/20)
FIGURE 19. Execution by server for the winter semester(19/20)
TABLE 3. Initial list of 18 APASs with exclusion criteria the field and a large body of potential users, the choice of APAS software for a university looking to introduce auto- mated assessment is very limited. It boils down to just few systems that are still lacking in certain areas. TABLE 4. Feature comparison matrix for selected systems from Table 3. Columns: F1-Course administration; F3-Exam logging and monitoring; F4-Problem mitigation; F5-Analysis & visualisation; F6-Data import & export; F7-Content authoring; F8-Rich question types and grading facilities; F9-User friendly online testing; P/OS-proprietary or open source
TABLE 3. Initial list of 18 APASs with exclusion criteria the field and a large body of potential users, the choice of APAS software for a university looking to introduce auto- mated assessment is very limited. It boils down to just few systems that are still lacking in certain areas. TABLE 4. Feature comparison matrix for selected systems from Table 3. Columns: F1-Course administration; F3-Exam logging and monitoring; F4-Problem mitigation; F5-Analysis & visualisation; F6-Data import & export; F7-Content authoring; F8-Rich question types and grading facilities; F9-User friendly online testing; P/OS-proprietary or open source
Mekterovic et a/.: Building a Comprehensive Automated Programming Assessment System
Mekterovic et a/.: Building a Comprehensive Automated Programming Assessment System
TABLE 5. SQL DELETE statement question definition and student’s answer TABLE 6. Evaluating correctness for the example in Table 5
TABLE 5. SQL DELETE statement question definition and student’s answer TABLE 6. Evaluating correctness for the example in Table 5
and members) using reflection and use test-case inputs to select set of unit tests contained inside suffix! For SQL questions the use of prefix+source+suffix scheme allows to test not only SELECT statements but also record updates, deletions, and even DML expressions like ALTER TABLE or CREATE INDEX by querying the system catalogue in suffix part. Execution of the assembled code is delegated to an external runner (as described in Section IV) to be run in an isolated environment. Results are then returned to Edgar to be assessed and graded. Edgar features dif- ferent data structures for different programming languages (Table 2) with corresponding comparison parameters used to assess the solutions. Here we present SQL and C as- sessments. SQL assessment is illustrated with the example in Table 5 using comparison parameters shown in Table 6. In this example, INSERT statements in the prefix are used to temporarily change the database’s content with rows that were not visible to student during the exam. This is typi- cally used to prevent students from hard-coding the solution (e.g. SELECT 1, ’first row’ UNION SELECT 2, ‘second row’ ...). When the complete SQL code is assembled for the teacher and the student (Table 6), record sets presenting correct solution (denoted RST - RecordSet- Teacher) and student’s solution (denoted RSS - RecordSet- Student) are acquired and subsequently compared. Since in this scenario, student’s solution is erroneous, RSS will con- tain both rows inserted with statements from the prefix part, while RST will not contain row with id=100. The student will be presented with a RSP (RecordSetPresentation) rather than RST as it will be easier to spot and correct error based on a smaller data set. Note that unlike code questions or multiple- choice questions the correctness of SQL questions is binary — 0% or 100%.
and members) using reflection and use test-case inputs to select set of unit tests contained inside suffix! For SQL questions the use of prefix+source+suffix scheme allows to test not only SELECT statements but also record updates, deletions, and even DML expressions like ALTER TABLE or CREATE INDEX by querying the system catalogue in suffix part. Execution of the assembled code is delegated to an external runner (as described in Section IV) to be run in an isolated environment. Results are then returned to Edgar to be assessed and graded. Edgar features dif- ferent data structures for different programming languages (Table 2) with corresponding comparison parameters used to assess the solutions. Here we present SQL and C as- sessments. SQL assessment is illustrated with the example in Table 5 using comparison parameters shown in Table 6. In this example, INSERT statements in the prefix are used to temporarily change the database’s content with rows that were not visible to student during the exam. This is typi- cally used to prevent students from hard-coding the solution (e.g. SELECT 1, ’first row’ UNION SELECT 2, ‘second row’ ...). When the complete SQL code is assembled for the teacher and the student (Table 6), record sets presenting correct solution (denoted RST - RecordSet- Teacher) and student’s solution (denoted RSS - RecordSet- Student) are acquired and subsequently compared. Since in this scenario, student’s solution is erroneous, RSS will con- tain both rows inserted with statements from the prefix part, while RST will not contain row with id=100. The student will be presented with a RSP (RecordSetPresentation) rather than RST as it will be easier to spot and correct error based on a smaller data set. Note that unlike code questions or multiple- choice questions the correctness of SQL questions is binary — 0% or 100%.
TABLE 7. Code question definition and student's answer REFERENCES
TABLE 7. Code question definition and student's answer REFERENCES
TABLE 8. Evaluating correctness for the example in Table 7. Shaded cells within a row are compared JSON questions are evaluated in a similar way as SQL questions, also with binary outcome. When comparing ob- jects that are results of teachers and students code, JSON comparison can do shallow (deep) and strict comparison of objects.
TABLE 8. Evaluating correctness for the example in Table 7. Shaded cells within a row are compared JSON questions are evaluated in a similar way as SQL questions, also with binary outcome. When comparing ob- jects that are results of teachers and students code, JSON comparison can do shallow (deep) and strict comparison of objects.