Usage Table for Instructors

Instructors looking to design randomized computer-based questions for their courses should take into account a range of considerations. From a pedagogical standpoint, each level of randomization in the framework introduces distinct effects on student problem-solving approaches and may be most suitable for different stages of learning. As the level of randomization increases, the complexity and overhead of implementation may also rise. Hence, it is critical to align decisions with available resources and instructional goals. For the design of questions in assessments, careful attention should also be given to logistics, where instructors should find a balance between implementing randomization as a security measure and ensuring fairness in evaluation.

The table below outlines these pedagogical and logistical considerations associated with each level of our randomization framework.

Level	Label	Effect on Approach	Example Features	Pedagogical Considerations	Logistical Considerations
0	Unrandomized	N/A	N/A	May result in students memorizing answers.	Offers weakest exam security.
1	Surface Features	Students do not need to account for the feature in their approach.	Changing order of MC options, variable names	Helps students practice question understanding and extract important information. Tests student understanding on a specific type or pattern of question.	Discourages answer sharing within exam room.
2	Conditions	Students can reuse the exact same approach across question variants.	Changing initial values and states	Helps students practice calculations and execute algorithms. Tests student understanding on a specific type or pattern of question.	Discourages answer sharing within the exam room.
3	Scenarios	Students can use a similar approach with slight adjustments across question variants.	Giving different cases of the same algorithm	Modifies question scenarios to help students build critical thinking skills. Tests student understanding of multiple scenarios within a concept.	Protects against meaningful information leak. May lead to a slight variance in difficulty.
4	Concepts	Students have to use different approaches across question variants.	Asking about different data structures and implementations	Modifies question concepts to help students build critical thinking skills. Tests student understanding of multiple concepts.	Protects against meaningful information leak. May lead to a moderate variance in difficulty.
5	Different Questions	N/A	N/A	Enables multiple attempts of assessments.	Offers strongest exam security. May lead to a wide variance in difficulty.