1 Evaluating the evidence in assessment

2008

What is our warrant for saying "Student X deserves a Grade C" ? It must be based on evidence, and the only evidence we see is what students produce during the exam. For valid assessment two criteria must be met: the examination must elicit proper evidence of the trait, and we must evaluate the evidence properly.

2020

We are indebted to too many people to list here for enlightening discussions of topics addressed in this paper. We would like to acknowledge Lyle Bachman, Irwin Kirsch, Mary Schedl, and John Norris with regard to issues in language assessment, and, for their comments on an earlier draft, the editor Mark Wilson and two anonymous referees.

PsycEXTRA Dataset

Applied Measurement in Education, 2019

Despite the call for an argument-based approach to validity over 25 years ago, few examples exist in the published literature. One possible explanation for this outcome is that the complexity of the argument-based approach makes implementation difficult. To counter this claim, we propose that the Assessment Triangle can serve as the overarching framework for operationalizing and instantiating the argument-based approach to validation. Integrating these frameworks can streamline the validation process by providing a conceptual lens for identifying, collecting, and evaluating relevant sources of evidence throughout the testing process. To fully examine this proposed conceptualization, we apply the integrated framework to an example case of a universal screener for middleschool mathematics. We articulate an interpretation and use argument for the universal screener, and then present relevant sources of evidence to evaluate the plausibility of the inferences and warrants underlying test score use. Based on this applied example, the strengths and limitations of the integrated framework are considered, and recommendations are made for future instantiations. Multiple frameworks exist in educational assessment for making sense of validity. Perhaps the most comprehensive treatment of validity within the broader context of educational assessment is the National Research Council's report titled Knowing What Students Know (Pellegrino, Chudowsky, & Glaser, 2001). In this report, validity (interpretation) is integrally connected to theories of learning (cognition) and students' response processes on tested items, tasks, or situations (observation) in the Assessment Triangle. Interpretation is defined as the process of making meaning from the evidence obtained about students' knowledge, skills, and abilities. Cognition refers to the models of thinking and learning that underlie students' development of knowledge, skills, and abilities in the domain. Observation includes the items, tasks, or situations designed to elicit students' knowledge, skills, and abilities. Representing these components as the Assessment Triangle underscores the interdependent relationship and importance of alignment between these dimensions of educational assessment. Although more than 25 years have passed since the introduction of this conceptual model of educational assessment, the three integrated components are often examined separately. However, some connections between the components have begun to receive greater attention in the past 15 years. For example, recent research in mathematics and science education has called for designing classroom assessments based on learning trajectories and learning progressions (cf., Corcoran, Mosher, & Rogat, 2009; Daro, Mosher, & Corcoran, 2011), especially as it relates to providing teachers with instructionally relevant information. This work seeks to connect the cognition and interpretation vertices of the Assessment Triangle. Relatedly, researchers investigating students' response processes when interacting with items, tasks, or situations are attempting to examine the alignment between the intended and elicited cognitive processes, thereby connecting the cognition and observation vertices (cf., Leighton & Gierl, 2011; Padilla & Benitez, 2014).

Psychometrics in practice at RCEC, 2012

To evaluate the quality of educational assessments, several evaluation systems are available. These systems are, however, focused around the evaluation of a single type of test. Furthermore, within these systems, quality is defined as a non-flexible construct, whereas in this paper it is argued that the evaluation of test quality should depend on the test's purpose. Within this paper, we compare several available evaluation systems. From this comparison, design principles are derived to guide the development of a new, comprehensive quality evaluation system. The paper concludes with an outline of the new evaluation system, which intends to incorporate an argument-based approach to quality.

Psicothema, 2014

Validation is the process of providing evidence that tests and questionnaires are adequately and appropriately fulfilling the purposes for which they are developed. In this special issue, experts from several countries describe specific approaches to test validation and provide examples of their approach. These approaches and examples illustrate the validation framework implied by the Standards for Educational and Psychological Testing. We describe the Standards' approach for building a validity argument based on validity evidence based on test content, response processes, internal structure, relations to other variables, and testing consequences. The five articles provide comprehensive examples of gathering data regarding these five sources of evidence and how they contribute to the validation of the use of test scores for particular purposes. These five articles provide concrete examples of how the five sources of validity evidence suggested by the Standards can be used to dev...

2000

Many authors over the past twenty years have argued that the prevailing ‘psychometric’ paradigm for educational assessment is inappropriate and have proposed that educational assessment should develop its own distinctive paradigm. More recently (and particularly within the last five years) it has become almost commonplace to argue that changes in assessment methods are required because of changing views of human cognition, and in particular, the shift from ‘behaviourist’ towards ‘constructivist’ views of the nature of human learning. However, these changes are still firmly rooted within the psychometric paradigm, since within this perspective, the development of assessment is an essentially ‘rationalist’ project in which values play only a minor (if any) role. The validation of an assessment proceeds in a ‘scientific’ manner, and the claim is that the results of any validation exercise would be agreed by all informed observers. Developing on the work of Samuel Messick, in this paper...

How can we effectively tell whether learners have acquired, and can exhibit outcomes that were initially established for them, and instructions tailored to? This question leads to assessment of learning outcomes or instructional results. It is for instance held that an outcomes-based approach to this requires assessment, in authentic ways, of what is considered to be most important of students' attainments. Unfortunately, the use of inappropriate assessment/test items/instruments is a widespread phenomenon and has become a practice/malpractice most urgently in need of improvement. To ensure such improvement is to satisfy the most important criteria in assessment/test administration; validity. The prevailing assessment culture is however still steeped in the preoccupation with reliability. This is due to the notion that for an assessment to be reliable it must first be valid, and the subsequent assumption that the reliability of an assessment invariably ensures its validity, as there is no structured/formulaic way of determining validity. It is however known that an assessment can be reliable without necessarily being valid. This paper therefore attempts to fill this validity void, by presenting two well-structured models/flowcharts; one, for verifying the validity or usefulness/appropriateness of assessment items and the other for the construction/writing of valid/appropriate assessment items.

Measurement: Interdisciplinary Research & Perspective, 2003

2001). The committee issuing this report was charged with synthesizing advances in the cognitive sciences and measurement, and exploring their implications for improving educational assessment. The article opens with a vision for the future of educational assessment that represents a significant departure from the types of assessments typically available today, and from the ways in which such assessments are most commonly used. This vision is driven by an interpretation of what is both necessary and possible for educational assessment to positively impact student achievement. The argument is made that realizing this vision requires a fundamental rethinking of the foundations and principles guiding assessment design and use. These foundations and principles and their implications are then summarized in the remainder of the article. The argument is made that every assessment, regardless of its purpose, rests on three pillars: (1) a model of how students represent knowledge and develop competence in the subject domain, (2) tasks or situations that allow one to observe students' performance, and (3) interpretation methods for drawing inferences from the performance evidence collected. These three elements-cognition, observation, and interpretation-must be explicitly connected and designed as a coordinated whole. Section II summarizes research and theory on thinking and learning which should serve as the source of the cognition element of the assessment triangle. This large body of research suggests aspects of student achievement that one would want to make inferences about, and the types of observations, or tasks, that will provide evidence to support those inferences. Also described are significant advances in methods of educational measurement that make new approaches to assessment feasible. The argument is presented that measurement models, which are statistical exam-ples of the interpretation element of the assessment triangle, are cuuently available to support the kinds of inferences about student achievement that cognitive science suggests are important to pursue. Section III describes how the contemporary understanding of cognition and methods of measurement jointly provide a set of principles and methods for guiding the processes of assessment design and use. This section explores how the scientific foundations presented in Section II play out in the design of real assessment situations ranging from classroom to large-scale testing contexts. It also considers the role of technology in enhancing assessment design and use. Section IV presents a discussion of the research, development, policy, and practice issues that must be addressed for the field of assessment to move forward and achieve the vision described in Section I.