On the Structure of Educational Assessments

Measurement: Interdisciplinary Research & Perspective, 2003

2001). The committee issuing this report was charged with synthesizing advances in the cognitive sciences and measurement, and exploring their implications for improving educational assessment. The article opens with a vision for the future of educational assessment that represents a significant departure from the types of assessments typically available today, and from the ways in which such assessments are most commonly used. This vision is driven by an interpretation of what is both necessary and possible for educational assessment to positively impact student achievement. The argument is made that realizing this vision requires a fundamental rethinking of the foundations and principles guiding assessment design and use. These foundations and principles and their implications are then summarized in the remainder of the article. The argument is made that every assessment, regardless of its purpose, rests on three pillars: (1) a model of how students represent knowledge and develop competence in the subject domain, (2) tasks or situations that allow one to observe students' performance, and (3) interpretation methods for drawing inferences from the performance evidence collected. These three elements-cognition, observation, and interpretation-must be explicitly connected and designed as a coordinated whole. Section II summarizes research and theory on thinking and learning which should serve as the source of the cognition element of the assessment triangle. This large body of research suggests aspects of student achievement that one would want to make inferences about, and the types of observations, or tasks, that will provide evidence to support those inferences. Also described are significant advances in methods of educational measurement that make new approaches to assessment feasible. The argument is presented that measurement models, which are statistical exam-ples of the interpretation element of the assessment triangle, are cuuently available to support the kinds of inferences about student achievement that cognitive science suggests are important to pursue. Section III describes how the contemporary understanding of cognition and methods of measurement jointly provide a set of principles and methods for guiding the processes of assessment design and use. This section explores how the scientific foundations presented in Section II play out in the design of real assessment situations ranging from classroom to large-scale testing contexts. It also considers the role of technology in enhancing assessment design and use. Section IV presents a discussion of the research, development, policy, and practice issues that must be addressed for the field of assessment to move forward and achieve the vision described in Section I.

PsycEXTRA Dataset

1992

The model presented in this document integrates authentic assessment with traditional evaluation practices to create comprehensive assessment systems for elementary and secondary school students. Specifically, the model poses six categories of competencies that enjoy high levels of acceptance and construct validity: (1) knowledge of concepts, generalizations, processes, and strategies considered critical to specific content areas; (2) the ability to use complex reasoning processes; (3) the ability to gather and utilize information from a variety of sources in a variety of modes; (4) the ability to communicate effectively; (5) the ability to regulate one's own learning and development; and (6) the ability to work in a cooperative/collaborative manner. These competencies are meant to be assessed in three basic ways: through multiple validations, through secured tasks, and through portfolios. Implicit in this model is the identification of world class standards within content areas identified as important at the local, state, or national level. Six appendices, which comprise most of the document, provide definitions of 14 complex reasoning processes; subcompetencies of the 14 complex reasoning processes; benchmarks demonstrating student abilities in content areas; authentic classroom tasks; generalized rubrics for declarative knowledge; and elementary and secondary school tasks. (Contains approximately 85 references.) (LL)

" There are three sides to every story—your side, my side, and the truth. " —JOHN ADAMS The goal of assessment is to collect objective evidence that represents the truth about student performance. In order to assure objectivity the assessment plan must be well grounded in the principles of assessment. The first step in developing an objective assessment plan is to become familiar with the terminology of assessment to facilitate your understanding of the bigger picture. The purpose of this chapter is to review the basic terminology and principles of assessment and provide you with a basic understanding of the framework on which to base an objective and comprehensive systematic assessment plan. These concepts are discussed in detail in subsequent chapters. Many of you are familiar with these terms. Some readers may even prefer to move past this chapter and delve right into the strategies for developing assessment tools. However, as further discussion demonstrates, you cannot start collecting data until your assessment plan is established. Unless you consistently work in the area of assessment, you will find this refresher beneficial. Reviewing this chapter will increase your fluency in the Language of Assessment and your understanding of the proposed guidelines. Assessment Chapter 1, " The Role of Assessment in Instruction, " introduces you to the concept of assessment as the broad and comprehensive process of collecting quantitative and qualitative data to make informed educational decisions about students. It is a process that encompasses the full range of procedures used to obtain information about student

Applied Measurement in Education, 2019

Despite the call for an argument-based approach to validity over 25 years ago, few examples exist in the published literature. One possible explanation for this outcome is that the complexity of the argument-based approach makes implementation difficult. To counter this claim, we propose that the Assessment Triangle can serve as the overarching framework for operationalizing and instantiating the argument-based approach to validation. Integrating these frameworks can streamline the validation process by providing a conceptual lens for identifying, collecting, and evaluating relevant sources of evidence throughout the testing process. To fully examine this proposed conceptualization, we apply the integrated framework to an example case of a universal screener for middleschool mathematics. We articulate an interpretation and use argument for the universal screener, and then present relevant sources of evidence to evaluate the plausibility of the inferences and warrants underlying test score use. Based on this applied example, the strengths and limitations of the integrated framework are considered, and recommendations are made for future instantiations. Multiple frameworks exist in educational assessment for making sense of validity. Perhaps the most comprehensive treatment of validity within the broader context of educational assessment is the National Research Council's report titled Knowing What Students Know (Pellegrino, Chudowsky, & Glaser, 2001). In this report, validity (interpretation) is integrally connected to theories of learning (cognition) and students' response processes on tested items, tasks, or situations (observation) in the Assessment Triangle. Interpretation is defined as the process of making meaning from the evidence obtained about students' knowledge, skills, and abilities. Cognition refers to the models of thinking and learning that underlie students' development of knowledge, skills, and abilities in the domain. Observation includes the items, tasks, or situations designed to elicit students' knowledge, skills, and abilities. Representing these components as the Assessment Triangle underscores the interdependent relationship and importance of alignment between these dimensions of educational assessment. Although more than 25 years have passed since the introduction of this conceptual model of educational assessment, the three integrated components are often examined separately. However, some connections between the components have begun to receive greater attention in the past 15 years. For example, recent research in mathematics and science education has called for designing classroom assessments based on learning trajectories and learning progressions (cf., Corcoran, Mosher, & Rogat, 2009; Daro, Mosher, & Corcoran, 2011), especially as it relates to providing teachers with instructionally relevant information. This work seeks to connect the cognition and interpretation vertices of the Assessment Triangle. Relatedly, researchers investigating students' response processes when interacting with items, tasks, or situations are attempting to examine the alignment between the intended and elicited cognitive processes, thereby connecting the cognition and observation vertices (cf., Leighton & Gierl, 2011; Padilla & Benitez, 2014).

International Journal of Educational Research, 1990

2008

What is our warrant for saying “Student X deserves a Grade C ” ? It must be based on evidence, and the only evidence we see is what students produce during the exam. For valid assessment two criteria must be met: the examination must elicit proper evidence of the trait, and we must evaluate the evidence properly. This highlights the importance of ensuring quality in the mark schemes with which we evaluate the evidence as well as in the questions which elicit it. Our recent research shows that improving mark schemes can make more impact on validity than further work on improving questions. In this paper we will outline a procedural model for maximising construct validity: at its heart is the concept of Outcome Space, the range of evidence that students produce. The model aims to ensure that our mark schemes evaluate this evidence properly in terms of the achievement trait we want to assess. This model has been developed in consultation with senior examiners and exam board personnel. ...

education policy analysis archives, 1996

ABSTRACT We question the utility of traditional conceptualizations of validity and reliability, developed in the context of large scale, external testing, and the psychology of individual differences, for the context of the classroom. We compare traditional views of validity and reliability to alternate frameworks that situate these constructs in teachers' work in classrooms. We describe how we used these frameworks to design an assessment course for preservice teachers, and present data that suggest students in the redesigned course not only saw the course as more valuable in their work as teachers, but developed deeper understandings of validity and reliability than did their counterparts in a traditional tests and measurement course. We close by discussing the implications of these data for the teaching of assessment, and for the use and interpretation of classroom assessment data for purposes of local and state accountability.

2000

The reasons for carrying out educational assessments can be grouped under three broad headings:

2013