Critical thinking includes the component skills of analyzing arguments, making inferences using i... more Critical thinking includes the component skills of analyzing arguments, making inferences using inductive or deductive reasoning, judging or evaluating, and making decisions or solving problems. Background knowledge is a necessary but not a sufficient condition for enabling critical thought within a given subject. Critical thinking involves both cognitive skills and dispositions. These dispositions, which can be seen as attitudes or habits of mind, include openand fair-mindedness, inquisitiveness, flexibility, a propensity to seek reason, a desire to be wellinformed, and a respect for and willingness to entertain diverse viewpoints. There are both general-and domain-specific aspects of critical thinking. Empirical research suggests that people begin developing critical thinking competencies at a very young age. Although adults often exhibit deficient reasoning, in theory all people can be taught to think critically. Instructors are urged to provide explicit instruction in critical t...
This report summarizes two empirical studies that address two related topics-rater halo and how m... more This report summarizes two empirical studies that address two related topics-rater halo and how much unique information is provided by multiple analytic scores. Specifically, we address the issue of whether unique information is provided by analytic scores assigned to student writing, beyond what is depicted by holistic scores, and to what degree multiple analytic scores assigned by a single rater display evidence of a halo effect. To that end, we analyze scored student responses to an expository writing prompt that were scored by six groups of raters-four groups assigned single analytic scores, one group assigned multiple analytic scores, and one group assigned holistic scores-using structural equation modeling. Our results suggest that there is evidence of a halo effect when raters assign multiple analytic scores to a single student response and that, at best, only two factors seem to be distinguishable in analytic writing scores assigned to expository essays.
... Research Report Emily R. Lai June 2011 Page 2. ... For example, a number of researchers have ... more ... Research Report Emily R. Lai June 2011 Page 2. ... For example, a number of researchers have linked collaborative learning to the development of critical thinking (Bailin et al., 1999; Bonk & Smith, 1998; Heyman, 2008; Nelson, 1994; Paul, 1992; Thayer-Bacon, 2000). ...
This paper synthesizes research evidence pertaining to several so-called 21st century skills: cri... more This paper synthesizes research evidence pertaining to several so-called 21st century skills: critical thinking, creativity, collaboration, metacognition, and motivation. We provide a rationale for focusing on these five skills and explain their importance as educational outcomes. We then summarize peer-reviewed research published in education and psychology in order to answer several questions about these skills: (1) how do researchers define them; (2) how are they related to one another both theoretically and empirically; and (3) how do researchers traditionally measure them. We use the answers to these questions to make several recommendations regarding how best to assess these skills. Briefly, research suggests these skills are inter-related in complex ways. Researchers have used a number of approaches to measuring these skills, including (1) self-reports, (2) global rating scales, (3) standardized assessments, both multiplechoice and performance-based, and (4) observational mea...
Metacognition: A Literature Review Research Report Emily R. Lai April 2011 ... mindedness, inquis... more Metacognition: A Literature Review Research Report Emily R. Lai April 2011 ... mindedness, inquisitiveness, flexibility, a propensity to seek reason, a desire to be well-informed, and a respect for and willingness to entertain diverse viewpoints (Bailin et al., 1999; Ennis, 1985; ...
where he developed simulations and educational games to assess and support twenty-first-century s... more where he developed simulations and educational games to assess and support twenty-first-century skills. Matthew received his Ph.D.
This paper will share the author’s experiences working with a large and diverse school district t... more This paper will share the author’s experiences working with a large and diverse school district to design curriculum-embedded, performance-based assessments (PBAs) that measure 21 century skills in K-5 students. These assessments are designed to integrate seamlessly with classroom instruction; measure skills such as critical thinking, creativity, and collaboration in a developmentally-appropriate way; and provide information useful for supporting classroom teaching and learning. I begin by describing the context for the partnership. Next, I outline general principles that support the construction of the PBAs and explain the task development process. I also explore challenges in the development and validation of such assessments. Finally, I close the paper by speculating about future trends in PBA.
Critical thinking includes the component skills of analyzing arguments, making inferences using i... more Critical thinking includes the component skills of analyzing arguments, making inferences using inductive or deductive reasoning, judging or evaluating, and making decisions or solving problems. Background knowledge is a necessary but not a sufficient condition for enabling critical thought within a given subject. Critical thinking involves both cognitive skills and dispositions. These dispositions, which can be seen as attitudes or habits of mind, include openand fair-mindedness, inquisitiveness, flexibility, a propensity to seek reason, a desire to be wellinformed, and a respect for and willingness to entertain diverse viewpoints. There are both generaland domain-specific aspects of critical thinking. Empirical research suggests that people begin developing critical thinking competencies at a very young age. Although adults often exhibit deficient reasoning, in theory all people can be taught to think critically. Instructors are urged to provide explicit instruction in critical th...
Our field has devised many terms to describe assessments in which examinees demonstrate some type... more Our field has devised many terms to describe assessments in which examinees demonstrate some type of performance or create some type of product (e.g., performance, performancebased, “authentic,” constructed response, open-ended). Whatever you call them, performance-based assessments (PBAs) have a long history in educational measurement with cycles of ups and downs. And once again, PBAs are currently in vogue. Why? To address the federal government’s requirements for assessment systems that represent “the full performance continuum,” the two consortia formed in response to Race to the Top funding have both publicized assessment plans that involve a heavy dose of performance-based tasks (PARCC, 2010; SBAC, 2010). Thus, PBAs are relevant to any discussion about the future of testing in America.
This paper synthesizes research evidence pertaining to several so-called 21st century skills: cri... more This paper synthesizes research evidence pertaining to several so-called 21st century skills: critical thinking, creativity, collaboration, metacognition, and motivation. We provide a rationale for focusing on these five skills and explain their importance as educational outcomes. We then summarize peer-reviewed research published in education and psychology in order to answer several questions about these skills: (1) how do researchers define them; (2) how are they related to one another both theoretically and empirically; and (3) how do researchers traditionally measure them. We use the answers to these questions to make several recommendations regarding how best to assess these skills. Briefly, research suggests these skills are interrelated in complex ways. Researchers have used a number of approaches to measuring these skills, including (1) self-reports, (2) global rating scales, (3) standardized assessments, both multiplechoice and performance-based, and (4) observational measures. We recommend several practices for assessing 21st century skills: incorporating multiple measures to permit triangulation of inferences; designing complex and/or challenging tasks; including open-ended and/or illstructured tasks; using tasks that employ meaningful or authentic, real-world problem contexts; making student thinking and reasoning visible; and exploring innovative approaches that utilize new technology and psychometric models.
Metacognition is defined most simply as “thinking about thinking.” Metacognition consists of two ... more Metacognition is defined most simply as “thinking about thinking.” Metacognition consists of two components: knowledge and regulation. Metacognitive knowledge includes knowledge about oneself as a learner and the factors that might impact performance, knowledge about strategies, and knowledge about when and why to use strategies. Metacognitive regulation is the monitoring of one’s cognition and includes planning activities, awareness of comprehension and task performance, and evaluation of the efficacy of monitoring processes and strategies. Recent research suggests that young children are capable of rudimentary forms of metacognitive thought, particularly after the age of 3. Although individual developmental models vary, most postulate massive improvements in metacognition during the first 6 years of life. Metacognition also improves with appropriate instruction, with empirical evidence supporting the notion that students can be taught to reflect on their own thinking. Assessment o...
leading on research concerning instruction design, professional development of instructors and ef... more leading on research concerning instruction design, professional development of instructors and effective pedagogy. Her passion is to put research into teaching practice and teaching practice into research. She previously worked as a mathematics teacher in the UK, researcher at the University of Oxford, academic at the Open University and education consultant. She currently works as an international education consultant advising institutions, companies, governments and their agencies on using and implementing effective pedagogies and instruction design at scale. She holds a Ph.D. in mathematics education, a general degree majoring in social sciences, a mathematics degree and a postgraduate teaching qualification, all gained through the Open University in the UK.
Pearson. She leads a team of researchers focused on conducting and translating research about lea... more Pearson. She leads a team of researchers focused on conducting and translating research about learners and learning in order to influence the development of curricula and digital tools. Her personal research program centers on interactive technologies, particularly the use of evidence from learner activity in games and simulations, to understand what learners know and can do. Prior to joining Pearson, Kristen provided research support to the networking academies at Cisco and was a school psychologist in a local school district in Arizona. Kristen received her master's degree and Ph.D.
Critical Thinking: A Literature Review Research Report Emily R. Lai June 2011 Page 2. ... thought... more Critical Thinking: A Literature Review Research Report Emily R. Lai June 2011 Page 2. ... thought. For example, Bailin (2002) defines critical thinking as thinking of a particular quality essentially good thinking that meets specified criteria or standards of adequacy and accuracy. ...
Although performance assessments are not new, recent initiatives have proposed to use performance... more Although performance assessments are not new, recent initiatives have proposed to use performance tasks in ambitious new ways, including monitoring student growth and evaluating teacher effectiveness. Given these innovative proposals, it is important to pursue a research agenda for systematically investigating (1) the validity of proposed uses and interpretations of test results and (2) the reliability of scores and decisions. First, we discuss dimensions of test score use that are important to consider when planning a validity research agenda. Next, we offer a framework for collecting and organizing validity evidence over time, which includes five important sources of validity evidence: test content, examinee response processes, internal test structure, external relationships, and consequences of test use. We then discuss scoring issues and the reliability of performance assessment results. Finally, we explore how disparate sources of evidence can be integrated into an overall vali...
Critical thinking includes the component skills of analyzing arguments, making inferences using i... more Critical thinking includes the component skills of analyzing arguments, making inferences using inductive or deductive reasoning, judging or evaluating, and making decisions or solving problems. Background knowledge is a necessary but not a sufficient condition for enabling critical thought within a given subject. Critical thinking involves both cognitive skills and dispositions. These dispositions, which can be seen as attitudes or habits of mind, include openand fair-mindedness, inquisitiveness, flexibility, a propensity to seek reason, a desire to be wellinformed, and a respect for and willingness to entertain diverse viewpoints. There are both general-and domain-specific aspects of critical thinking. Empirical research suggests that people begin developing critical thinking competencies at a very young age. Although adults often exhibit deficient reasoning, in theory all people can be taught to think critically. Instructors are urged to provide explicit instruction in critical t...
This report summarizes two empirical studies that address two related topics-rater halo and how m... more This report summarizes two empirical studies that address two related topics-rater halo and how much unique information is provided by multiple analytic scores. Specifically, we address the issue of whether unique information is provided by analytic scores assigned to student writing, beyond what is depicted by holistic scores, and to what degree multiple analytic scores assigned by a single rater display evidence of a halo effect. To that end, we analyze scored student responses to an expository writing prompt that were scored by six groups of raters-four groups assigned single analytic scores, one group assigned multiple analytic scores, and one group assigned holistic scores-using structural equation modeling. Our results suggest that there is evidence of a halo effect when raters assign multiple analytic scores to a single student response and that, at best, only two factors seem to be distinguishable in analytic writing scores assigned to expository essays.
... Research Report Emily R. Lai June 2011 Page 2. ... For example, a number of researchers have ... more ... Research Report Emily R. Lai June 2011 Page 2. ... For example, a number of researchers have linked collaborative learning to the development of critical thinking (Bailin et al., 1999; Bonk & Smith, 1998; Heyman, 2008; Nelson, 1994; Paul, 1992; Thayer-Bacon, 2000). ...
This paper synthesizes research evidence pertaining to several so-called 21st century skills: cri... more This paper synthesizes research evidence pertaining to several so-called 21st century skills: critical thinking, creativity, collaboration, metacognition, and motivation. We provide a rationale for focusing on these five skills and explain their importance as educational outcomes. We then summarize peer-reviewed research published in education and psychology in order to answer several questions about these skills: (1) how do researchers define them; (2) how are they related to one another both theoretically and empirically; and (3) how do researchers traditionally measure them. We use the answers to these questions to make several recommendations regarding how best to assess these skills. Briefly, research suggests these skills are inter-related in complex ways. Researchers have used a number of approaches to measuring these skills, including (1) self-reports, (2) global rating scales, (3) standardized assessments, both multiplechoice and performance-based, and (4) observational mea...
Metacognition: A Literature Review Research Report Emily R. Lai April 2011 ... mindedness, inquis... more Metacognition: A Literature Review Research Report Emily R. Lai April 2011 ... mindedness, inquisitiveness, flexibility, a propensity to seek reason, a desire to be well-informed, and a respect for and willingness to entertain diverse viewpoints (Bailin et al., 1999; Ennis, 1985; ...
where he developed simulations and educational games to assess and support twenty-first-century s... more where he developed simulations and educational games to assess and support twenty-first-century skills. Matthew received his Ph.D.
This paper will share the author’s experiences working with a large and diverse school district t... more This paper will share the author’s experiences working with a large and diverse school district to design curriculum-embedded, performance-based assessments (PBAs) that measure 21 century skills in K-5 students. These assessments are designed to integrate seamlessly with classroom instruction; measure skills such as critical thinking, creativity, and collaboration in a developmentally-appropriate way; and provide information useful for supporting classroom teaching and learning. I begin by describing the context for the partnership. Next, I outline general principles that support the construction of the PBAs and explain the task development process. I also explore challenges in the development and validation of such assessments. Finally, I close the paper by speculating about future trends in PBA.
Critical thinking includes the component skills of analyzing arguments, making inferences using i... more Critical thinking includes the component skills of analyzing arguments, making inferences using inductive or deductive reasoning, judging or evaluating, and making decisions or solving problems. Background knowledge is a necessary but not a sufficient condition for enabling critical thought within a given subject. Critical thinking involves both cognitive skills and dispositions. These dispositions, which can be seen as attitudes or habits of mind, include openand fair-mindedness, inquisitiveness, flexibility, a propensity to seek reason, a desire to be wellinformed, and a respect for and willingness to entertain diverse viewpoints. There are both generaland domain-specific aspects of critical thinking. Empirical research suggests that people begin developing critical thinking competencies at a very young age. Although adults often exhibit deficient reasoning, in theory all people can be taught to think critically. Instructors are urged to provide explicit instruction in critical th...
Our field has devised many terms to describe assessments in which examinees demonstrate some type... more Our field has devised many terms to describe assessments in which examinees demonstrate some type of performance or create some type of product (e.g., performance, performancebased, “authentic,” constructed response, open-ended). Whatever you call them, performance-based assessments (PBAs) have a long history in educational measurement with cycles of ups and downs. And once again, PBAs are currently in vogue. Why? To address the federal government’s requirements for assessment systems that represent “the full performance continuum,” the two consortia formed in response to Race to the Top funding have both publicized assessment plans that involve a heavy dose of performance-based tasks (PARCC, 2010; SBAC, 2010). Thus, PBAs are relevant to any discussion about the future of testing in America.
This paper synthesizes research evidence pertaining to several so-called 21st century skills: cri... more This paper synthesizes research evidence pertaining to several so-called 21st century skills: critical thinking, creativity, collaboration, metacognition, and motivation. We provide a rationale for focusing on these five skills and explain their importance as educational outcomes. We then summarize peer-reviewed research published in education and psychology in order to answer several questions about these skills: (1) how do researchers define them; (2) how are they related to one another both theoretically and empirically; and (3) how do researchers traditionally measure them. We use the answers to these questions to make several recommendations regarding how best to assess these skills. Briefly, research suggests these skills are interrelated in complex ways. Researchers have used a number of approaches to measuring these skills, including (1) self-reports, (2) global rating scales, (3) standardized assessments, both multiplechoice and performance-based, and (4) observational measures. We recommend several practices for assessing 21st century skills: incorporating multiple measures to permit triangulation of inferences; designing complex and/or challenging tasks; including open-ended and/or illstructured tasks; using tasks that employ meaningful or authentic, real-world problem contexts; making student thinking and reasoning visible; and exploring innovative approaches that utilize new technology and psychometric models.
Metacognition is defined most simply as “thinking about thinking.” Metacognition consists of two ... more Metacognition is defined most simply as “thinking about thinking.” Metacognition consists of two components: knowledge and regulation. Metacognitive knowledge includes knowledge about oneself as a learner and the factors that might impact performance, knowledge about strategies, and knowledge about when and why to use strategies. Metacognitive regulation is the monitoring of one’s cognition and includes planning activities, awareness of comprehension and task performance, and evaluation of the efficacy of monitoring processes and strategies. Recent research suggests that young children are capable of rudimentary forms of metacognitive thought, particularly after the age of 3. Although individual developmental models vary, most postulate massive improvements in metacognition during the first 6 years of life. Metacognition also improves with appropriate instruction, with empirical evidence supporting the notion that students can be taught to reflect on their own thinking. Assessment o...
leading on research concerning instruction design, professional development of instructors and ef... more leading on research concerning instruction design, professional development of instructors and effective pedagogy. Her passion is to put research into teaching practice and teaching practice into research. She previously worked as a mathematics teacher in the UK, researcher at the University of Oxford, academic at the Open University and education consultant. She currently works as an international education consultant advising institutions, companies, governments and their agencies on using and implementing effective pedagogies and instruction design at scale. She holds a Ph.D. in mathematics education, a general degree majoring in social sciences, a mathematics degree and a postgraduate teaching qualification, all gained through the Open University in the UK.
Pearson. She leads a team of researchers focused on conducting and translating research about lea... more Pearson. She leads a team of researchers focused on conducting and translating research about learners and learning in order to influence the development of curricula and digital tools. Her personal research program centers on interactive technologies, particularly the use of evidence from learner activity in games and simulations, to understand what learners know and can do. Prior to joining Pearson, Kristen provided research support to the networking academies at Cisco and was a school psychologist in a local school district in Arizona. Kristen received her master's degree and Ph.D.
Critical Thinking: A Literature Review Research Report Emily R. Lai June 2011 Page 2. ... thought... more Critical Thinking: A Literature Review Research Report Emily R. Lai June 2011 Page 2. ... thought. For example, Bailin (2002) defines critical thinking as thinking of a particular quality essentially good thinking that meets specified criteria or standards of adequacy and accuracy. ...
Although performance assessments are not new, recent initiatives have proposed to use performance... more Although performance assessments are not new, recent initiatives have proposed to use performance tasks in ambitious new ways, including monitoring student growth and evaluating teacher effectiveness. Given these innovative proposals, it is important to pursue a research agenda for systematically investigating (1) the validity of proposed uses and interpretations of test results and (2) the reliability of scores and decisions. First, we discuss dimensions of test score use that are important to consider when planning a validity research agenda. Next, we offer a framework for collecting and organizing validity evidence over time, which includes five important sources of validity evidence: test content, examinee response processes, internal test structure, external relationships, and consequences of test use. We then discuss scoring issues and the reliability of performance assessment results. Finally, we explore how disparate sources of evidence can be integrated into an overall vali...
Uploads
Papers by Emily Lai