e-Grammar: An Assessment-based Learning Environment for English Grammar
... Gucker, P.(1966). Essential English Grammar. NJ: Dover, Publications. New York and London: Do... more ... Gucker, P.(1966). Essential English Grammar. NJ: Dover, Publications. New York and London: Dover Publications. Johnson, WL, Rickel, JW, & Lester, JC (2000). ... New York: Academic Press.Zapata-Rivera, JD, & Greer, J. (2004) Interacting with Bayesian Student Models. ...
In this paper, we focus on a technology enhanced prototype formative assessment task from the mat... more In this paper, we focus on a technology enhanced prototype formative assessment task from the mathematics stream of the ETS research initiative Cognitively Based Assessment of, for, and as Learning (CBAL™;). The task, called Proportional Punch, is part of a family of assessments designed to capture evidence of students' understanding of, and skills in, proportional reasoning.
The current pace of technological advance has provided an unprecedented opportunity to use innova... more The current pace of technological advance has provided an unprecedented opportunity to use innovative simulated tasks in computerized assessment. A primary challenge for the successful use of innovation in assessment rests with the application of sound principles of design to produce a valid assessment. An additional challenge is to maximize the utility from the investment in innovative design through leveraging successful innovation to new assessment tasks and new educational tools. This paper describes the Evidence Centered Design (ECD)approach to design of an innovative simulation-based assessment of computer networking ability. The paper emphasizes the design components and how these components may be leveraged for reusability in a variety of ways, including the generation of new assessment tasks, applications to alternative purposes within the domain of computer networking, or as a basis for extending knowledge of proficiencies needed for performance in the domain, and the exte...
One of the main problems facing U.S. education is the growing educational achievement gap (Kirsch... more One of the main problems facing U.S. education is the growing educational achievement gap (Kirsch, Braun, Yamamoto, & Sum, 2007). Snow and Biancarosa (2003) have argued that this gap is largely due to differential language proficiencies. Although there is a strong interest in helping English language learners achieve their language-learning goals, these students are under great pressure to improve not only their language skills, but also competency in content areas (e.g., math). While there are English language learning (ELL) computer-based learning tools that help students improve their English skills, the tools usually focus on informal vocabulary and day-to-day situations. We have developed a game-inspired assessment and learning environment aimed at supporting vocabulary learning in academic subject areas (BELLA/EM-ABLE). BELLA/EM-ABLE focuses on Tier II vocabulary, the language students need to enable them to learn more content-specific material from teachers and textbooks. Tie...
How To Create Complex Measurement Models: A Case Study of Principled Assessment Design
In computer-based simulations, students must bring a wide range of relevant knowledge, skills, an... more In computer-based simulations, students must bring a wide range of relevant knowledge, skills, and abilities to bear jointly as they solve meaningful problems in a learning domain. To function effectively as an assessment, a simulation system must additionally be able to evoke and interpret observable evidence about targeted knowledge in a manner that is principled, defensible, and suited to the purpose at hand (e.g., licensure, achievement testing, coached practice). This study focused on the grounding for a simulation-based assessment of design and troubleshooting in the domain of computer networks. The application was designed as a prototype for assessing these skills in an instructional program, as interim practice tests and as chapter or end-of-course assessments. An evidence-centered assessment design framework was used to guide the work. An important part of this work is a cognitive task analysis, designed to tap the knowledge network engineers and students use when they desi...
Assessment-based learning environments (ABLEs) make use of assessment information coming from a v... more Assessment-based learning environments (ABLEs) make use of assessment information coming from a variety of sources (e.g., formative and summative assessments) to guide instruction. We have developed English ABLE, an assessment-based learning environment designed to help English language learners (ELLs) learn about English grammar. Key features of English ABLE include item/task reuse (900 enhanced Test of English as a Foreign Language™ [TOEFL®] items were recalibrated based on data from native Spanish speakers who have taken the test) and a Bayesian psychometric student model that makes use of item statistics, adaptive feedback, adaptive sequencing of tasks, pedagogical agents (i.e., Dr. Grammar, Jorge, and Carmen), and an indirectly visible student model. This paper describes main issues regarding the design and implementation of English ABLE including findings from a literature review, results from a preliminary market scan, and initial evaluations by domain experts, and the use of...
Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215), 1998
Digital Technology is constantly improving as information becomes wireless. These advances demand... more Digital Technology is constantly improving as information becomes wireless. These advances demand more wearable and mobile form factors for products that access information. A product that is wearable should have wearability. This paper explores the concept of dynamic wearability through design research. Wearability is defined as the interaction between the human body and the wearable object.
This paper describes the role developmental models can play in linking formative and summative as... more This paper describes the role developmental models can play in linking formative and summative assessment. We explore this issue in the context of a larger assessment design and pilot effort that is focused on middle school mathematics and English language arts, called Cognitively Based Assessment of, for and as Learning (CBAL) (Bennett & Gitomer, in press). The overall structure consists of a summative assessment system, and a formative assessment system accompanied by professional support. Research-based competency models provide a foundation for the design overall. In this context, we describe a developmental model of proportional reasoning, based upon Baxter and Junker's model in Weaver and Junker (2004) and how it was integrated into the overall design of the mathematics assessment. The developmental model helped improve the validity argument of the summative assessment by enabling us to define principled levels of competency. Because the developmental model contains resea...
This paper describes the characteristics of games and how they can be applied to the design of in... more This paper describes the characteristics of games and how they can be applied to the design of innovative assessment tasks for formative and summative purposes. Examples of current educational games and game-like assessment tasks in mathematics, science, and English language learning are used to illustrate some of these concepts. We argue that the inclusion of some aspects from gaming technology may have a positive effect in the development of innovative assessment systems (e.g., by supporting the development of highly engaging assessment tasks). However, integrating game elements as part of assessment tasks is a complex process that needs to take into account not only the engaging or motivational aspects of the activity but also the quality criteria that are needed according to the type of assessment that is being developed. 1992). We argue that to achieve these particular goals, several conditions need to exist: (a) students need to be motivated to learn, (b) students need to be provided with appropriate learning opportunities, and (c) an assessment system that provides educational stakeholders with information needed to keep the education system functioning appropriately should be in place. This assessment system should be
In computer-based interactive environments meant to support learning, students must bring a wide ... more In computer-based interactive environments meant to support learning, students must bring a wide range of relevant knowledge, skills, and abilities to bear jointly as they solve meaningful problems in a learning domain. To function effectively as an assessment, a computer system must additionally be able to evoke and interpret observable evidence about targeted knowledge in a manner that is principled, defensible, and suited to the purpose at hand (e.g., licensure, achievement testing, coached practice). This paper concerns the grounding for the design of an interactive computer-based assessment of design and troubleshooting in the domain of computer networking. The application is a prototype for assessing these skills as part of an instructional program, as interim practice tests and as chapter or end-of-course assessments. An Evidence Centered Design (ECD) framework was used to guide the work. An important part of this work is a cognitive task analysis designed to (a) tap the knowledge computer network specialists and students use when they design and troubleshoot networks, and (b) elicit behaviors that manifest this knowledge. After summarizing its results, we discuss implications of this analysis, as well as information gathered through other methods of domain analysis, for designing psychometric models, automated scoring algorithms, and task frameworks, and for the capabilities required for the delivery of this example of a complex computer-based interactive assessment.
The use of the Internet for course preparation is ill served by traditional, content-based search... more The use of the Internet for course preparation is ill served by traditional, content-based search engines. This paper describes SourceFinder, a web search engine that locates text material based on linguistic characteristics, such as reading level. Combining SourceFinder with content-based searches may allow instructors more easily to identify material relevant for their courses. We present examples from the teaching of reading comprehension, history, and statistics of how SourceFinder might aid instructors' use of the Internet for course preparation.
In this paper we describeWitt, a computational model of categorization and conceptual clustering ... more In this paper we describeWitt, a computational model of categorization and conceptual clustering that has been motivated and guided by research on human categorization. Properties of categories to which humans are sensitive include best or prototypical members, relative contrasts between categories, and polymorphy (neither necessary nor sufficient feature rules). The system uses pairwise feature correlations to determine the “similarity” between
e-Grammar: An Assessment-based Learning Environment for English Grammar
... Gucker, P.(1966). Essential English Grammar. NJ: Dover, Publications. New York and London: Do... more ... Gucker, P.(1966). Essential English Grammar. NJ: Dover, Publications. New York and London: Dover Publications. Johnson, WL, Rickel, JW, & Lester, JC (2000). ... New York: Academic Press.Zapata-Rivera, JD, & Greer, J. (2004) Interacting with Bayesian Student Models. ...
In this paper, we focus on a technology enhanced prototype formative assessment task from the mat... more In this paper, we focus on a technology enhanced prototype formative assessment task from the mathematics stream of the ETS research initiative Cognitively Based Assessment of, for, and as Learning (CBAL™;). The task, called Proportional Punch, is part of a family of assessments designed to capture evidence of students' understanding of, and skills in, proportional reasoning.
The current pace of technological advance has provided an unprecedented opportunity to use innova... more The current pace of technological advance has provided an unprecedented opportunity to use innovative simulated tasks in computerized assessment. A primary challenge for the successful use of innovation in assessment rests with the application of sound principles of design to produce a valid assessment. An additional challenge is to maximize the utility from the investment in innovative design through leveraging successful innovation to new assessment tasks and new educational tools. This paper describes the Evidence Centered Design (ECD)approach to design of an innovative simulation-based assessment of computer networking ability. The paper emphasizes the design components and how these components may be leveraged for reusability in a variety of ways, including the generation of new assessment tasks, applications to alternative purposes within the domain of computer networking, or as a basis for extending knowledge of proficiencies needed for performance in the domain, and the exte...
One of the main problems facing U.S. education is the growing educational achievement gap (Kirsch... more One of the main problems facing U.S. education is the growing educational achievement gap (Kirsch, Braun, Yamamoto, & Sum, 2007). Snow and Biancarosa (2003) have argued that this gap is largely due to differential language proficiencies. Although there is a strong interest in helping English language learners achieve their language-learning goals, these students are under great pressure to improve not only their language skills, but also competency in content areas (e.g., math). While there are English language learning (ELL) computer-based learning tools that help students improve their English skills, the tools usually focus on informal vocabulary and day-to-day situations. We have developed a game-inspired assessment and learning environment aimed at supporting vocabulary learning in academic subject areas (BELLA/EM-ABLE). BELLA/EM-ABLE focuses on Tier II vocabulary, the language students need to enable them to learn more content-specific material from teachers and textbooks. Tie...
How To Create Complex Measurement Models: A Case Study of Principled Assessment Design
In computer-based simulations, students must bring a wide range of relevant knowledge, skills, an... more In computer-based simulations, students must bring a wide range of relevant knowledge, skills, and abilities to bear jointly as they solve meaningful problems in a learning domain. To function effectively as an assessment, a simulation system must additionally be able to evoke and interpret observable evidence about targeted knowledge in a manner that is principled, defensible, and suited to the purpose at hand (e.g., licensure, achievement testing, coached practice). This study focused on the grounding for a simulation-based assessment of design and troubleshooting in the domain of computer networks. The application was designed as a prototype for assessing these skills in an instructional program, as interim practice tests and as chapter or end-of-course assessments. An evidence-centered assessment design framework was used to guide the work. An important part of this work is a cognitive task analysis, designed to tap the knowledge network engineers and students use when they desi...
Assessment-based learning environments (ABLEs) make use of assessment information coming from a v... more Assessment-based learning environments (ABLEs) make use of assessment information coming from a variety of sources (e.g., formative and summative assessments) to guide instruction. We have developed English ABLE, an assessment-based learning environment designed to help English language learners (ELLs) learn about English grammar. Key features of English ABLE include item/task reuse (900 enhanced Test of English as a Foreign Language™ [TOEFL®] items were recalibrated based on data from native Spanish speakers who have taken the test) and a Bayesian psychometric student model that makes use of item statistics, adaptive feedback, adaptive sequencing of tasks, pedagogical agents (i.e., Dr. Grammar, Jorge, and Carmen), and an indirectly visible student model. This paper describes main issues regarding the design and implementation of English ABLE including findings from a literature review, results from a preliminary market scan, and initial evaluations by domain experts, and the use of...
Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215), 1998
Digital Technology is constantly improving as information becomes wireless. These advances demand... more Digital Technology is constantly improving as information becomes wireless. These advances demand more wearable and mobile form factors for products that access information. A product that is wearable should have wearability. This paper explores the concept of dynamic wearability through design research. Wearability is defined as the interaction between the human body and the wearable object.
This paper describes the role developmental models can play in linking formative and summative as... more This paper describes the role developmental models can play in linking formative and summative assessment. We explore this issue in the context of a larger assessment design and pilot effort that is focused on middle school mathematics and English language arts, called Cognitively Based Assessment of, for and as Learning (CBAL) (Bennett & Gitomer, in press). The overall structure consists of a summative assessment system, and a formative assessment system accompanied by professional support. Research-based competency models provide a foundation for the design overall. In this context, we describe a developmental model of proportional reasoning, based upon Baxter and Junker's model in Weaver and Junker (2004) and how it was integrated into the overall design of the mathematics assessment. The developmental model helped improve the validity argument of the summative assessment by enabling us to define principled levels of competency. Because the developmental model contains resea...
This paper describes the characteristics of games and how they can be applied to the design of in... more This paper describes the characteristics of games and how they can be applied to the design of innovative assessment tasks for formative and summative purposes. Examples of current educational games and game-like assessment tasks in mathematics, science, and English language learning are used to illustrate some of these concepts. We argue that the inclusion of some aspects from gaming technology may have a positive effect in the development of innovative assessment systems (e.g., by supporting the development of highly engaging assessment tasks). However, integrating game elements as part of assessment tasks is a complex process that needs to take into account not only the engaging or motivational aspects of the activity but also the quality criteria that are needed according to the type of assessment that is being developed. 1992). We argue that to achieve these particular goals, several conditions need to exist: (a) students need to be motivated to learn, (b) students need to be provided with appropriate learning opportunities, and (c) an assessment system that provides educational stakeholders with information needed to keep the education system functioning appropriately should be in place. This assessment system should be
In computer-based interactive environments meant to support learning, students must bring a wide ... more In computer-based interactive environments meant to support learning, students must bring a wide range of relevant knowledge, skills, and abilities to bear jointly as they solve meaningful problems in a learning domain. To function effectively as an assessment, a computer system must additionally be able to evoke and interpret observable evidence about targeted knowledge in a manner that is principled, defensible, and suited to the purpose at hand (e.g., licensure, achievement testing, coached practice). This paper concerns the grounding for the design of an interactive computer-based assessment of design and troubleshooting in the domain of computer networking. The application is a prototype for assessing these skills as part of an instructional program, as interim practice tests and as chapter or end-of-course assessments. An Evidence Centered Design (ECD) framework was used to guide the work. An important part of this work is a cognitive task analysis designed to (a) tap the knowledge computer network specialists and students use when they design and troubleshoot networks, and (b) elicit behaviors that manifest this knowledge. After summarizing its results, we discuss implications of this analysis, as well as information gathered through other methods of domain analysis, for designing psychometric models, automated scoring algorithms, and task frameworks, and for the capabilities required for the delivery of this example of a complex computer-based interactive assessment.
The use of the Internet for course preparation is ill served by traditional, content-based search... more The use of the Internet for course preparation is ill served by traditional, content-based search engines. This paper describes SourceFinder, a web search engine that locates text material based on linguistic characteristics, such as reading level. Combining SourceFinder with content-based searches may allow instructors more easily to identify material relevant for their courses. We present examples from the teaching of reading comprehension, history, and statistics of how SourceFinder might aid instructors' use of the Internet for course preparation.
In this paper we describeWitt, a computational model of categorization and conceptual clustering ... more In this paper we describeWitt, a computational model of categorization and conceptual clustering that has been motivated and guided by research on human categorization. Properties of categories to which humans are sensitive include best or prototypical members, relative contrasts between categories, and polymorphy (neither necessary nor sufficient feature rules). The system uses pairwise feature correlations to determine the “similarity” between
Uploads
Papers by Malcolm Bauer