Skip to main content

Eva L Baker

Ucla, Education, Faculty Member

Followers

56

Following

8

Co-authors

2

Public Views

Professor and organization builder in assessment, Learning, evaluation and technology

less

University of Texas at El Paso (UTEP)

University of Melbourne

Kwantlen Polytechnic University

San Martin de Porres University

Royal School of Library and Information Science

Emiliano Treré

Cardiff University

Mahmoud A. Moussa

Washington Center for Consciousness Studies

Prof. Dr. Gökhan DUMAN

Gazi University

Stavroula Kaldi

UNIVERSITY OF THESSALY, GREECE

InterestsView All (7)

Uploads

Papers by Eva L Baker

Simulations and the Transfer of Problem-Solving Knowledge and Skills

An Experimental Appraisal of the Generalizability of Empirical Revision Procedures for Curriculum Materials

An experiment was conducted to determine if the use of rules to guide revisions has generalizable... more An experiment was conducted to determine if the use of rules to guide revisions has generalizable effects across programs and programers. The hypothesis was that pupil achievement would be greater for instructional programs revised according to specified rules than for unrevised programs. Each of 10 programers produced a self-instructional sequence for fifth graders. The 50-frame programs including pretest and 10-item posttests were to take approximately one hour to complete. After each was administered to two fifth graders the first draft programs with technical performance data were randomly distributed to the programers with a set of rules for use in revision. Then 64 fifth graders were randomly assigned to receive either a first draft or revised program. Scores were averaged for each condition, first draft or revision, for each program and the mean for each program per condition provided the unit of analysis. A t test based on the paired observation model was employed using the S's mean performance per condition, matched by program, as the entry. An omega square value to test the strength of association was computed. It was inferred from the results that empirical revision based on rules pays off modestly, i.e., instructions can be provided so that the efforts expanded in revision will have some effect on the obtained levels of pupil performance. Confidence limits for the difference were calculated so that cost of revision could be assessed. [Not available in hardcopy due to marginal legibility of original document.] (JS)

The Practicum: A Department in Development Training

A brief history of the UCLA Practicum for Educational Developers is given as are the goals of the... more A brief history of the UCLA Practicum for Educational Developers is given as are the goals of the practicum and some of the obstacles in its path. The practicum seeks to train developmentpersonnel in the context of real development tasks, requires proven competence, is a team effort, and is based on the assumption that novices need assistance and feedback. All projects are group produced and designed to help prepare technical and professional staff competent in instructional development. Areas of continuing concern are discussed. (RS) FILMED FROM BEST AVAILABLE COPY

Issues in Intelligent Computer-Assisted Instruction. Testing Study Group: The Impact of Advances in Artificial Intelligence on Test Development

Among opportunities to advance the state of the art of intelligent computer-assisted instruction ... more Among opportunities to advance the state of the art of intelligent computer-assisted instruction (ICAI) are the evaluation of ICAI systems and the use of the underlying technology in ICAI systems to develop tests. Each issue is addressed via its theoretical context, key constructs, appropriate references to the literature, methodological aspects, and c.Increte examples of the feasibility of resolving the issue. ICAI systems use artificial intelligence and cognitive science to reach a range of subject matters. Several computer programs are discussed. The key components of ICAI systems include a knowledge base, a student model, and instructional techniques for tea:hing declarative or procedural knowledge. Research that has contributed to the development of ICAI includes research into both formative and summative evaluation, measurement of student achievement outcomes, measurement of individual differences among students, and process measurement and analysis. A list of 75 references is presented. (TJH)

Exploring Career-Readiness Features in High School Test Items through Cognitive Laboratory Interviews. CRESST Report 857

National Center for Research on Evaluation, Standards, and Student Testing, Dec 1, 2018

The findings and opinions expressed here do not necessarily reflect the positions or policies of ... more

Improving Accountability Models by Using Technology-Enabled Knowledge Systems (TEKS). CSE Report 656

National Center for Research on Evaluation, Standards, and Student Testing, Jul 1, 2005

This paper addresses accountability and the role emerging technologies can play in improving educ... more This paper addresses accountability and the role emerging technologies can play in improving education. The connection between accountability and technology can be best summarized in one term: feedback. Technological supports provide ways of designing, collecting and sharing information in order to provide the basis for the improvement of systems and outcomes. Because technological options are so varied and the future of their development unpredictable, the paper argues that the first screen for the review of technology is one addressing core functions the technology serves. Technologies can provide greater efficiency in managing information or serve to improve the quality of the information used to guide the system. At this point, these functions are frequently mutually exclusive, but in the future, wise selection of technology should involve a consideration of how the technology adds value by increasing quality and efficiency in the enterprise. Quality is added when more knowledge is available to the user. For choices that need to be made about investments, a set of criteria is provided to guide decision logic. Criteria include validity, accuracy, and utility, with an emphasis on users directly involved in learning (students and teachers). In addition, criteria are reviewed related to software/hardware operation, including security, backup, compatibility, resilience, and transparency. The assumption is that systematic application of criteria will influence the market to drive development of more useful systems for education at a cost that can be withstood. Throughout, examples of technology uses now and soon to be available are discussed. The paper concludes with two extended educational scenarios constructed to show elements of design and technology in practice in two very different settings. This paper will analyze the potential use of technology-enabled knowledge systems (TEKS) as integral components in educational accountability systems. The relationship between accountability and knowledge is obvious, for accountability systems depend upon usable information (Baker, 2003) to support their operations. Technology functions, in part, to increase the availability and usability of information. Because there are so many different examples of technology-and more, unimaginable advances left to be developed-it is plainly not practical to give

Report of State Level Activities: Guidelines for the RFP Process and Selected Technical Issues in Large-Scale Assessment Programs. Report of State Level Activities. Project: Monitoring the Implementation of Testing and Evaluation Innovations

Guidelines for evaluating responses to requests for proposals (RFPs) for state-level educational ... more Guidelines for evaluating responses to requests for proposals (RFPs) for state-level educational assessment programs are presented. The guidelines articulate key choice points, options, and considerations by state (and district) testing directors as they solicit services for large-scale assessment from commercial testing companies. Focus is on broadening the scope of issues and concarns that state and district testing officers consider in preparing RFPs for large-scale testing programs. Components that must be considered in dealing with the RFP process include: (1) planning approaches; (2) communications with bidders; (3) the RFP structure; (4) the review process; and (5) issues associated with equating portions of RFPs, item bias portions of RFPs, and content validity portions of RFPs. (TJH)

Plan for the Assessment and Evaluation of Individual and Team Proficiencies Developed by the DARWARS Environments

*Data Collection; *Educational Assessment; Educational Innovation; *Formative Evaluation; *Information Utilization; Political Influences; Research

In November 2002, the Institute for the Advancement of Emerging Technologies in Education at AEL ... more In November 2002, the Institute for the Advancement of Emerging Technologies in Education at AEL gathered education practitioners to help shepherd new assessments from the research laboratory to the classroom. Approximately 60 teachers and administrators attended the conference. The discussion focused on the possibility of aggregating formative data to meet the purposes of large-scale tests and attempted to develop a technology-based method of collecting and analyzing student outcome data to support testing decisions. A panel of forum presented offered an overview of research, specific examples of attempts to create new forms of assessment, and opinions about how the current state of research and the political climate combine to create an extraordinary opportunity to redesign assessments and make them more useful. Panelists agreed that the new systems require sophistication on the part of the teacher. Teachers and administrators in attendance wanted tools that were immediately useable, and it was evident that such tools do not yet exist. The gap between researcher needs and practitioner functioning is frustrating, but frustration is tempered by the attraction of the possibilities of new technology approaches. Panelists were: (1) John Bailey; (2) Linda Roberts; (3) Jim Pellegrino; (4) Eva L. Baker; (5) Christopher Dede; and (6) Lajeane Thomas. (SLD) Reproductions supplied by EDRS are the best that can be made from the original document.

Simulation as a Performance Assessment Technique for the Interpersonal Skill of Negotiation. Report of Task Forces on Technology and Workforce Issues. Project 2.3: Enhancing the Utility of Performance Assessments: Domain-Independent R&D--Workforce

A prototype simulation measure of the negotiating skill of interpersonal competence is described,... more A prototype simulation measure of the negotiating skill of interpersonal competence is described, and results of an initial validation study are reported. Of eight negotiating skills identified by the Secretary's Commission on Achieving Necessary Skills of the U.S. Department of Labor, two were the focus of the measure: proposing and examining possible options and making reasonable compromises. A simulation was developed focusing on these skills in a job contract negotiation between a potential employee and an employer. The validity study of this measur, was based on an expert/novice criterion group approach, assuming that expert negotiators would receive better counter-offers than would novices. Ten experts were chosen from graduate students who had completed a course in negotiation in a Masters in Business Administration program and 21 novices were chosen from a high school business class. Results, generally supportive of the hypothesis, provide evidence of the validity of the simulation as a measure of negotiation skills. Both the assessment of options and the consideration of compromises are important components of negotiating skill. Eight tables of study results are included. (Contains 37 references.) (SLD)

The Effects of Manipulated Item Writing Constraints on the Homogeneity of Test Items. Center for the Study of Evaluation Reprint Series No. 11

The greatest curse, wise men have said, way be to have your wishes corn tree.

New Directions in Evaluation Research: Implications for Vocational Education. Occasional Paper No. 55

In the early 1970s educational evaluation's fusion of respect for rationality, power to iiplement... more In the early 1970s educational evaluation's fusion of respect for rationality, power to iiplement rational procedure, and an altruistic objective (improved instruction) appealed to educational researchers. Dealing essentially with closed systems, it measured program success by student performance on measurement 21struments. Critics exclaimed that program developers and evaluators ok away personal decision rights from students and that outcome measures were incomplete and inaccurate. Changes occured due to reanalysis showing the futility of earlier research, operating focus on procedures (not outcom6s), open student participation, and loss ol stability required for longitudinal study. In present-day open systeE evaluation settings, emphasis is on multiple objectives, and selection of what is to be evaluated is left open. No longer committed to methodology and provision of clear information, evaluators use case studies and looser, more interactive designs. Their reactions to political demands are to use needs assessments,. develop systematic procedures, and write more reports. Problems . between politics and evaluation include these: (1) the technical disagreement that Lny evaluati9a is subject to erodes its credibility with its.contracting agency,.(2) evaluation results can be used to discredit politicians whose olaims outstrip their programs, and (3) success of political efforts gives politicians a sense of personal power. (YLB)

Templates and Objects in Authoring Problem-Solving Assessments. CRESST Report 735

National Center for Research on Evaluation, Standards, and Student Testing, May 1, 2008

Assessing whether students can both re-present a corpus of learned knowledge and also demonstrate... more Assessing whether students can both re-present a corpus of learned knowledge and also demonstrate that they can apply that knowledge to solve problems is key to assessing student understanding. This notion, in turn, impacts our thinking about what we assess, how we author such assessments, and how we interpret assessment results. The diffusion of technology into venues of learning offers new opportunities in the area of student assessment. Specifically, computer-based simulations seem to provide sufficiently rich environments and the tools necessary to allow us to infer accurately how well a student's individual mental model of the world can accommodate, integrate, and be used to exploit concepts from a domain of interest. In this paper then, we first identify the characteristics of simulations that our experience suggests are necessary to make them appropriate for pedagogical and assessment purposes. Next, we discuss the models and frameworks (templates) we have used to ensure these characteristics are considered. Finally, we describe two computerized instantiations (objects) of these frameworks and implications for the follow-on design of simulations.

Templates and Objects in Authoring Problem-Solving Assessments

Routledge eBooks, Sep 25, 2017

Assessing whether students can both re-present a corpus of learned knowledge and also demonstrate... more Assessing whether students can both re-present a corpus of learned knowledge and also demonstrate that they can apply that knowledge to solve problems is key to assessing student understanding. This notion, in turn, impacts our thinking about what we assess, how we author such assessments, and how we interpret assessment results. The diffusion of technology into venues of learning offers new opportunities in the area of student assessment. Specifically, computer-based simulations seem to provide sufficiently rich environments and the tools necessary to allow us to infer accurately how well a student's individual mental model of the world can accommodate, integrate, and be used to exploit concepts from a domain of interest. In this paper then, we first identify the characteristics of simulations that our experience suggests are necessary to make them appropriate for pedagogical and assessment purposes. Next, we discuss the models and frameworks (templates) we have used to ensure these characteristics are considered. Finally, we describe two computerized instantiations (objects) of these frameworks and implications for the follow-on design of simulations.

The Achievement Gap in California: Context, Status, and Approaches for Improvement

Using Measurement to Improve Instruction

Instructional improvement within the context of criterion-referenced and norm-referenced tests is... more Instructional improvement within the context of criterion-referenced and norm-referenced tests is described. Such categories cveremphasize test interpretation rather than design characteristics of achievement tests. Data from most measurement situations may be reported or interpreted either according to criterion-or norm-eferenced standards. HOw the test is developed and what it represents is of critical importance. The paper proposes alternative conceptualizations of test design: construct-referenced, objectives-referenced and domain-referenced. Using student data, the teacher needs to identify deficiencies in achievement, possible explanations, and remedies and to put the remedies into operation. An analysis of the utility of each test type results in the appraisal that domain referenced tests provide the most information for teachers and therefore are the most desirable as data sources for instructional improvement. However, because of lack of knowledge about instruction, poor training in available instructional principles, and lack of resources to encourage changes in instructional habits, it is concluded that instructional improvement, even if measurement considerations were satisfied, is not imminent.

Sensitive Technology Assessment of ACOT

This paper explores the ideas and the model underlying the evaluation of the Apple Classroom of T... more This paper explores the ideas and the model underlying the evaluation of the Apple Classroom of Tomorrow project (ACOT), a 2-year-old research and development project incorporating at least seven different grade levels which is located in five different school sites in four states. The major features of ACOT are identified as the ideas of computer saturation and local site development. Of particular concern in this paper is the evaluation of the technology within the context of the usual goal-focused educational assessment. It is suggested that the assessment of the technology assists in the identification of appropriate goals through a process entitled technology push, which allows for outcomes to be empirically identified as desired consequences or the technology's implementation. A model for the assessment is provided that attempts to incorporate evaluation features central to the schools in which ACOT has been developed, i.e., educational goals, processes, and outcomes. Finally, specific evaluation questions are formulated for the comparison of students at different ACOT sites, and specific achievement measurements are suggested. The text is supplemented by two figures, and three references are provided. (EW)

What Probably Works in Alternative Assessment. CRESST Report 772

National Center for Research on Evaluation, Standards, and Student Testing, Jul 1, 2010

This report provides an overview of what was known about alternative assessment at the time that ... more This report provides an overview of what was known about alternative assessment at the time that the article was written in 1991. Topics include beliefs about assessment reform, overview of alternative assessment including research knowledge, evidence of assessment impact, and critical features of alternative assessment. The author notes that in the short term, alternative assessment will generate negative news about student learning and will require massive support to make it a successful reform strategy.

The Five Families of Cognitive Learning: A Context in Which To Conduct Cognitive Demands Analyses of Innovative Technologies

American Educational Research Association Annual Meeting, Mar 1, 1997

A cognitive demands analysis of a learning technology, a term that includes the hardware and the ... more A cognitive demands analysis of a learning technology, a term that includes the hardware and the computer software products that form learning environments, attempts to describe the types of cognitive learning expected of the individual by the technology. This paper explores the context of cognitive learning, suggesting five families of cognitive learning. These families are: (1) content understanding; (2) collaboration; (3) communication; (4) problem solving; and (5) metacognition. Three technologies are analyzed with reference to these types of learning. The first technology, Algebra Tutor, is an intelligent tutoring system to help students learn algebra skills. "Hands-On Universe" is an astronomical research environment for high school students. "Function Machines" is an application for a visual programming language designed for mathematics and science education. Analyses of the types of learning expected for each of these technologies have resulted in the development of a suite of performance tasks (an integrated simulation) that includes individual and collaborative concept mapping tasks, a problem-solving search task, an explanation task, and a metacognitive questionnaire. This assessment environment is integrated across grade levels and within content areas to blend real-world tasks with a project-based scenario that captures many types of cognitive learning. (Contains 1 table and 79 references.) (SLD)

Aligning Curriculum, Standards, and Assessments: Fulfilling the Promise of School Reform. CSE Report 645

Simulations and the Transfer of Problem-Solving Knowledge and Skills

An Experimental Appraisal of the Generalizability of Empirical Revision Procedures for Curriculum Materials

An experiment was conducted to determine if the use of rules to guide revisions has generalizable... more An experiment was conducted to determine if the use of rules to guide revisions has generalizable effects across programs and programers. The hypothesis was that pupil achievement would be greater for instructional programs revised according to specified rules than for unrevised programs. Each of 10 programers produced a self-instructional sequence for fifth graders. The 50-frame programs including pretest and 10-item posttests were to take approximately one hour to complete. After each was administered to two fifth graders the first draft programs with technical performance data were randomly distributed to the programers with a set of rules for use in revision. Then 64 fifth graders were randomly assigned to receive either a first draft or revised program. Scores were averaged for each condition, first draft or revision, for each program and the mean for each program per condition provided the unit of analysis. A t test based on the paired observation model was employed using the S's mean performance per condition, matched by program, as the entry. An omega square value to test the strength of association was computed. It was inferred from the results that empirical revision based on rules pays off modestly, i.e., instructions can be provided so that the efforts expanded in revision will have some effect on the obtained levels of pupil performance. Confidence limits for the difference were calculated so that cost of revision could be assessed. [Not available in hardcopy due to marginal legibility of original document.] (JS)

The Practicum: A Department in Development Training

A brief history of the UCLA Practicum for Educational Developers is given as are the goals of the... more A brief history of the UCLA Practicum for Educational Developers is given as are the goals of the practicum and some of the obstacles in its path. The practicum seeks to train developmentpersonnel in the context of real development tasks, requires proven competence, is a team effort, and is based on the assumption that novices need assistance and feedback. All projects are group produced and designed to help prepare technical and professional staff competent in instructional development. Areas of continuing concern are discussed. (RS) FILMED FROM BEST AVAILABLE COPY

Issues in Intelligent Computer-Assisted Instruction. Testing Study Group: The Impact of Advances in Artificial Intelligence on Test Development

Among opportunities to advance the state of the art of intelligent computer-assisted instruction ... more Among opportunities to advance the state of the art of intelligent computer-assisted instruction (ICAI) are the evaluation of ICAI systems and the use of the underlying technology in ICAI systems to develop tests. Each issue is addressed via its theoretical context, key constructs, appropriate references to the literature, methodological aspects, and c.Increte examples of the feasibility of resolving the issue. ICAI systems use artificial intelligence and cognitive science to reach a range of subject matters. Several computer programs are discussed. The key components of ICAI systems include a knowledge base, a student model, and instructional techniques for tea:hing declarative or procedural knowledge. Research that has contributed to the development of ICAI includes research into both formative and summative evaluation, measurement of student achievement outcomes, measurement of individual differences among students, and process measurement and analysis. A list of 75 references is presented. (TJH)

Exploring Career-Readiness Features in High School Test Items through Cognitive Laboratory Interviews. CRESST Report 857

National Center for Research on Evaluation, Standards, and Student Testing, Dec 1, 2018

The findings and opinions expressed here do not necessarily reflect the positions or policies of ... more

Improving Accountability Models by Using Technology-Enabled Knowledge Systems (TEKS). CSE Report 656

National Center for Research on Evaluation, Standards, and Student Testing, Jul 1, 2005

This paper addresses accountability and the role emerging technologies can play in improving educ... more This paper addresses accountability and the role emerging technologies can play in improving education. The connection between accountability and technology can be best summarized in one term: feedback. Technological supports provide ways of designing, collecting and sharing information in order to provide the basis for the improvement of systems and outcomes. Because technological options are so varied and the future of their development unpredictable, the paper argues that the first screen for the review of technology is one addressing core functions the technology serves. Technologies can provide greater efficiency in managing information or serve to improve the quality of the information used to guide the system. At this point, these functions are frequently mutually exclusive, but in the future, wise selection of technology should involve a consideration of how the technology adds value by increasing quality and efficiency in the enterprise. Quality is added when more knowledge is available to the user. For choices that need to be made about investments, a set of criteria is provided to guide decision logic. Criteria include validity, accuracy, and utility, with an emphasis on users directly involved in learning (students and teachers). In addition, criteria are reviewed related to software/hardware operation, including security, backup, compatibility, resilience, and transparency. The assumption is that systematic application of criteria will influence the market to drive development of more useful systems for education at a cost that can be withstood. Throughout, examples of technology uses now and soon to be available are discussed. The paper concludes with two extended educational scenarios constructed to show elements of design and technology in practice in two very different settings. This paper will analyze the potential use of technology-enabled knowledge systems (TEKS) as integral components in educational accountability systems. The relationship between accountability and knowledge is obvious, for accountability systems depend upon usable information (Baker, 2003) to support their operations. Technology functions, in part, to increase the availability and usability of information. Because there are so many different examples of technology-and more, unimaginable advances left to be developed-it is plainly not practical to give

Report of State Level Activities: Guidelines for the RFP Process and Selected Technical Issues in Large-Scale Assessment Programs. Report of State Level Activities. Project: Monitoring the Implementation of Testing and Evaluation Innovations

Guidelines for evaluating responses to requests for proposals (RFPs) for state-level educational ... more Guidelines for evaluating responses to requests for proposals (RFPs) for state-level educational assessment programs are presented. The guidelines articulate key choice points, options, and considerations by state (and district) testing directors as they solicit services for large-scale assessment from commercial testing companies. Focus is on broadening the scope of issues and concarns that state and district testing officers consider in preparing RFPs for large-scale testing programs. Components that must be considered in dealing with the RFP process include: (1) planning approaches; (2) communications with bidders; (3) the RFP structure; (4) the review process; and (5) issues associated with equating portions of RFPs, item bias portions of RFPs, and content validity portions of RFPs. (TJH)

Plan for the Assessment and Evaluation of Individual and Team Proficiencies Developed by the DARWARS Environments

*Data Collection; *Educational Assessment; Educational Innovation; *Formative Evaluation; *Information Utilization; Political Influences; Research

In November 2002, the Institute for the Advancement of Emerging Technologies in Education at AEL ... more In November 2002, the Institute for the Advancement of Emerging Technologies in Education at AEL gathered education practitioners to help shepherd new assessments from the research laboratory to the classroom. Approximately 60 teachers and administrators attended the conference. The discussion focused on the possibility of aggregating formative data to meet the purposes of large-scale tests and attempted to develop a technology-based method of collecting and analyzing student outcome data to support testing decisions. A panel of forum presented offered an overview of research, specific examples of attempts to create new forms of assessment, and opinions about how the current state of research and the political climate combine to create an extraordinary opportunity to redesign assessments and make them more useful. Panelists agreed that the new systems require sophistication on the part of the teacher. Teachers and administrators in attendance wanted tools that were immediately useable, and it was evident that such tools do not yet exist. The gap between researcher needs and practitioner functioning is frustrating, but frustration is tempered by the attraction of the possibilities of new technology approaches. Panelists were: (1) John Bailey; (2) Linda Roberts; (3) Jim Pellegrino; (4) Eva L. Baker; (5) Christopher Dede; and (6) Lajeane Thomas. (SLD) Reproductions supplied by EDRS are the best that can be made from the original document.

Simulation as a Performance Assessment Technique for the Interpersonal Skill of Negotiation. Report of Task Forces on Technology and Workforce Issues. Project 2.3: Enhancing the Utility of Performance Assessments: Domain-Independent R&D--Workforce

A prototype simulation measure of the negotiating skill of interpersonal competence is described,... more A prototype simulation measure of the negotiating skill of interpersonal competence is described, and results of an initial validation study are reported. Of eight negotiating skills identified by the Secretary's Commission on Achieving Necessary Skills of the U.S. Department of Labor, two were the focus of the measure: proposing and examining possible options and making reasonable compromises. A simulation was developed focusing on these skills in a job contract negotiation between a potential employee and an employer. The validity study of this measur, was based on an expert/novice criterion group approach, assuming that expert negotiators would receive better counter-offers than would novices. Ten experts were chosen from graduate students who had completed a course in negotiation in a Masters in Business Administration program and 21 novices were chosen from a high school business class. Results, generally supportive of the hypothesis, provide evidence of the validity of the simulation as a measure of negotiation skills. Both the assessment of options and the consideration of compromises are important components of negotiating skill. Eight tables of study results are included. (Contains 37 references.) (SLD)

The Effects of Manipulated Item Writing Constraints on the Homogeneity of Test Items. Center for the Study of Evaluation Reprint Series No. 11

The greatest curse, wise men have said, way be to have your wishes corn tree.

New Directions in Evaluation Research: Implications for Vocational Education. Occasional Paper No. 55

In the early 1970s educational evaluation's fusion of respect for rationality, power to iiplement... more In the early 1970s educational evaluation's fusion of respect for rationality, power to iiplement rational procedure, and an altruistic objective (improved instruction) appealed to educational researchers. Dealing essentially with closed systems, it measured program success by student performance on measurement 21struments. Critics exclaimed that program developers and evaluators ok away personal decision rights from students and that outcome measures were incomplete and inaccurate. Changes occured due to reanalysis showing the futility of earlier research, operating focus on procedures (not outcom6s), open student participation, and loss ol stability required for longitudinal study. In present-day open systeE evaluation settings, emphasis is on multiple objectives, and selection of what is to be evaluated is left open. No longer committed to methodology and provision of clear information, evaluators use case studies and looser, more interactive designs. Their reactions to political demands are to use needs assessments,. develop systematic procedures, and write more reports. Problems . between politics and evaluation include these: (1) the technical disagreement that Lny evaluati9a is subject to erodes its credibility with its.contracting agency,.(2) evaluation results can be used to discredit politicians whose olaims outstrip their programs, and (3) success of political efforts gives politicians a sense of personal power. (YLB)

Templates and Objects in Authoring Problem-Solving Assessments. CRESST Report 735

National Center for Research on Evaluation, Standards, and Student Testing, May 1, 2008

Assessing whether students can both re-present a corpus of learned knowledge and also demonstrate... more Assessing whether students can both re-present a corpus of learned knowledge and also demonstrate that they can apply that knowledge to solve problems is key to assessing student understanding. This notion, in turn, impacts our thinking about what we assess, how we author such assessments, and how we interpret assessment results. The diffusion of technology into venues of learning offers new opportunities in the area of student assessment. Specifically, computer-based simulations seem to provide sufficiently rich environments and the tools necessary to allow us to infer accurately how well a student's individual mental model of the world can accommodate, integrate, and be used to exploit concepts from a domain of interest. In this paper then, we first identify the characteristics of simulations that our experience suggests are necessary to make them appropriate for pedagogical and assessment purposes. Next, we discuss the models and frameworks (templates) we have used to ensure these characteristics are considered. Finally, we describe two computerized instantiations (objects) of these frameworks and implications for the follow-on design of simulations.

Templates and Objects in Authoring Problem-Solving Assessments

Routledge eBooks, Sep 25, 2017

Assessing whether students can both re-present a corpus of learned knowledge and also demonstrate... more Assessing whether students can both re-present a corpus of learned knowledge and also demonstrate that they can apply that knowledge to solve problems is key to assessing student understanding. This notion, in turn, impacts our thinking about what we assess, how we author such assessments, and how we interpret assessment results. The diffusion of technology into venues of learning offers new opportunities in the area of student assessment. Specifically, computer-based simulations seem to provide sufficiently rich environments and the tools necessary to allow us to infer accurately how well a student's individual mental model of the world can accommodate, integrate, and be used to exploit concepts from a domain of interest. In this paper then, we first identify the characteristics of simulations that our experience suggests are necessary to make them appropriate for pedagogical and assessment purposes. Next, we discuss the models and frameworks (templates) we have used to ensure these characteristics are considered. Finally, we describe two computerized instantiations (objects) of these frameworks and implications for the follow-on design of simulations.

The Achievement Gap in California: Context, Status, and Approaches for Improvement

Using Measurement to Improve Instruction

Instructional improvement within the context of criterion-referenced and norm-referenced tests is... more Instructional improvement within the context of criterion-referenced and norm-referenced tests is described. Such categories cveremphasize test interpretation rather than design characteristics of achievement tests. Data from most measurement situations may be reported or interpreted either according to criterion-or norm-eferenced standards. HOw the test is developed and what it represents is of critical importance. The paper proposes alternative conceptualizations of test design: construct-referenced, objectives-referenced and domain-referenced. Using student data, the teacher needs to identify deficiencies in achievement, possible explanations, and remedies and to put the remedies into operation. An analysis of the utility of each test type results in the appraisal that domain referenced tests provide the most information for teachers and therefore are the most desirable as data sources for instructional improvement. However, because of lack of knowledge about instruction, poor training in available instructional principles, and lack of resources to encourage changes in instructional habits, it is concluded that instructional improvement, even if measurement considerations were satisfied, is not imminent.

Sensitive Technology Assessment of ACOT

This paper explores the ideas and the model underlying the evaluation of the Apple Classroom of T... more This paper explores the ideas and the model underlying the evaluation of the Apple Classroom of Tomorrow project (ACOT), a 2-year-old research and development project incorporating at least seven different grade levels which is located in five different school sites in four states. The major features of ACOT are identified as the ideas of computer saturation and local site development. Of particular concern in this paper is the evaluation of the technology within the context of the usual goal-focused educational assessment. It is suggested that the assessment of the technology assists in the identification of appropriate goals through a process entitled technology push, which allows for outcomes to be empirically identified as desired consequences or the technology's implementation. A model for the assessment is provided that attempts to incorporate evaluation features central to the schools in which ACOT has been developed, i.e., educational goals, processes, and outcomes. Finally, specific evaluation questions are formulated for the comparison of students at different ACOT sites, and specific achievement measurements are suggested. The text is supplemented by two figures, and three references are provided. (EW)

What Probably Works in Alternative Assessment. CRESST Report 772

National Center for Research on Evaluation, Standards, and Student Testing, Jul 1, 2010

This report provides an overview of what was known about alternative assessment at the time that ... more This report provides an overview of what was known about alternative assessment at the time that the article was written in 1991. Topics include beliefs about assessment reform, overview of alternative assessment including research knowledge, evidence of assessment impact, and critical features of alternative assessment. The author notes that in the short term, alternative assessment will generate negative news about student learning and will require massive support to make it a successful reform strategy.

The Five Families of Cognitive Learning: A Context in Which To Conduct Cognitive Demands Analyses of Innovative Technologies

American Educational Research Association Annual Meeting, Mar 1, 1997

A cognitive demands analysis of a learning technology, a term that includes the hardware and the ... more A cognitive demands analysis of a learning technology, a term that includes the hardware and the computer software products that form learning environments, attempts to describe the types of cognitive learning expected of the individual by the technology. This paper explores the context of cognitive learning, suggesting five families of cognitive learning. These families are: (1) content understanding; (2) collaboration; (3) communication; (4) problem solving; and (5) metacognition. Three technologies are analyzed with reference to these types of learning. The first technology, Algebra Tutor, is an intelligent tutoring system to help students learn algebra skills. "Hands-On Universe" is an astronomical research environment for high school students. "Function Machines" is an application for a visual programming language designed for mathematics and science education. Analyses of the types of learning expected for each of these technologies have resulted in the development of a suite of performance tasks (an integrated simulation) that includes individual and collaborative concept mapping tasks, a problem-solving search task, an explanation task, and a metacognitive questionnaire. This assessment environment is integrated across grade levels and within content areas to blend real-world tasks with a project-based scenario that captures many types of cognitive learning. (Contains 1 table and 79 references.) (SLD)

Aligning Curriculum, Standards, and Assessments: Fulfilling the Promise of School Reform. CSE Report 645