Although first popular as an instructional tool in the classroom, knowledge mapping has been used... more Although first popular as an instructional tool in the classroom, knowledge mapping has been used increasingly as an assessment tool. Knowledge mapping is expected to measure deep conceptual understanding and allow students to characterize relationships among concepts in a domain visually. Our research examines the validity of knowledge mapping as an assessment tool in science. Our approach to investigating this validity is three-pronged. First, we outline a model for the creation of knowledge mapping tasks, proposing a standard set of steps and using content area and educational experts to ensure the content validity of the measures. Next, we describe a scoring method used to evaluate student performance, including a discussion of the method's reliability and its relationship to other possible scoring systems. Finally, we present our statistical results including comparative analyses, our multitrait-multimethod validity analyses involving two traits (students' understanding of hearing and of vision) and three different measurement methods (knowledge mapping, essay, and multiple-choice tasks), critical proposition analyses, and analyses of students' propositional elaborations. Results show knowledge maps to be sensitive to students' competency level, with mixed MTMM results. We conclude with a discussion of implications and directions for future work.
We examined a novel formative assessment and instructional approach with 89 students in three ele... more We examined a novel formative assessment and instructional approach with 89 students in three electrical engineering classes in special computer-based discussion sections. The technique involved students individually solving circuit problems online, with their real-time responses observed by the instructor. While exploratory, survey and interview responses from 26 students suggest the technique offers important instructional and assessment advantages: Compared to typical discussion sessions, a large majority of respondents reported being more engaged, learning more, and interacting more with the instructor. Students reported the anonymous mode allowed them to ask "dumb" questions. The instructor was able to address student problems and questions immediately, and the amount of formative assessment information from the interaction far exceeded what was available in typical settings.
In this report, we examine the feasibility of scoring essays using computerbased techniques. We r... more In this report, we examine the feasibility of scoring essays using computerbased techniques. We review two approaches that have been used to classify documents, surface-and word-based analysis techniques. We omit other text analysis techniques, such as content analyses (perform text classification rather than text interpretation. The objective of this report is to first review candidate approaches to the automated classification of documents, and then outline how these approaches could be used to achieve our overarching goal of the automated scoring of essays. Our stance is to evaluate these approaches from an applied perspective. We evaluate strengths and weaknesses of each approach and consider both near-and long-term issues of availability, application to text processing in general, and computational complexity.
The goal of this report was to test the use of sensor-based skill measures in evaluating performa... more The goal of this report was to test the use of sensor-based skill measures in evaluating performance differences in rifle marksmanship. Ten shots were collected from 30 novices and 9 experts. Three measures for breath control and one for trigger control were used to predict skill classification. The data were fitted with a logistic regression model using holdout validation to assess the quality of model classifications. Individually, all four measures were significant; when considered together, only three measures were significant predictors for level of expertise (p < .05). Overall percent correct in shot classification for the testing data was 90.0%, with a sensitivity of 67.5%, and 96.0% specificity.
Our research question was whether we could develop a feasible technique, using Bayesian networks,... more Our research question was whether we could develop a feasible technique, using Bayesian networks, to diagnose gaps in student knowledge. Thirty-four college-age participants completed tasks designed to measure conceptual knowledge, procedural knowledge, and problem-solving skills related to circuit analysis. A Bayesian network was used to model the knowledge dependencies among the circuit analysis concepts.
Marksmanship training involves a combination of classroom instructional learning and field practi... more Marksmanship training involves a combination of classroom instructional learning and field practice involving the instantiation of a well-defined set of sensory, motor and cognitive skills. Current training procedures rely heavily on conventional classroom instruction often with qualitative assessment based on observation (i.e. coaching). We have developed a novel device called the Adaptive Performance Trainer (APT TM) which can accelerate the progression from novice-to-expert based on automated inferences from neurophysiological measurements. Our previous work has revealed specific EEG correlates to stages of skill acquisition in simple learning and memory tasks. We have incorporated this knowledge as well as an array of other physiological metrics to develop a field-deployable training technology with continuous physiological monitoring in combination with simultaneous measures of performance, workload, engagement and distraction, accuracy, speed and efficiency. This paper outline...
Marksmanship training involves a combination of classroom instructional learning and field practi... more Marksmanship training involves a combination of classroom instructional learning and field practice involving the instantiation of a welldefined set of sensory, motor and cognitive skills. Current training procedures rely heavily on conventional classroom instruction often with qualitative assessment based on observation (i.e. coaching). We have developed a novel device called the Adaptive Performance Trainer (APT TM ) which can accelerate the progression from novice-to-expert based on automated inferences from neurophysiological measurements. Our previous work has revealed specific EEG correlates to stages of skill acquisition in simple learning and memory tasks. We have incorporated this knowledge as well as an array of other physiological metrics to develop a field-deployable training technology with continuous physiological monitoring in combination with simultaneous measures of performance, workload, engagement and distraction, accuracy, speed and efficiency. This paper outlines the features of the APT and the preliminary results of its use in marksmanship training.
Conference proceedings : ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual Conference, 2009
The learning of a novel task currently rely heavily on conventional classroom instruction with qu... more The learning of a novel task currently rely heavily on conventional classroom instruction with qualitative assessment and observation. Introduction of individualized tutorials with integrated neuroscience-based evaluation techniques could significantly accelerate skill acquisition and provide quantitative evidence of successful training. We have created a suite of adaptive and interactive neuro-educational technologies (I-NET) to increase the pace and efficiency of skill learning. It covers four major themes: 1) Integration of brain monitoring into paced instructional tutorials, 2) Identifying psychophysiological characteristics of expertise using a model population, 3) Developing sensor-based feedback to accelerate novice-to-expert transition, 4) Identifying neurocognitive factors that are predictive of skill acquisition to allow early triage and interventions. We selected rifle marksmanship training as the field of application. Rifle marksmanship is a core skill for the Army and M...
Abstract. Marksmanship training includes a combination of classroom instruc-tion and field practi... more Abstract. Marksmanship training includes a combination of classroom instruc-tion and field practice involving the instantiation of a well-defined set of sensory, motor, and cognitive skills. 10 expert marksmen and 30 novices participated in a study that measured marksman performance ...
Our research objective is to use neuroscience-based assessments to accelerate military skill acqu... more Our research objective is to use neuroscience-based assessments to accelerate military skill acquisition and provide quantitative evidence of successful training by detecting, in realtime, cognitive and physiological states of the trainee under various conditions. The research focuses on: (a) integrating brain monitoring capabilities into rifle marksmanship training; (b) identifying psychophysiological characteristics of expertise using expert marksman as a model population; (c) developing a sensor-based feedback systeminformation that would not be available under current training conditions-to accelerate novices in the acquisition of marksmanship skills, and (d) identifying neurocognitive factors that predict marksmanship skill acquisition.
National Center For Research on Evaluation Standards and Student Testing, Feb 1, 2004
Collaborative problem-solving skills are considered necessary skills for success in today's world... more Collaborative problem-solving skills are considered necessary skills for success in today's world of work and school. Cooperative learning refers to learning environments in which small groups of people work together to achieve a common goal, and problem solving is defined as "cognitive processing directed at achieving a common goal when no solution method is obvious to the problem solver" (R. E. Mayer & M. C. Wittrock, 1996, p. 47). Thus, collaborative problem solving is defined as problem-solving activities that involve interactions among a group of individuals. This paper will address several key issues (e.g., theory and measurement of collaborative problem solving and issues in measuring problem-solving processes). We rely on computerization of the administration, scoring, and reporting of collaborative problem-solving skills, thus potentially increasing reliability and validity.
National Center For Research on Evaluation Standards and Student Testing, Mar 1, 2009
In this report, researchers examined whether individualized multimedia-based instruction would in... more In this report, researchers examined whether individualized multimedia-based instruction would influence the development of rifle marksmanship skills in novice shooters with little or no prior rifle marksmanship experience. Forty-eight novice shooters used an M4 rifle training simulator system to shoot at an 8-inch target at a simulated distance of 200 yards. Participants received either (a) no instruction, (b) only an overview of rifle marksmanship, or (c) an overview and instruction targeted at particular skill gaps. Support was found for the idea that multimedia-based instruction can be highly effective for novices, with a large increase in shooting performance observed after 10 to 15 minutes of multimedia instruction. Subsequent individualized instruction using very short multimedia instruction appeared to be effective in shaping participants' skills toward an "ideal" state consistent with shooting doctrine.
Although first popular as an instructional tool in the classroom, knowledge mapping has been used... more Although first popular as an instructional tool in the classroom, knowledge mapping has been used increasingly as an assessment tool. Knowledge mapping is expected to measure deep conceptual understanding and allow students to characterize relationships among concepts in a domain visually. Our research examines the validity of knowledge mapping as an assessment tool in science. Our approach to investigating this validity is three-pronged. First, we outline a model for the creation of knowledge mapping tasks, proposing a standard set of steps and using content area and educational experts to ensure the content validity of the measures. Next, we describe a scoring method used to evaluate student performance, including a discussion of the method's reliability and its relationship to other possible scoring systems. Finally, we present our statistical results including comparative analyses, our multitrait-multimethod validity analyses involving two traits (students' understanding of hearing and of vision) and three different measurement methods (knowledge mapping, essay, and multiple-choice tasks), critical proposition analyses, and analyses of students' propositional elaborations. Results show knowledge maps to be sensitive to students' competency level, with mixed MTMM results. We conclude with a discussion of implications and directions for future work.
We examined a novel formative assessment and instructional approach with 89 students in three ele... more We examined a novel formative assessment and instructional approach with 89 students in three electrical engineering classes in special computer-based discussion sections. The technique involved students individually solving circuit problems online, with their real-time responses observed by the instructor. While exploratory, survey and interview responses from 26 students suggest the technique offers important instructional and assessment advantages: Compared to typical discussion sessions, a large majority of respondents reported being more engaged, learning more, and interacting more with the instructor. Students reported the anonymous mode allowed them to ask "dumb" questions. The instructor was able to address student problems and questions immediately, and the amount of formative assessment information from the interaction far exceeded what was available in typical settings.
In this report, we examine the feasibility of scoring essays using computerbased techniques. We r... more In this report, we examine the feasibility of scoring essays using computerbased techniques. We review two approaches that have been used to classify documents, surface-and word-based analysis techniques. We omit other text analysis techniques, such as content analyses (perform text classification rather than text interpretation. The objective of this report is to first review candidate approaches to the automated classification of documents, and then outline how these approaches could be used to achieve our overarching goal of the automated scoring of essays. Our stance is to evaluate these approaches from an applied perspective. We evaluate strengths and weaknesses of each approach and consider both near-and long-term issues of availability, application to text processing in general, and computational complexity.
The goal of this report was to test the use of sensor-based skill measures in evaluating performa... more The goal of this report was to test the use of sensor-based skill measures in evaluating performance differences in rifle marksmanship. Ten shots were collected from 30 novices and 9 experts. Three measures for breath control and one for trigger control were used to predict skill classification. The data were fitted with a logistic regression model using holdout validation to assess the quality of model classifications. Individually, all four measures were significant; when considered together, only three measures were significant predictors for level of expertise (p < .05). Overall percent correct in shot classification for the testing data was 90.0%, with a sensitivity of 67.5%, and 96.0% specificity.
Our research question was whether we could develop a feasible technique, using Bayesian networks,... more Our research question was whether we could develop a feasible technique, using Bayesian networks, to diagnose gaps in student knowledge. Thirty-four college-age participants completed tasks designed to measure conceptual knowledge, procedural knowledge, and problem-solving skills related to circuit analysis. A Bayesian network was used to model the knowledge dependencies among the circuit analysis concepts.
Marksmanship training involves a combination of classroom instructional learning and field practi... more Marksmanship training involves a combination of classroom instructional learning and field practice involving the instantiation of a well-defined set of sensory, motor and cognitive skills. Current training procedures rely heavily on conventional classroom instruction often with qualitative assessment based on observation (i.e. coaching). We have developed a novel device called the Adaptive Performance Trainer (APT TM) which can accelerate the progression from novice-to-expert based on automated inferences from neurophysiological measurements. Our previous work has revealed specific EEG correlates to stages of skill acquisition in simple learning and memory tasks. We have incorporated this knowledge as well as an array of other physiological metrics to develop a field-deployable training technology with continuous physiological monitoring in combination with simultaneous measures of performance, workload, engagement and distraction, accuracy, speed and efficiency. This paper outline...
Marksmanship training involves a combination of classroom instructional learning and field practi... more Marksmanship training involves a combination of classroom instructional learning and field practice involving the instantiation of a welldefined set of sensory, motor and cognitive skills. Current training procedures rely heavily on conventional classroom instruction often with qualitative assessment based on observation (i.e. coaching). We have developed a novel device called the Adaptive Performance Trainer (APT TM ) which can accelerate the progression from novice-to-expert based on automated inferences from neurophysiological measurements. Our previous work has revealed specific EEG correlates to stages of skill acquisition in simple learning and memory tasks. We have incorporated this knowledge as well as an array of other physiological metrics to develop a field-deployable training technology with continuous physiological monitoring in combination with simultaneous measures of performance, workload, engagement and distraction, accuracy, speed and efficiency. This paper outlines the features of the APT and the preliminary results of its use in marksmanship training.
Conference proceedings : ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual Conference, 2009
The learning of a novel task currently rely heavily on conventional classroom instruction with qu... more The learning of a novel task currently rely heavily on conventional classroom instruction with qualitative assessment and observation. Introduction of individualized tutorials with integrated neuroscience-based evaluation techniques could significantly accelerate skill acquisition and provide quantitative evidence of successful training. We have created a suite of adaptive and interactive neuro-educational technologies (I-NET) to increase the pace and efficiency of skill learning. It covers four major themes: 1) Integration of brain monitoring into paced instructional tutorials, 2) Identifying psychophysiological characteristics of expertise using a model population, 3) Developing sensor-based feedback to accelerate novice-to-expert transition, 4) Identifying neurocognitive factors that are predictive of skill acquisition to allow early triage and interventions. We selected rifle marksmanship training as the field of application. Rifle marksmanship is a core skill for the Army and M...
Abstract. Marksmanship training includes a combination of classroom instruc-tion and field practi... more Abstract. Marksmanship training includes a combination of classroom instruc-tion and field practice involving the instantiation of a well-defined set of sensory, motor, and cognitive skills. 10 expert marksmen and 30 novices participated in a study that measured marksman performance ...
Our research objective is to use neuroscience-based assessments to accelerate military skill acqu... more Our research objective is to use neuroscience-based assessments to accelerate military skill acquisition and provide quantitative evidence of successful training by detecting, in realtime, cognitive and physiological states of the trainee under various conditions. The research focuses on: (a) integrating brain monitoring capabilities into rifle marksmanship training; (b) identifying psychophysiological characteristics of expertise using expert marksman as a model population; (c) developing a sensor-based feedback systeminformation that would not be available under current training conditions-to accelerate novices in the acquisition of marksmanship skills, and (d) identifying neurocognitive factors that predict marksmanship skill acquisition.
National Center For Research on Evaluation Standards and Student Testing, Feb 1, 2004
Collaborative problem-solving skills are considered necessary skills for success in today's world... more Collaborative problem-solving skills are considered necessary skills for success in today's world of work and school. Cooperative learning refers to learning environments in which small groups of people work together to achieve a common goal, and problem solving is defined as "cognitive processing directed at achieving a common goal when no solution method is obvious to the problem solver" (R. E. Mayer & M. C. Wittrock, 1996, p. 47). Thus, collaborative problem solving is defined as problem-solving activities that involve interactions among a group of individuals. This paper will address several key issues (e.g., theory and measurement of collaborative problem solving and issues in measuring problem-solving processes). We rely on computerization of the administration, scoring, and reporting of collaborative problem-solving skills, thus potentially increasing reliability and validity.
National Center For Research on Evaluation Standards and Student Testing, Mar 1, 2009
In this report, researchers examined whether individualized multimedia-based instruction would in... more In this report, researchers examined whether individualized multimedia-based instruction would influence the development of rifle marksmanship skills in novice shooters with little or no prior rifle marksmanship experience. Forty-eight novice shooters used an M4 rifle training simulator system to shoot at an 8-inch target at a simulated distance of 200 yards. Participants received either (a) no instruction, (b) only an overview of rifle marksmanship, or (c) an overview and instruction targeted at particular skill gaps. Support was found for the idea that multimedia-based instruction can be highly effective for novices, with a large increase in shooting performance observed after 10 to 15 minutes of multimedia instruction. Subsequent individualized instruction using very short multimedia instruction appeared to be effective in shaping participants' skills toward an "ideal" state consistent with shooting doctrine.
Uploads
Papers by Gregory Chung