Papers by Shane Gallagher
Cognitive Task Analysis: Analyzing the Cognition of Gameplay and Game Design
A prior study performed by ADL, which measured the effects of certain video game design features ... more A prior study performed by ADL, which measured the effects of certain video game design features thought to increase cognitive adaptability, brought to light how little is understood of the cognitive elements of video games, even by those who design them, let alone those who wish to study them or utilize them for learning or improving cognitive functioning. It has long been standard practice by instructional designers and those in industrial/organizational psychology to utilize a process known as cognitive task analysis (CTA) in order to analyze the cognitive and behavioral requirements of an expert-level performance of a certain task. They are broadly recognized as an effective tool for cognitive mapping. ADL researchers scoured the existing base of research in industrial/organizational psychology, cognitive psychology, and gaming, as well as consulted with experts in the video game industry, and found that no efforts to apply CTA methods to or cognitively map a video game— identifying and enumerating on features such as implicit and explicit rules and reinforcement, rule and environment shifts, audio and visual cues, behavioral and cognitive requirements of players, and goals, sub-goals, and micropuzzles—in order to analyze the cognitive effects of its design could be found. It was towards the end of filling these gaps in knowledge about the cognitive makeup of Portal 2, as well as developing a methodology for applying CTA techniques to video games for future research and utilization for learning, that the researchers at the ADL Initiative are undertaking an effort to perform a cognitive task analysis on Portal 2 play. This paper details the novel methods the team has developed, both by adapting traditional CTA methods to analyze the cognitive requirements of gameplay as well as creating techniques to capture the cognitive effects of unique video game design principles, as well as initial findings.
Supporting Cognitive Adaptability through Game Design
Adaptability is a metacompetency critically important to the United States Department of Defense ... more Adaptability is a metacompetency critically important to the United States Department of Defense and is considered a key component of 21st Century skills by the U.S. Department of Labor and the U.S. Department of Education). Video games are seen as learning environments supporting the acquisition of 21st century skills. Can games, then, be used as components of an effective learning environment that support the development of adaptability? Initially this paper describes the metacompetency of adaptability. Next is how adaptability can be functionally and discretely measured by focusing on its most granular or micromomentary level which we describe as cognitive adaptability. Finally, the authors examine both the nature of cognitive adaptability, interventions that support its development, and how those interventions might be translated into game design features. Toward this end, the paper will also discuss how these features are exhibited in a popular commercially available video game...
Total Learning Architecture Development: A Design-Based Research Approach
Organizations that use learning technology to educate and train are facing a new set of interoper... more Organizations that use learning technology to educate and train are facing a new set of interoperability problems. Many new products—including adaptive systems, intelligent digital tutors, real-time data analytics, and interactive e-books—offer dramatic learning benefits. However, these products primarily “stand alone” and work outside of typical browser-based delivery environments controlled by traditional learning management systems. Furthermore, the effectiveness of these “intelligent systems” often depends on their access to data generated by and stored in other systems. The U.S. DoD Advanced Distributed Learning (ADL) Initiative is designing a framework of specifications, called the Total Learning Architecture (TLA), to ultimately enable “plug-and-play” interoperability of learning technologies. That is, the TLA will allow these new products to interoperate with each other, with other existing learning systems, and even human capital management technologies. Because of the rapi...
Assessing Performance in an Innovative Cybersecurity Pilot Course
In 2014, 25% of all organizations polled across industry said the lack of infosec skills were a p... more In 2014, 25% of all organizations polled across industry said the lack of infosec skills were a problem. In 2015, an Enterprise Strategy Group (ESG) survey found that 28% reported a shortage of infosec skills (Trendmicro, 2015). With the growing threat of cybercrime and national security issues, growing the number of qualified cybersecurity professionals has become a national imperative. As the cybersecurity universe is shaped by new technologies, unknown threats, and increasing vulnerability in a dynamic environment, there is an established need to rapidly establish innovative, effective, efficient and responsive cybersecurity education initiatives (Dark & Mirkovic, 2015). One such initiative recently piloted by the Department of Defense is the Cyber Operations Academy Course (COAC). The first pilot began in May 2015 at the Fort McNair campus in Washington D.C. As a six-month immersive course, participants consisted of 20 mostly military personnel from all four branches of the mili...
Mind, Brain, and Education, 2009
We are happy to announce that Kurt Fischer, PhD, recently received the second award for Transform... more We are happy to announce that Kurt Fischer, PhD, recently received the second award for Transforming Education through Neuroscience for his seminal work in establishing the new field of Mind, Brain, and Education. The award honors an exceptional scholar whose work bridges the fields of neuroscience and education. Co-sponsored by the
Videogame Design for Cognitive Enhancement through Micro-Puzzle Cognitive Profiling
The Advanced Distributed Learning (ADL) Initiative’s Next Generation Learner researchers previous... more The Advanced Distributed Learning (ADL) Initiative’s Next Generation Learner researchers previously investigated whether five video game design features hypothesized to be contained within Portal 2 might increase cognitive adaptability (CA). Their results highlighted a lack of understanding of the cognitive elements of video games within the literature. Subsequently, a protocol for applying cognitive task analysis (CTA) to video games was developed and a CTA was performed on Portal 2 to understand the cognitive components, decisions, and knowledge needed for successful gameplay, as well as to gain a detailed understanding of its design. As a result of the CTA, a compendium of within-level tasks and puzzles the player must complete, referred to as “micro-puzzles,” was compiled, and mapped to the five design features for CA. Results from the initial study showed that certain measures of CA were increased in those playing Portal 2; however, the design of Portal 2 was treated as a “black box.” Through performing a CTA, the presence of the five design characteristics for adaptability was validated by location and by micro-puzzle. Although precisely identified and mapped by game location, there were no specific alignments identified between cognitive measures and micro-puzzle attributes, or between micro-puzzle typology and design feature support. For this reason, the researchers are cognitively codifying micro-puzzles in Portal 2 by type according to their measurable cognitive attributes. This involves defining the micro-puzzles and mapping them to cognitive skills, measurable by the CANTAB battery of tests for CA, followed by empirical testing in the game environment. This paper details this codification and mapping, as well as efforts to build levels in Portal 2 based upon this information in order to cultivate specific cognitive skills, empirically validate the correlation of puzzle type in-game to cognitive gains, and further validate hypothesized game design features to improve cognitive functioning.
In assessing SCORM 2004 for its affordances facilitating the implementation of specific requireme... more In assessing SCORM 2004 for its affordances facilitating the implementation of specific requirements representing a simulation-based model optimized for interoperability and reusability several implications have come to light ranging from gaps in the technical architecture to standard implementation practice to instructional designers and programmers perspectives and understanding. They were identified technically within the RTE and Sequencing as well as in the common implementation practice of designing SCOs purely for content presentation. Findings also point to the need for persistent arbitrary SCO to SCO communication and the ability to conceptualize, design, and implement reusable functional SCOs to fully implement a simulation as an interoperable model within a SCORM environment. Also implied, are gaps in instructional design practice for SCORM-based solutions as well as gaps in the understanding of IT engineers and practitioners in relation to learning theories and practices. In respect to SCORM 2004 and simulations in general as a valuable reusable pedagogical model, the underlying behaviorist pedagogy inherent in SCORM's design needs to be revisited and in so doing the academic community needs to become more involved in its evolution. These findings were derived from a gap analysis using a specific set of requirements derived from an existing online simulation learning environment as the criterion and the Run-time Environment (RTE) and Sequencing of the SCORM 2004 technical architecture as the condition. Results were based on an analysis of quantitative and qualitative data collected from 26 members of the SCORM community employed in industry, government, standards/specifications entities, and academia. Participants were asked to provide levels of agreement to indicator statements of the relevance of the SCORM 2004 targets to the SIMREF at both the individual and set levels. They were also asked to describe alternate standards, specifications, technologies, and capabilities necessary to fulfill the requirements. ABOUT THE AUTHOR Patrick Shane Gallagher, Ph. D. is currently the technical lead for the Advanced Distributed Learning (ADL) initiative and a program manager for SI International. He was the Chief Knowledge Engineer/Instructional Technologist for the Performance Improvement Operation within the Analysis, Simulations, Systems Engineering, and Training (ASSET) Business Unit of Science Applications International Corporation (SAIC). Dr. Gallagher has a successful track record in leading, designing, and implementing enterprise learning and development solutions specializing in the convergence of enterprise learning and knowledge technologies. He led the knowledge management support for the NASA Johnson Space Center office of the CKO and was also the analysis team lead and knowledge architect for the Joint Knowledge Development and Distribution Capability (JKDDC) project for the Department of Defense (DoD) and the JKDDC Joint Management Office. Dr. Gallagher has extensive experience in designing and developing learning and knowledge architectures and systems and has led an internal research and development project for SAIC in the design and development of new content object models to support smart enterprise systems. Dr. Gallagher has accrued customer recognition and awards for thought leadership, innovation, and design and continues to pursue models for convergence in the areas of e-learning, learning technology standards, and knowledge management.
Organizations that use learning technology to educate and train are facing a new set of interoper... more Organizations that use learning technology to educate and train are facing a new set of interoperability problems. Many new products—including adaptive systems, intelligent digital tutors, real-time data analytics, and interactive e-books—offer dramatic learning benefits. However, these products primarily " stand alone " and work outside of typical browser-based delivery environments controlled by traditional learning management systems. Yet, these intelligent systems' effectiveness often depends on their access to data generated and stored in other systems. The U.S. DoD Advanced Distributed Learning (ADL) Initiative is designing a framework of specifications, called the Total Learning Architecture (TLA), to ultimately enable " plug-and-play " interoperability of learning technologies. That is, the TLA will allow these new products to interoperate with each other, with other existing learning systems, and even human capital management technologies to not only track learner activity while managing learner roles and identities but to share intelligent or inferential data and adapt their behavior accordingly. Because of the rapid rate of innovation in such distributed technologies, we adopted a multiyear design-based research approach. During the project's first year, an initial set of specifications have been developed and evaluated for technical and functional adequacy using a multi-round Delphi approach with a panel of international participants (n = 54). Also, in partnership with the U.S. Army JFK Special Warfare Center and School's Special Warfare Education Group, we conducted a live prototype test and demonstration with Special Operations Soldiers (n = 73). This yielded data on the nascent system's functionality, performance, user experience, and learning potential. Analysis of these data will lead to recommendations, which in turn will inform the second cycle of TLA development process. This paper summarizes the TLA concept, development process, first-year analysis efforts and outcomes, and lessons learned leading to design improvements for the second year of TLA development.
In 2014, 25% of all organizations polled across industry said the lack of infosec skills were a p... more In 2014, 25% of all organizations polled across industry said the lack of infosec skills were a problem. In 2015, an Enterprise Strategy Group (ESG) survey found that 28% reported a shortage of infosec skills (Trendmicro, 2015). With the growing threat of cybercrime and national security issues, growing the number of qualified cybersecurity professionals has become a national imperative. As the cybersecurity universe is shaped by new technologies, unknown threats, and increasing vulnerability in a dynamic environment, there is an established need to rapidly establish innovative, effective, efficient and responsive cybersecurity education initiatives (Dark & Mirkovic, 2015). One such initiative recently piloted by the Department of Defense is the Cyber Operations Academy Course (COAC). The first pilot began in May 2015 at the Fort McNair campus in Washington D.C. As a six-month immersive course, participants consisted of 20 mostly military personnel from all four branches of the military services, various backgrounds and little if any cyber experience. Employing an authentic problem-based course using cooperative and collaborative learning models, the pilot consisted of instruction in foundations, defensive/offensive operations, programming, social engineering, and skills integration. Leveraging cyber ranges and capture the flag (CTF) activities, the course was also supported by four " fireteam " leads as facilitators, coaches, and subject matter experts. At the end of the course, students developed cyber capabilities and tools, developed and deployed exploits, detected and responded to incidents, and used social engineering to exploit " targets. " In comparison with existing cyber protection teams deployed in DoD installations, the students were as capable and in some cases more capable in comparisons of performance. In pre/post comparisons, students exhibited potentially large knowledge gains. This paper discusses the nature of the course's pedagogy; the challenge of developing representations of learning outcomes and performance; and the challenges in developing performance-based assessments to authentically and objectively assess students' knowledge and skills in the context of the course provides learning science and methodological direction for applied research projects and cybersecurity assessment and is the lead researcher for assessing the development of the ADL Total Learning Architecture. Dr. Gallagher has directed research on video game design for cognitive adaptability and learning science implications of the design of the xAPI and is also researching methods to apply the xAPI and its syntax to describe social learning interactions and human performance especially within cyber-physical contexts. He has led research projects in cognition and game design and R&D projects in learning object content models, simulations, reusable pedagogical models, organizational readiness, and knowledge management. He has been recognized by NASA for his work on assessing the Johnson Space Center on knowledge management readiness by the JSC Chief Knowledge Officer and has authored papers and chapters on neuroscience, cognition, game design, and innovative learning technology applications and specifications.
Learning Cyber Operations Through Gaming: An overview of current and up and coming gamified learning environments
Cyber warfare, cyberterrorism, and cybercrime are serious existential threats to the national sec... more Cyber warfare, cyberterrorism, and cybercrime are serious existential threats to the national security of the United States. This is driving a demand signal for expert cyber operators that is far outpacing the supply. As a result, there is a pressing need to rapidly establish innovative, effective, efficient and responsive cybersecurity education and training programs. One program, the Cyber Operators Academy Course (COAC) began to solve the problem but is just one component of a comprehensive cybersecurity education and training strategy. As a part of that strategy, other methods including well-designed games should be considered and interestingly enough, there are many parallels between COAC and what occurs in well-designed serious games or gamified learning environments. This article discusses games and gamified learning environments’ place in the cybersecurity training and culminates with brief overviews of four programs currently or imminently available.
As a result of next-generation networking and the Internet of Things (IoT) technologies, big data... more As a result of next-generation networking and the Internet of Things (IoT) technologies, big data analysis is possible
and has been shown to have a positive impact on areas of national significance yet requires new tools to deal with
the variety and quantity of data multiplying at an exponential rate.. Concurrently, IoT technologies are rapidly
becoming a mainstream data source. Training simulations have historically been limited either to computer-based
simulations or live human-observable field-based simulations ;however, IoT technologies can open up innovative,
hybrid digital-physical opportunities both for delivering and for understanding the outcomes of training in a much
more dynamic and comprehensive way. The feasibility of IoT technologies in training has historically been limited
by interoperability and scale. However, Advanced Distributed Learning’s Experience Application Programming
Interface (xAPI) allows interoperability and scale in next-generation training environments and provides a way to
standardize the formative data of human experience captured through digital context. It also provides a way to
capture information and formalize human experience from multiple and varied networked devices into standardized,
human-readable statements. These can inform both human and machine learning through leveraging big data
analysis and interoperability of the IoT technologies. By leveraging the xAPI and IoT technologies as a cyberphysical
system embedded in virtual and live training scenarios, it is possible to capture and measure real-time team
performance for immediate analysis and remediation or for post hoc analysis in after action reviews. This paper
discusses the application of learning analytics and design for an IoT context through describing the implementation
of 1) a live action medical simulation as part of the Global Smart Cities Challenge (sponsored by the NIST and the
OSTP) and 2) the proposed capture and analysis of communication performance data and measures within specific
coalition training scenarios supporting the 2015 Bold Quest Assessment sponsored by the Joint Fires Division of the
Joint Staff.
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2014, Dec 2014
The Advanced Distributed Learning (ADL) Initiative’s Next Generation Learner researchers previous... more The Advanced Distributed Learning (ADL) Initiative’s Next Generation Learner researchers previously investigated whether five video game design features hypothesized to be contained within Portal 2 might increase cognitive adaptability (CA). Their results highlighted a lack of understanding of the cognitive elements of video games within the literature. Subsequently, a protocol for applying cognitive task analysis (CTA) to video games was developed and a CTA was performed on Portal 2 to understand the cognitive components, decisions, and knowledge needed for successful gameplay, as well as to gain a detailed understanding of its design. As a result of the CTA, a compendium of within-level tasks and puzzles the player must complete, referred to as “micro-puzzles,” was compiled, and mapped to the five design features for CA. Results from the initial study showed that certain measures of CA were increased in those playing Portal 2; however, the design of Portal 2 was treated as a “black box.” Through performing a CTA, the presence of the five design characteristics for adaptability was validated by location and by micro-puzzle. Although precisely identified and mapped by game location, there were no specific alignments identified between cognitive measures and micro-puzzle attributes, or between micro-puzzle typology and design feature support. For this reason, the researchers are cognitively codifying micro-puzzles in Portal 2 by type according to their measurable cognitive attributes. This involves defining the micro-puzzles and mapping them to cognitive skills, measurable by the CANTAB battery of tests for CA, followed by empirical testing in the game environment. This paper details this codification and mapping, as well as efforts to build levels in Portal 2 based upon this information in order to cultivate specific cognitive skills, empirically validate the correlation of puzzle type in-game to cognitive gains, and further validate hypothesized game design features to improve cognitive functioning.
A prior study performed by ADL, which measured the effects of certain video game design features ... more A prior study performed by ADL, which measured the effects of certain video game design features thought to increase cognitive adaptability, brought to light how little is understood of the cognitive elements of video games, even by those who design them, let alone those who wish to study them or utilize them for learning or improving cognitive functioning. It has long been standard practice by instructional designers and those in industrial/organizational psychology to utilize a process known as cognitive task analysis (CTA) in order to analyze the cognitive and behavioral requirements of an expert-level performance of a certain task. They are broadly recognized as an effective tool for cognitive mapping. ADL researchers scoured the existing base of research in industrial/organizational psychology, cognitive psychology, and gaming, as well as consulted with experts in the video game industry, and found that no efforts to apply CTA methods to or cognitively map a video gameidentifying and enumerating on features such as implicit and explicit rules and reinforcement, rule and environment shifts, audio and visual cues, behavioral and cognitive requirements of players, and goals, sub-goals, and micropuzzles-in order to analyze the cognitive effects of its design could be found. It was towards the end of filling these gaps in knowledge about the cognitive makeup of Portal 2, as well as developing a methodology for applying CTA techniques to video games for future research and utilization for learning, that the researchers at the ADL Initiative are undertaking an effort to perform a cognitive task analysis on Portal 2 play. This paper details the novel methods the team has developed, both by adapting traditional CTA methods to analyze the cognitive requirements of gameplay as well as creating techniques to capture the cognitive effects of unique video game design principles, as well as initial findings.
Adaptability is a metacompetency critically important to the United States Department of Defense ... more Adaptability is a metacompetency critically important to the United States Department of Defense (DOD). 1 There is a need for organizations, leaders, and individuals to adapt to an increase in the type and intensity of stressors and ambiguity existing in today's business, political, and defense environments, a need that is not limited by organizational or generational boundaries. ACT21S (Assessment and Teaching of 21 st century Skills) has identified three "ways of thinking" skills as part of defining 21 st century skills: creativity and innovation; critical thinking, problem solving, decisionmaking; and learning to learn, metacognition. 2 The National Research Council has discussed the need for these ways of thinking to be acquired generally across problem domains to be effective. 3 Taught as general cognitive skills, creativity, problem-solving, and metacognition are all crucial skills facilitating desirable competencies including adaptability.
adlnet.gov
Adaptability is a metacompetency critically important to the United States Department of Defense ... more Adaptability is a metacompetency critically important to the United States Department of Defense and is considered a key component of 21 st Century skills by the U.S. Department of Labor and the U.S. Department of Education). Video games are seen as learning environments supporting the acquisition of 21 st century skills. Can games, then, be used as components of an effective learning environment that support the development of adaptability?
6th European Conference on Games …, 2012
Adaptability is a metacompetency critically important to the United States Department of Defense ... more Adaptability is a metacompetency critically important to the United States Department of Defense and is considered a key component of 21 st Century skills by the U.S. Department of Labor and the U.S. Department of Education). Video games are seen as learning environments supporting the acquisition of 21 st century skills. Can games, then, be used as components of an effective learning environment that support the development of adaptability?
Assessing SCORM 2004 for its Affordances in Facilitating a Simulation as a Pedagogical Model
Gaps in SCORM Implementation and Practice Using an Online Simulation
The Interservice/Industry Training, Simulation & …, 2008
... Supporting this shift, the convergences of Information and Communication Technology or ICT-ba... more ... Supporting this shift, the convergences of Information and Communication Technology or ICT-based knowledge management and e-learning systems are providing more learner control. ... Yet, there exists confusion over scope and definition usually due to terminology. ...
Simulation Representation using SCORM
The Interservice/Industry Training, Simulation & …, 2006
... Patrick S.Gallagher Hasan Altalib SAIC SAIC ... ABOUT THE AUTHORS Patrick Shane Gallagher i... more ... Patrick S.Gallagher Hasan Altalib SAIC SAIC ... ABOUT THE AUTHORS Patrick Shane Gallagher is a Senior Knowledge Engineer and project manager in the Strategies, Simulations, and Training Business Unit of Science Applications International Corporation (SAIC). ...
Uploads
Papers by Shane Gallagher
and has been shown to have a positive impact on areas of national significance yet requires new tools to deal with
the variety and quantity of data multiplying at an exponential rate.. Concurrently, IoT technologies are rapidly
becoming a mainstream data source. Training simulations have historically been limited either to computer-based
simulations or live human-observable field-based simulations ;however, IoT technologies can open up innovative,
hybrid digital-physical opportunities both for delivering and for understanding the outcomes of training in a much
more dynamic and comprehensive way. The feasibility of IoT technologies in training has historically been limited
by interoperability and scale. However, Advanced Distributed Learning’s Experience Application Programming
Interface (xAPI) allows interoperability and scale in next-generation training environments and provides a way to
standardize the formative data of human experience captured through digital context. It also provides a way to
capture information and formalize human experience from multiple and varied networked devices into standardized,
human-readable statements. These can inform both human and machine learning through leveraging big data
analysis and interoperability of the IoT technologies. By leveraging the xAPI and IoT technologies as a cyberphysical
system embedded in virtual and live training scenarios, it is possible to capture and measure real-time team
performance for immediate analysis and remediation or for post hoc analysis in after action reviews. This paper
discusses the application of learning analytics and design for an IoT context through describing the implementation
of 1) a live action medical simulation as part of the Global Smart Cities Challenge (sponsored by the NIST and the
OSTP) and 2) the proposed capture and analysis of communication performance data and measures within specific
coalition training scenarios supporting the 2015 Bold Quest Assessment sponsored by the Joint Fires Division of the
Joint Staff.
and has been shown to have a positive impact on areas of national significance yet requires new tools to deal with
the variety and quantity of data multiplying at an exponential rate.. Concurrently, IoT technologies are rapidly
becoming a mainstream data source. Training simulations have historically been limited either to computer-based
simulations or live human-observable field-based simulations ;however, IoT technologies can open up innovative,
hybrid digital-physical opportunities both for delivering and for understanding the outcomes of training in a much
more dynamic and comprehensive way. The feasibility of IoT technologies in training has historically been limited
by interoperability and scale. However, Advanced Distributed Learning’s Experience Application Programming
Interface (xAPI) allows interoperability and scale in next-generation training environments and provides a way to
standardize the formative data of human experience captured through digital context. It also provides a way to
capture information and formalize human experience from multiple and varied networked devices into standardized,
human-readable statements. These can inform both human and machine learning through leveraging big data
analysis and interoperability of the IoT technologies. By leveraging the xAPI and IoT technologies as a cyberphysical
system embedded in virtual and live training scenarios, it is possible to capture and measure real-time team
performance for immediate analysis and remediation or for post hoc analysis in after action reviews. This paper
discusses the application of learning analytics and design for an IoT context through describing the implementation
of 1) a live action medical simulation as part of the Global Smart Cities Challenge (sponsored by the NIST and the
OSTP) and 2) the proposed capture and analysis of communication performance data and measures within specific
coalition training scenarios supporting the 2015 Bold Quest Assessment sponsored by the Joint Fires Division of the
Joint Staff.