This paper reports outcomes of 57 students' exploration of urban planning and environmental scien... more This paper reports outcomes of 57 students' exploration of urban planning and environmental science identities through Virtual City Planning, a course implemented in a science museum that leveraged a virtual learning environment supported by in-class play-based experiences. Identity exploration trajectories were assessed using the Projective Reflection framework, which consists of constructs that capture cognitive, affective, and behavioral features of the self in addition to learners' self-perceptions and definitions. Researchers constructed a parsimonious epistemic network that was supported by in-depth qualitative interpretations to a) visualize students' general trends of student self-reflection across the course experience and b) highlight which Projective Reflection constructs were highly nascent to participants as they engaged in identity exploration. Results further theoretical understandings of how courses designed to support identity exploration influence the sophistication and content of learners' reflections on the self and illustrate the utility of epistemic networks for visualizing identity exploration trajectories over time.
This paper explores methodological questions in the study of identity through an examination and ... more This paper explores methodological questions in the study of identity through an examination and discussion of the empirical papers in this special issue. Particular attention is paid to the ways identity is operationalized in the study of how learning environments foster changes in students' sense of self. The paper concludes that identity is a difficult construct to study in the context of learning environments because it is simultaneously performative and subjective, and these dual aspects of identity may be best operationalized in an interactional view, in which identity is conceptualized as a set of relations between aspects of identity rather than as a state that can be coded directly in data on learning.
Epistemic network analysis (ENA) has been used in more than 300 published studies to date. Howeve... more Epistemic network analysis (ENA) has been used in more than 300 published studies to date. However, there is no work in publication that describes the transformations that constitute ENA in formal mathematical terms. This paper provides such a description, focusing on the mathematical formulations that lead to two key affordances of ENA that are not present in other network analysis tools or multivari-ate analyses: (1) summary statistics that can be used to compare the differences in the content rather than the structure of networks and (2) network visualizations that provide information that is mathematically consistent with those statistics. Specifically, we describe the mathematical transformations by which ENA constructs matrix representations of discourse data, uses those representations to generate networks for units of analysis, places those networks into a metric space, identifies meaningful dimensions in the space, and positions the nodes of network graphs within that space so as to enable interpretation of those dimensions in terms of the content of the networks. We conclude with a discussion of how the mathematical formalisms of ENA can be used to model networks more generally.
Coding data-defining concepts and identifying where they occur in data-is a critical aspect of qu... more Coding data-defining concepts and identifying where they occur in data-is a critical aspect of qualitative data analysis, and especially so in quantitative ethnography. Coding is a central process for creating meaning from data, and while much has been written about coding methods and theory, relatively little has been written about what constitutes best practices for fair and valid coding, what justifies those practices, and how to implement them. In this paper, our goal is not to address these issues comprehensively, but to provide guidelines for good coding practice and to highlight some of the issues and key questions that quantitative ethnographers and other researchers should consider when coding data.
A key goal of quantitative ethnographic (QE) models, and statistical models more generally, is to... more A key goal of quantitative ethnographic (QE) models, and statistical models more generally, is to produce the most parsimonious model that adequately explains or predicts the phenomenon of interest. In epistemic network analysis (ENA), for example, this entails constructing network models with the fewest number of codes whose interaction structure provides sufficient explanatory power in a given context. Unlike most statistical models, however, modification of ENA models can affect not only the statistical properties but also the interpretive alignment between quantitative features and qualitative meaning that is a central goal in QE analyses. In this study, we propose a novel method, Parsimonious Removal with Interpretive Alignment, for systematically identifying more parsimonious ENA models that are likely to maintain interpretive alignment with an existing model. To test the efficacy of the method, we implemented it on a well-studied dataset for which there is a published, validated ENA model, and we show that the method successfully identifies reduced models likely to maintain explanatory power and interpretive alignment.
We propose an extension to epistemic network analysis (ENA) that enables researchers to model the... more We propose an extension to epistemic network analysis (ENA) that enables researchers to model the nested effects of multiple grouping variables in an ENA context.
Quantitative ethnographers across a range of domains study complex collaborative thinking (CCT): ... more Quantitative ethnographers across a range of domains study complex collaborative thinking (CCT): the processes by which members of a group or team develop shared understanding by making cognitive connections from the statements and actions of the group. CCT is difficult to model because the actions of group members are interdependent-the activity of any individual is influenced by the actions of other members of the group. Moreover, the actions of group members engaged in some collaborative tasks may need to follow a particular order. However, current techniques can account for either interdependence or order, but not both. In this paper, we present directed epistemic network analysis (dENA), an extension of epistemic network analysis (ENA), as a method that can simultaneously account for the interdependent and ordered aspects of CCT. To illustrate the method, we compare a qualitative analysis of two U.S. Navy commanders working in a simulation to ENA and dENA analyses of their performance. We find that by accounting for interdependence but not order, ENA was not able to model differences between the commanders seen in the qualitative analysis, but by accounting for both interdependence and order, dENA was able to do so.
this study explores the concept of authenticity in education, which has been, over the last 25 ye... more this study explores the concept of authenticity in education, which has been, over the last 25 years, a powerful metaphor for educational practice, particularly as a guiding principle for some technological innovations that support student learning. the concept of authenticity has a variety of meanings , although a dominant interpretation is that authenticity refers to the alignment of educational practice with activities in the real world. Based on a meta-analysis of research on the topic, Shaffer and resnick (1999) proposed unifying the array of meanings for authenticity with their concept of thick authenticity, which included four different aspects of authentic learning: real-world, assessment, disciplinary, and personal authenticity. thick authenticity synthesizes these different facets into a mutually interdependent whole. this study revisits this discussion two decades later, again conducting a meta-review of recent literature, and re-finding the four meanings for authenticity. however, we also identify in the current literature on authenticity a fifth meaning: teacher authenticity. We then analyze two cases of technology-based teaching to explore how teachers manage the elements of authenticity in their classrooms, what this tells us about the theoretical construct of thick authenticity, and specifically the
We analyze pre-service teachers' diagnostic argumentations in learning with simulated cases by ex... more We analyze pre-service teachers' diagnostic argumentations in learning with simulated cases by exploring relations within and between the two dimensions of domain-specific conceptual knowledge and cross-domain epistemic activities. Using the method of ENA, we conclude that both dimensions are relevant in pre-service teachers' diagnostic argumentation and hence in designing learning interventions for its facilitation. The results also suggest that conceptual knowledge better explains differences between accurately vs. inaccurately diagnosing pre-service teachers compared to the dimension of epistemic activities.
Zachari Swiecki is a Ph.D. student in the Department of Educational Psychology (Learning Sciences... more Zachari Swiecki is a Ph.D. student in the Department of Educational Psychology (Learning Sciences Program) at the University of Wisconsin-Madison. His research focuses on modeling complex, collaborative thinking.
Health care is fundamentally about people, and therefore, engineering approaches for studying hea... more Health care is fundamentally about people, and therefore, engineering approaches for studying health-care systems must consider the perspective, concepts and methods offered by the human factors and ergonomics (HFE) discipline. HFE analysis is often qualitative to provide in-depth description of work systems and processes. To deepen our understanding of care processes, we propose the next level of analysis; i.e., quantification of qualitative data. Here, we describe epistemic network analysis (ENA) as a novel method to quantify qualitative data and present a case study applying ENA to assess communication in a primary care team. One high-performing primary care team, consisting of a physician, nurse, medical assistant and unit clerk, was observed for 15 hours. We analyzed task-allocation communications and identified the sender, receiver, synchronicity and acceptance. We used logistic regression and ENA to evaluate sender, receiver and synchronicity impact on task acceptance. The physician and unit clerk were most successful at allocating tasks. Future work should consider the role of synchronous, interruptive communication as potentially useful in time-critical tasks and further investigate the role of the unit clerk. HFE researchers should consider ENA as a tool to expand and deepen their understanding of care processes by quantifying qualitative data.
Collaborative Problem Solving (CPS) is an interactive, interdependent, and temporal process. Howe... more Collaborative Problem Solving (CPS) is an interactive, interdependent, and temporal process. However, current methods for measuring the CPS processes of individuals, such as coding and counting, treat these processes as sets of isolated and independent events. In contrast, Epistemic Network Analysis (ENA) models how the contributions of a given individual relate to the contributions of others. This article examines the communications of air defense warfare teams from an experiment comparing two different computer-based decision support systems, using this data to ask whether ENA provides a more ecologically valid quantitative model of CPS than coding and counting. Qualitative analysis showed that commanders using one system asked questions to understand the tactical situation, while commanders using an experimental system focused more on actions in response to the tactical situation. Neither of the coding and counting approaches we tested corroborated these findings with statistically significant results. In contrast, ENA created models of the individual contributions of commanders that (a) showed statistical differences between commanders using the two systems to corroborate the qualitative analysis, and (b) revealed differences in individual performance. This suggests that ENA is a more powerful tool for CPS assessment than coding and counting approaches.
Many researchers have argued that models of collaborative processes should account for temporalit... more Many researchers have argued that models of collaborative processes should account for temporality, but there exist different approaches for doing so. We compared two specific approaches to modeling collaborative processes in a CSCL context: Epistemic Network Analysis, which models events cotemporally (unordered and temporally proximate), and Sequential Pattern Mining, which models events sequentially (ordered and temporally proximate). Our results suggest that in this context cotemporal models constructed with Epistemic Network Analysis outperform sequential models constructed with Sequential Pattern Mining in terms of (a) explanatory power, (b) efficiency, and (c) interpretability.
The process of labeling, categorizing, or otherwise annotating data-or coding in the computer-sup... more The process of labeling, categorizing, or otherwise annotating data-or coding in the computer-supported collaborative learning (CSCL) literature-is a fundamental process in CSCL research. It is the process by which researchers identify salient properties about segments of CSCL data: what they are, what they contain, or what they mean. Coding, like all processes in research, is subject to error. To reduce the potential impact of coding error, CSCL researchers typically measure inter-rater reliability (IRR). However, there is no extant method to determine what level of IRR would invalidate a CSCL result or model. One way of assessing the potential impact of such inaccuracies is by conducting sensitivity analyses, which measure the level of error that would need to be present in the data to invalidate a given inference. This paper introduces a new method for conducting sensitivity analyses in CSCL: the Binary Replicate Test.
Educators who develop and use place-based curricula argue that engaging students in authentic pro... more Educators who develop and use place-based curricula argue that engaging students in authentic problem-based learning situated in students' own local place helps them understand the relevance of their academic work, which improves learning. However, while place-based curricula are localized, they are also authentic, problem-based experiences, which have been shown to be effective in their own right. Without controlling for the impact of authenticity, it is not clear whether localization itself improves student learning. In this study, we used a place-based simulation to engage students with authentic problem-based learning about a specific place: a city that is local for some students and not for others. We assessed differences in cognitive, affective, and behavioral outcomes between these two groups. Our results suggest that learning outcomes are better when students engage in a place-based simulation about their own locality, indicating that place does play a critical role in the success of placebased education.
This paper examines whether and to what extent long and short readers make different contribution... more This paper examines whether and to what extent long and short readers make different contributions to collaborative design discussions in a CSCL environment-that is, we investigate whether a simple measure of reading behavior based on clickstream data is a good proxy for engagement with readings. Our approach to addressing this question is multimodal, involving two sources of data: (a) a record of students' online conversations, and (b) the frequency and duration with which documents were open on each student's screen. This study suggests that in this specific case, relatively thin data about reading frequency and mean reading duration can be used to make inferences about students' reading behavior in a CSCL context where it is impossible to observe directly. It also shows the power of a multimodal approach to the data-here, we are using one mode of data (discussion) to get a better understanding of another mode (clickstream).
When assessing scientific reasoning both (1) modeling connections in the discourse and (2) doing ... more When assessing scientific reasoning both (1) modeling connections in the discourse and (2) doing so at an appropriate grain size can be challenging for researchers. Our study suggests combining a novel theoretical and a novel methodological framework to respond to these challenges by detecting epistemic networks of scientific reasoning processes in the context of collaborative vs individual problem solving of pre-service teachers. We investigated (1) whether the combination of these frameworks can be fruitfully applied to model scientific reasoning processes and what unit of analysis researchers or instructors should choose to answer questions of interest. One novel aspect of our study is that we compared epistemic networks in case of collaborative vs individual reasoning processes. Our results show that (1) epistemic networks of scientific reasoning can reliably capture reasoning processes when comparing collaborative vs individual reasoning; and (2) propositional and potentially larger units might be considered as "optimal" units of analysis to detect such differences.
Research on computer-supported collaborative learning (CSCL) is often concerned with the question... more Research on computer-supported collaborative learning (CSCL) is often concerned with the question of how scaffolds or other characteristics of learning may affect learners' social and cognitive engagement. Such engagement in socio-cognitive activities frequently materializes in discourse. In quantitative analyses of discourse, utterances are typically coded, and differences in the frequency of codes are compared between conditions. However, such traditional coding-and-counting-based strategies neglect the temporal nature of verbal data, and therefore provide limited and potentially misleading information about CSCL activities. Instead, we argue that analyses of the temporal proximity, specifically temporal co-occurrences of codes, provide a more appropriate way to characterize socio-cognitive activities of learning in CSCL settings. We investigate this claim by comparing and contrasting a traditional coding-andcounting analysis with epistemic network analysis (ENA), a discourse analysis technique that models temporal co-occurrences of codes in discourse. We apply both methods to data from a study that compared the effects of individual vs. collaborative problem solving. The results suggest that compared to a traditional coding-and-counting approach, ENA provides more insight into the socio-cognitive learning activities of students.
When assessing scientific reasoning both (1) modeling connections in the discourse and (2) doing ... more When assessing scientific reasoning both (1) modeling connections in the discourse and (2) doing so at an appropriate grain size can be challenging for researchers. Our study suggests combining a novel theoretical and a novel methodological framework to respond to these challenges by detecting epistemic networks of scientific reasoning processes in the context of collaborative vs individual problem solving of pre-service teachers. We investigated (1) whether the combination of these frameworks can be fruitfully applied to model scientific reasoning processes and what unit of analysis researchers or instructors should choose to answer questions of interest. One novel aspect of our study is that we compared epistemic networks in case of collaborative vs individual reasoning processes. Our results show that (1) epistemic networks of scientific reasoning can reliably capture reasoning processes when comparing collaborative vs individual reasoning; and (2) propositional and potentially larger units might be considered as "optimal" units of analysis to detect such differences.
Health care is fundamentally about people, and therefore, engineering approaches for studying hea... more Health care is fundamentally about people, and therefore, engineering approaches for studying healthcare systems must consider the perspective, concepts and methods offered by the human factors and ergonomics (HFE) discipline. HFE analysis is often qualitative to provide in-depth description of work systems and processes. To deepen our understanding of care processes, we propose the next level of analysis, i.e. quantification of qualitative data. Here, we describe epistemic network analysis (ENA) as a novel method to quantify qualitative data and present a case study applying ENA to assess communication in a primary care team. One high-performing primary care team consisting of a physician, nurse, medical assistant and unit clerk was observed for 15 hours. We analyzed task-allocation communications and identified the sender, receiver, synchronicity and acceptance. We used logistic regression and ENA to evaluate sender, receiver and synchronicity impact on task acceptance. The physician and unit clerk were most successful allocating tasks. Future work should consider the role of synchronous, interruptive communication as potentially useful in time-critical tasks and further investigate the role of the unit clerk. HFE researchers should consider ENA as a tool to expand and deepen their understanding of care processes by quantifying qualitative data.
This paper reports outcomes of 57 students' exploration of urban planning and environmental scien... more This paper reports outcomes of 57 students' exploration of urban planning and environmental science identities through Virtual City Planning, a course implemented in a science museum that leveraged a virtual learning environment supported by in-class play-based experiences. Identity exploration trajectories were assessed using the Projective Reflection framework, which consists of constructs that capture cognitive, affective, and behavioral features of the self in addition to learners' self-perceptions and definitions. Researchers constructed a parsimonious epistemic network that was supported by in-depth qualitative interpretations to a) visualize students' general trends of student self-reflection across the course experience and b) highlight which Projective Reflection constructs were highly nascent to participants as they engaged in identity exploration. Results further theoretical understandings of how courses designed to support identity exploration influence the sophistication and content of learners' reflections on the self and illustrate the utility of epistemic networks for visualizing identity exploration trajectories over time.
This paper explores methodological questions in the study of identity through an examination and ... more This paper explores methodological questions in the study of identity through an examination and discussion of the empirical papers in this special issue. Particular attention is paid to the ways identity is operationalized in the study of how learning environments foster changes in students' sense of self. The paper concludes that identity is a difficult construct to study in the context of learning environments because it is simultaneously performative and subjective, and these dual aspects of identity may be best operationalized in an interactional view, in which identity is conceptualized as a set of relations between aspects of identity rather than as a state that can be coded directly in data on learning.
Epistemic network analysis (ENA) has been used in more than 300 published studies to date. Howeve... more Epistemic network analysis (ENA) has been used in more than 300 published studies to date. However, there is no work in publication that describes the transformations that constitute ENA in formal mathematical terms. This paper provides such a description, focusing on the mathematical formulations that lead to two key affordances of ENA that are not present in other network analysis tools or multivari-ate analyses: (1) summary statistics that can be used to compare the differences in the content rather than the structure of networks and (2) network visualizations that provide information that is mathematically consistent with those statistics. Specifically, we describe the mathematical transformations by which ENA constructs matrix representations of discourse data, uses those representations to generate networks for units of analysis, places those networks into a metric space, identifies meaningful dimensions in the space, and positions the nodes of network graphs within that space so as to enable interpretation of those dimensions in terms of the content of the networks. We conclude with a discussion of how the mathematical formalisms of ENA can be used to model networks more generally.
Coding data-defining concepts and identifying where they occur in data-is a critical aspect of qu... more Coding data-defining concepts and identifying where they occur in data-is a critical aspect of qualitative data analysis, and especially so in quantitative ethnography. Coding is a central process for creating meaning from data, and while much has been written about coding methods and theory, relatively little has been written about what constitutes best practices for fair and valid coding, what justifies those practices, and how to implement them. In this paper, our goal is not to address these issues comprehensively, but to provide guidelines for good coding practice and to highlight some of the issues and key questions that quantitative ethnographers and other researchers should consider when coding data.
A key goal of quantitative ethnographic (QE) models, and statistical models more generally, is to... more A key goal of quantitative ethnographic (QE) models, and statistical models more generally, is to produce the most parsimonious model that adequately explains or predicts the phenomenon of interest. In epistemic network analysis (ENA), for example, this entails constructing network models with the fewest number of codes whose interaction structure provides sufficient explanatory power in a given context. Unlike most statistical models, however, modification of ENA models can affect not only the statistical properties but also the interpretive alignment between quantitative features and qualitative meaning that is a central goal in QE analyses. In this study, we propose a novel method, Parsimonious Removal with Interpretive Alignment, for systematically identifying more parsimonious ENA models that are likely to maintain interpretive alignment with an existing model. To test the efficacy of the method, we implemented it on a well-studied dataset for which there is a published, validated ENA model, and we show that the method successfully identifies reduced models likely to maintain explanatory power and interpretive alignment.
We propose an extension to epistemic network analysis (ENA) that enables researchers to model the... more We propose an extension to epistemic network analysis (ENA) that enables researchers to model the nested effects of multiple grouping variables in an ENA context.
Quantitative ethnographers across a range of domains study complex collaborative thinking (CCT): ... more Quantitative ethnographers across a range of domains study complex collaborative thinking (CCT): the processes by which members of a group or team develop shared understanding by making cognitive connections from the statements and actions of the group. CCT is difficult to model because the actions of group members are interdependent-the activity of any individual is influenced by the actions of other members of the group. Moreover, the actions of group members engaged in some collaborative tasks may need to follow a particular order. However, current techniques can account for either interdependence or order, but not both. In this paper, we present directed epistemic network analysis (dENA), an extension of epistemic network analysis (ENA), as a method that can simultaneously account for the interdependent and ordered aspects of CCT. To illustrate the method, we compare a qualitative analysis of two U.S. Navy commanders working in a simulation to ENA and dENA analyses of their performance. We find that by accounting for interdependence but not order, ENA was not able to model differences between the commanders seen in the qualitative analysis, but by accounting for both interdependence and order, dENA was able to do so.
this study explores the concept of authenticity in education, which has been, over the last 25 ye... more this study explores the concept of authenticity in education, which has been, over the last 25 years, a powerful metaphor for educational practice, particularly as a guiding principle for some technological innovations that support student learning. the concept of authenticity has a variety of meanings , although a dominant interpretation is that authenticity refers to the alignment of educational practice with activities in the real world. Based on a meta-analysis of research on the topic, Shaffer and resnick (1999) proposed unifying the array of meanings for authenticity with their concept of thick authenticity, which included four different aspects of authentic learning: real-world, assessment, disciplinary, and personal authenticity. thick authenticity synthesizes these different facets into a mutually interdependent whole. this study revisits this discussion two decades later, again conducting a meta-review of recent literature, and re-finding the four meanings for authenticity. however, we also identify in the current literature on authenticity a fifth meaning: teacher authenticity. We then analyze two cases of technology-based teaching to explore how teachers manage the elements of authenticity in their classrooms, what this tells us about the theoretical construct of thick authenticity, and specifically the
We analyze pre-service teachers' diagnostic argumentations in learning with simulated cases by ex... more We analyze pre-service teachers' diagnostic argumentations in learning with simulated cases by exploring relations within and between the two dimensions of domain-specific conceptual knowledge and cross-domain epistemic activities. Using the method of ENA, we conclude that both dimensions are relevant in pre-service teachers' diagnostic argumentation and hence in designing learning interventions for its facilitation. The results also suggest that conceptual knowledge better explains differences between accurately vs. inaccurately diagnosing pre-service teachers compared to the dimension of epistemic activities.
Zachari Swiecki is a Ph.D. student in the Department of Educational Psychology (Learning Sciences... more Zachari Swiecki is a Ph.D. student in the Department of Educational Psychology (Learning Sciences Program) at the University of Wisconsin-Madison. His research focuses on modeling complex, collaborative thinking.
Health care is fundamentally about people, and therefore, engineering approaches for studying hea... more Health care is fundamentally about people, and therefore, engineering approaches for studying health-care systems must consider the perspective, concepts and methods offered by the human factors and ergonomics (HFE) discipline. HFE analysis is often qualitative to provide in-depth description of work systems and processes. To deepen our understanding of care processes, we propose the next level of analysis; i.e., quantification of qualitative data. Here, we describe epistemic network analysis (ENA) as a novel method to quantify qualitative data and present a case study applying ENA to assess communication in a primary care team. One high-performing primary care team, consisting of a physician, nurse, medical assistant and unit clerk, was observed for 15 hours. We analyzed task-allocation communications and identified the sender, receiver, synchronicity and acceptance. We used logistic regression and ENA to evaluate sender, receiver and synchronicity impact on task acceptance. The physician and unit clerk were most successful at allocating tasks. Future work should consider the role of synchronous, interruptive communication as potentially useful in time-critical tasks and further investigate the role of the unit clerk. HFE researchers should consider ENA as a tool to expand and deepen their understanding of care processes by quantifying qualitative data.
Collaborative Problem Solving (CPS) is an interactive, interdependent, and temporal process. Howe... more Collaborative Problem Solving (CPS) is an interactive, interdependent, and temporal process. However, current methods for measuring the CPS processes of individuals, such as coding and counting, treat these processes as sets of isolated and independent events. In contrast, Epistemic Network Analysis (ENA) models how the contributions of a given individual relate to the contributions of others. This article examines the communications of air defense warfare teams from an experiment comparing two different computer-based decision support systems, using this data to ask whether ENA provides a more ecologically valid quantitative model of CPS than coding and counting. Qualitative analysis showed that commanders using one system asked questions to understand the tactical situation, while commanders using an experimental system focused more on actions in response to the tactical situation. Neither of the coding and counting approaches we tested corroborated these findings with statistically significant results. In contrast, ENA created models of the individual contributions of commanders that (a) showed statistical differences between commanders using the two systems to corroborate the qualitative analysis, and (b) revealed differences in individual performance. This suggests that ENA is a more powerful tool for CPS assessment than coding and counting approaches.
Many researchers have argued that models of collaborative processes should account for temporalit... more Many researchers have argued that models of collaborative processes should account for temporality, but there exist different approaches for doing so. We compared two specific approaches to modeling collaborative processes in a CSCL context: Epistemic Network Analysis, which models events cotemporally (unordered and temporally proximate), and Sequential Pattern Mining, which models events sequentially (ordered and temporally proximate). Our results suggest that in this context cotemporal models constructed with Epistemic Network Analysis outperform sequential models constructed with Sequential Pattern Mining in terms of (a) explanatory power, (b) efficiency, and (c) interpretability.
The process of labeling, categorizing, or otherwise annotating data-or coding in the computer-sup... more The process of labeling, categorizing, or otherwise annotating data-or coding in the computer-supported collaborative learning (CSCL) literature-is a fundamental process in CSCL research. It is the process by which researchers identify salient properties about segments of CSCL data: what they are, what they contain, or what they mean. Coding, like all processes in research, is subject to error. To reduce the potential impact of coding error, CSCL researchers typically measure inter-rater reliability (IRR). However, there is no extant method to determine what level of IRR would invalidate a CSCL result or model. One way of assessing the potential impact of such inaccuracies is by conducting sensitivity analyses, which measure the level of error that would need to be present in the data to invalidate a given inference. This paper introduces a new method for conducting sensitivity analyses in CSCL: the Binary Replicate Test.
Educators who develop and use place-based curricula argue that engaging students in authentic pro... more Educators who develop and use place-based curricula argue that engaging students in authentic problem-based learning situated in students' own local place helps them understand the relevance of their academic work, which improves learning. However, while place-based curricula are localized, they are also authentic, problem-based experiences, which have been shown to be effective in their own right. Without controlling for the impact of authenticity, it is not clear whether localization itself improves student learning. In this study, we used a place-based simulation to engage students with authentic problem-based learning about a specific place: a city that is local for some students and not for others. We assessed differences in cognitive, affective, and behavioral outcomes between these two groups. Our results suggest that learning outcomes are better when students engage in a place-based simulation about their own locality, indicating that place does play a critical role in the success of placebased education.
This paper examines whether and to what extent long and short readers make different contribution... more This paper examines whether and to what extent long and short readers make different contributions to collaborative design discussions in a CSCL environment-that is, we investigate whether a simple measure of reading behavior based on clickstream data is a good proxy for engagement with readings. Our approach to addressing this question is multimodal, involving two sources of data: (a) a record of students' online conversations, and (b) the frequency and duration with which documents were open on each student's screen. This study suggests that in this specific case, relatively thin data about reading frequency and mean reading duration can be used to make inferences about students' reading behavior in a CSCL context where it is impossible to observe directly. It also shows the power of a multimodal approach to the data-here, we are using one mode of data (discussion) to get a better understanding of another mode (clickstream).
When assessing scientific reasoning both (1) modeling connections in the discourse and (2) doing ... more When assessing scientific reasoning both (1) modeling connections in the discourse and (2) doing so at an appropriate grain size can be challenging for researchers. Our study suggests combining a novel theoretical and a novel methodological framework to respond to these challenges by detecting epistemic networks of scientific reasoning processes in the context of collaborative vs individual problem solving of pre-service teachers. We investigated (1) whether the combination of these frameworks can be fruitfully applied to model scientific reasoning processes and what unit of analysis researchers or instructors should choose to answer questions of interest. One novel aspect of our study is that we compared epistemic networks in case of collaborative vs individual reasoning processes. Our results show that (1) epistemic networks of scientific reasoning can reliably capture reasoning processes when comparing collaborative vs individual reasoning; and (2) propositional and potentially larger units might be considered as "optimal" units of analysis to detect such differences.
Research on computer-supported collaborative learning (CSCL) is often concerned with the question... more Research on computer-supported collaborative learning (CSCL) is often concerned with the question of how scaffolds or other characteristics of learning may affect learners' social and cognitive engagement. Such engagement in socio-cognitive activities frequently materializes in discourse. In quantitative analyses of discourse, utterances are typically coded, and differences in the frequency of codes are compared between conditions. However, such traditional coding-and-counting-based strategies neglect the temporal nature of verbal data, and therefore provide limited and potentially misleading information about CSCL activities. Instead, we argue that analyses of the temporal proximity, specifically temporal co-occurrences of codes, provide a more appropriate way to characterize socio-cognitive activities of learning in CSCL settings. We investigate this claim by comparing and contrasting a traditional coding-andcounting analysis with epistemic network analysis (ENA), a discourse analysis technique that models temporal co-occurrences of codes in discourse. We apply both methods to data from a study that compared the effects of individual vs. collaborative problem solving. The results suggest that compared to a traditional coding-and-counting approach, ENA provides more insight into the socio-cognitive learning activities of students.
When assessing scientific reasoning both (1) modeling connections in the discourse and (2) doing ... more When assessing scientific reasoning both (1) modeling connections in the discourse and (2) doing so at an appropriate grain size can be challenging for researchers. Our study suggests combining a novel theoretical and a novel methodological framework to respond to these challenges by detecting epistemic networks of scientific reasoning processes in the context of collaborative vs individual problem solving of pre-service teachers. We investigated (1) whether the combination of these frameworks can be fruitfully applied to model scientific reasoning processes and what unit of analysis researchers or instructors should choose to answer questions of interest. One novel aspect of our study is that we compared epistemic networks in case of collaborative vs individual reasoning processes. Our results show that (1) epistemic networks of scientific reasoning can reliably capture reasoning processes when comparing collaborative vs individual reasoning; and (2) propositional and potentially larger units might be considered as "optimal" units of analysis to detect such differences.
Health care is fundamentally about people, and therefore, engineering approaches for studying hea... more Health care is fundamentally about people, and therefore, engineering approaches for studying healthcare systems must consider the perspective, concepts and methods offered by the human factors and ergonomics (HFE) discipline. HFE analysis is often qualitative to provide in-depth description of work systems and processes. To deepen our understanding of care processes, we propose the next level of analysis, i.e. quantification of qualitative data. Here, we describe epistemic network analysis (ENA) as a novel method to quantify qualitative data and present a case study applying ENA to assess communication in a primary care team. One high-performing primary care team consisting of a physician, nurse, medical assistant and unit clerk was observed for 15 hours. We analyzed task-allocation communications and identified the sender, receiver, synchronicity and acceptance. We used logistic regression and ENA to evaluate sender, receiver and synchronicity impact on task acceptance. The physician and unit clerk were most successful allocating tasks. Future work should consider the role of synchronous, interruptive communication as potentially useful in time-critical tasks and further investigate the role of the unit clerk. HFE researchers should consider ENA as a tool to expand and deepen their understanding of care processes by quantifying qualitative data.
Uploads
Papers by David Shaffer