Visual cognition during real social interaction
Gustav Kuhn2012, Frontiers in Human Neuroscience
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
Laboratory studies of social visual cognition often simulate the critical aspects of joint attention by having participants interact with a computer-generated avatar. Recently, there has been a movement toward examining these processes during authentic social interaction. In this review, we will focus on attention to faces, attentional misdirection, and a phenomenon we have termed social inhibition of return (Social IOR), that have revealed aspects of social cognition that were hitherto unknown. We attribute these discoveries to the use of paradigms that allow for more realistic social interactions to take place. We also point to an area that has begun to attract a considerable amount of interest-that of Theory of Mind (ToM) and automatic perspective taking-and suggest that this too might benefit from adopting a similar approach.
Related papers
The dual function of social gaze
Cognition, 2015
Ears cannot speak, lips cannot hear, but eyes can both signal and perceive. For human beings, this dual function makes the eyes a remarkable tool for social interaction. For psychologists trying to understand eye movements, however, their dual function causes a fundamental ambiguity. In order to contrast signaling and perceiving functions of social gaze, we manipulated participants’ beliefs about social context as they looked at the same stimuli. Participants watched videos of faces of higher and lower ranked people, while they themselves were filmed. They believed either that the recordings of them would later be seen by the people in the videos or that no-one would see them. This manipulation significantly changed how participants responded to the social rank of the target faces. Specifically, when they believed that the targets would later be looking at them, and so could use gaze to signal information, participants looked proportionally less at the eyes of the higher ranked targets. We conclude that previous claims about eye movements and face perception that are based on a single social context can only be generalized with caution. A complete understanding of face perception needs to address both functions of social gaze.
Neural Correlates, Visual Attention and Facial Expression during Social Interaction with Virtual Others
We present two recent studies which explore the biological basis of social interaction with virtual characters. Anthropomorphic virtual characters were presented which appeared moving on-screen and turned either towards the participant or towards a third party who is out of view. In dynamic animations, virtual characters then exhibited FACS-coded facial expressions, which were either socially relevant (i.e., indicative of the intention to establish interpersonal contact) or arbitrary. These four conditions thus established a two-by-two factorial design. This paradigm was developed for the purpose of a functional magnetic resonance imaging (fMRI) study and a study recording eye movements and facial muscle activity (EMG). Functional neuroimaging revealed that medial prefrontal activation is observed not only during one's own personal involvement in social interaction -as indicated by adequate facial expressions -but also during the experience of an interaction between the virtual character and a third other. Similarly, differential EMG activity was observed not only when the virtual characters smiled towards the human observer, but also when the smiles were directed towards someone else. In contrast, eye movements of human participants showed that the intensity of visual attention as manifested in visual fixation duration is specifically related to having eye-to-eye contact with a virtual other. In sum, the data from these two studies demonstrate a clear-cut difference between visual attention and neuro-and electrophysiological correlates depending upon the observer's personal involvement (i.e., adopting a secondperson perspective) versus being a passive by-stander (i.e., adopting a third-person perspective). We conclude with a discussion of the evolutionary significance of these results.
Clarifying the role of theory of mind areas during visual perspective taking: Issues of spontaneity and domain-specificity
NeuroImage, 2015
Visual perspective taking is a fundamental feature of the human social brain. Previous research has mainly focused on explicit visual perspective taking and contrasted brain activation for other- versus self-perspective judgments. This produced a conceptual gap to theory of mind studies, where researchers mainly compared activation for taking another's mental perspective to non-mental control conditions. We compared brain activation for visual perspective taking to activation for non-mental control conditions where the avatar was replaced by directional (arrow, lamp) or non-directional (brick-wall) objects. We found domain-specific activation linked to the avatar's visual perspective in right TPJ, ventral mPFC and ventral precuneus. Interestingly, we found that these areas are spontaneously processing information linked to the other's perspective during self-perspective judgments. Based on a review of the visual perspective taking literature, we discuss how these finding...
Theory of mind affects the interpretation of another person's focus of attention
Scientific Reports, 2021
People are drawn to social, animate things more than inanimate objects. Previous research has also shown gaze following in humans, a process that has been linked to theory of mind (ToM). In three experiments, we investigated whether animacy and ToM are involved when making judgements about the location of a cursor in a scene. In Experiment 1, participants were told that this cursor represented the gaze of an observer and were asked to decide whether the observer was looking at a target object. This task is similar to that carried out by researchers manually coding eye-tracking data. The results showed that participants were biased to perceive the gaze cursor as directed towards animate objects (faces) compared to inanimate objects. In Experiments 2 and 3 we tested the role of ToM, by presenting the same scenes to new participants but now with the statement that the cursor was generated by a ‘random’ computer system or by a computer system designed to seek targets. The bias to report...
Sociality, Attention, and the Mind's Eye
Axel Seemann (ed.), Joint attention. Perspectives and Developments, MIT Press, Cambridge MA, January 2012, pp. 205-242.
Eyes on the mind: Investigating the influence of gaze dynamics on the perception of others in real-time social interaction
2012
Social gaze provides a window into the interests and intentions of others and allows us to actively point out our own. It enables us to engage in triadic interactions involving human actors and physical objects and to build an indispensable basis for coordinated action and collaborative efforts. The object-related aspect of gaze in combination with the fact that any motor act of looking encompasses both input and output of the minds involved makes this non-verbal cue system particularly interesting for research in embodied social cognition. Social gaze comprises several core components, such as gaze-following or gaze aversion. Gaze-following can result in situations of either “joint attention” or “shared attention.” The former describes situations in which the gaze-follower is aware of sharing a joint visual focus with the gazer. The latter refers to a situation in which gazer and gaze-follower focus on the same object and both are aware of their reciprocal awareness of this joint focus. Here, a novel interactive eye-tracking paradigm suited for studying triadic interactions was used to explore two aspects of social gaze. Experiments 1a and 1b assessed how the latency of another person’s gaze reactions (i.e., gaze-following or gaze version) affected participants’ sense of agency, which was measured by their experience of relatedness of these reactions. Results demonstrate that both timing and congruency of a gaze reaction as well as the other’s action options influence the sense of agency. Experiment 2 explored differences in gaze dynamics when participants were asked to establish either joint or shared attention. Findings indicate that establishing shared attention takes longer and requires a larger number of gaze shifts as compared to joint attention, which more closely seems to resemble simple visual detection. Taken together, novel insights into the sense of agency and the awareness of others in gaze-based interaction are provided.
There is more to eye contact than meets the eye
Cognition, 2015
Recent studies have shown enhanced brain and autonomic responses to seeing a face with a direct gaze. Interestingly, greater responses to eye contact vs. averted gaze have been observed when showing "live" faces as stimuli but not when showing pictures of faces on a computer screen. In this study, we provide unequivocal evidence that the differential responses observed in the "live" condition are dependent on the observer's mental attributions. Results from two experiments showed that eye contact resulted in greater autonomic and brain responses compared to averted gaze if a participant believed that the stimulus person sitting on the other side of an electronic shutter was able to see him or her through the shutter. Gaze direction had no effects if participants believed that the transparency from their side was blocked. The results suggest that mental attributions exert a powerful modulation on the processing of socially relevant sensory information.
Editorial: Reading Faces and Bodies: Behavioral and Neural Processes Underlying the Understanding of, and Interaction with, Others
Frontiers in psychology, 2016
A non-verbal Turing Test: Differentiating mind from machine in gaze-based social interaction
2011
In social interaction, gaze behavior provides important signals that have a significant impact on our perception of others. Previous investigations, however, have relied on paradigms in which participants are passive observers of other persons’ gazes and do not adjust their gaze behavior as is the case in real-life social encounters. We used an interactive eye-tracking paradigm that allows participants to interact with an anthropomorphic virtual character whose gaze behavior is responsive to where the participant looks on the stimulus screen in real time. The character’s gaze reactions were systematically varied along a continuum from a maximal probability of gaze aversion to a maximal probability of gaze-following during brief interactions, thereby varying contingency and congruency of the reactions. We investigated how these variations influenced whether participants believed that the character was controlled by another person (i.e., a confederate) or a computer program. In a series of experiments, the human confederate was either introduced as naïve to the task, cooperative, or competitive. Results demonstrate that the ascription of humanness increases with higher congruency of gaze reactions when participants are interacting with a naïve partner. In contrast, humanness ascription is driven by the degree of contingency irrespective of congruency when the confederate was introduced as cooperative. Conversely, during interaction with a competitive confederate, judgments were neither based on congruency nor on contingency. These results offer important insights into what renders the experience of an interaction truly social: Humans appear to have a default expectation of reciprocation that can be influenced drastically by the presumed disposition of the interactor to either cooperate or compete.
Attribution of vision and knowledge in ‘spontaneous perspective taking’
Psychological Research, 2019
Theory of mind is a ubiquitous notion that permeates many aspects of social cognition. A recent application has been in the context of 'spontaneous perspective taking' in which responses to target stimuli are facilitated if a human agent, present in a display, sees the same stimuli as an experimental participant. In the present work, we replicated results from a paradigm purporting to show such perspective taking in which participants tend to judge an ambiguous number from the position of an agent. We find, however (in Experiment 1) that this effect still occurs even when the agent cannot see the number due to an occluding object. This, therefore, does not support the perspective-taking hypothesis. An alternative explanation to the theory of mind account is posited in which the agent acts as a reference point that cues the observer to view the critical stimuli from that position and in the direction to which the agent faces. We test this hypothesis in Experiments 2 and 3, and show that non-human reference points can generate perspective-taking-like data. Overall, these results do not support the theory of mind account of previous studies.
Related papers
Potential social interactions are important to social attention
Proceedings of the …, 2011
Is there a 'social brain'? Lessons from eye-gaze following, joint attention, and autism
The aim of this volume is to explore what it means to understand other minds. Drawing on philosophical, developmental, and psychological perspectives, the chapters in this book address a myriad of issues including how an understanding of minds develops, its significance for social interaction, and its relation to other social and cognitive achievements. Our chapter brings yet another perspective to bear on these issues. Taking a neuroscientific approach, our discussion centers on how understanding other minds -a central aspect of social-information processing -is represented in the brain.
Two takes on the social brain: A comparison of theory of mind tasks
Journal of Cognitive Neuroscience, 2007
& We compared two tasks that are widely used in research on mentalizing-false belief stories and animations of rigid geometric shapes that depict social interactions-to investigate whether the neural systems that mediate the representation of others' mental states are consistent across these tasks. Whereas false belief stories activated primarily the anterior paracingulate cortex (APC), the posterior cingulate cortex/precuneus (PCC/ PC), and the temporo-parietal junction (TPJ)-components of the distributed neural system for theory of mind (ToM)-the social animations activated an extensive region along nearly the full extent of the superior temporal sulcus, including a locus in the posterior superior temporal sulcus (pSTS), as well as the frontal operculum and inferior parietal lobule (IPL)components of the distributed neural system for action under-standing-and the fusiform gyrus. These results suggest that the representation of covert mental states that may predict behavior and the representation of intentions that are implied by perceived actions involve distinct neural systems. These results show that the TPJ and the pSTS play dissociable roles in mentalizing and are parts of different distributed neural systems. Because the social animations do not depict articulated body movements, these results also highlight that the perception of the kinematics of actions is not necessary to activate the mirror neuron system, suggesting that this system plays a general role in the representation of intentions and goals of actions. Furthermore, these results suggest that the fusiform gyrus plays a general role in the representation of visual stimuli that signify agency, independent of visual form. & D
Social communication with virtual agents: The effects of body and gaze direction on attention and emotional responding in human observers
International journal of psychophysiology : official journal of the International Organization of Psychophysiology, 2015
In social communication, the gaze direction of other persons provides important information to perceive and interpret their emotional response. Previous research investigated the influence of gaze by manipulating mutual eye contact. Therefore, gaze and body direction have been changed as a whole, resulting in only congruent gaze and body directions (averted or directed) of another person. Here, we aimed to disentangle these effects by using short animated sequences of virtual agents posing with either direct or averted body or gaze. Attention allocation by means of eye movements, facial muscle response, and emotional experience to agents of different gender and facial expressions were investigated. Eye movement data revealed longer fixation durations, i.e., a stronger allocation of attention, when gaze and body direction were not congruent with each other or when both were directed towards the observer. This suggests that direct interaction as well as incongruous signals increase th...
Seeing minds: A neurophilosophical investigation of the role of perception-action coupling in social perception
Social Neuroscience, 2011
Examining the Sociality of the Joint Attention effect: Gaze Cues, Social significance and the Role of Face Identity in the Function of Joint Attention.
The function of joint attention (JA) is of crucial biological and social significance, as animal, developmental and neuropsychological studies show. Only recently though has the interest of attentional researchers been drawn to this effect. Numerous attention studies, by employing human gaze cues, demonstrate the unique effects elicited by such cues (i.e. prolonged facilitation and late inhibition), which are attributed to the heavy social significance of gaze cues. The present study aimed at examining whether social significance is, or is the only, source of such cueing effects, by, for the first time, employing animal in addition to human gaze stimuli in a JA paradigm (Experiment 1). Also, in contrast to evidence suggesting that face identity does not influence the JA effect, the authors manipulated different aspects of human face cues’ identity (i.e. trustworthiness evoked by a neutral face and familiarity with a face; Experiments 2 and 3), to gain a more in-depth understanding of how sociality of a face, which is inherent in that face’s identity, may account for the unique gaze cueing effects. Although results were mixed, overall it is suggested that social significance as well as specific properties of the identity of neutral face cues could modulate the function of joint attention. Limitations and future directions are further discussed.
Mental-state attribution drives rapid, reflexive gaze following
Attention, Perception, & Psychophysics, 2010
When presented with a face stimulus whose gaze is diverted, observers' attention shifts to locations fixated by the face. Such "gaze following" has been characterized by some previous studies as a consequence of sophisticated theory of mind processes, but by others (particularly those employing the "gaze-cuing" paradigm) as an involuntary response that is triggered directly and reflexively by the physical features of a face. To address this apparent contradiction, we modified the gaze-cuing paradigm using a deception procedure to convince observers that prerecorded videos of an experimenter making head turns and wearing mirrored goggles were a "live" video link to an adjacent room. In two experiments, reflexive gaze following was found when observers believed that the model was wearing transparent goggles and could see, but it was significantly reduced when they believed that the experimenter wore opaque goggles and could not see. These results indicate that the attribution of the mental state "seeing" to a face plays a role in controlling even reflexive gaze following.
When Computer Vision Gazes at Cognition
Joint attention is a core, early-developing form of social interaction. It is based on our ability to discriminate the third party objects that other people are looking at. While it has been shown that people can accurately determine whether another person is looking directly at them versus away, little is known about human ability to discriminate a third person gaze directed towards objects that are further away, especially in unconstraint cases where the looker can move her head and eyes freely. In this paper we address this question by jointly exploring human psychophysics and a cognitively motivated computer vision model, which can detect the 3D direction of gaze from 2D face images. The synthesis of behavioral study and computer vision yields several interesting discoveries. (1) Human accuracy of discriminating targets 8{\deg}-10{\deg} of visual angle apart is around 40% in a free looking gaze task; (2) The ability to interpret gaze of different lookers vary dramatically; (3) T...
From Gaze Perception to Social Cognition: The Shared-Attention System
Perspectives on Psychological Science
When two people look at the same object in the environment and are aware of each other’s attentional state, they find themselves in a shared-attention episode. This can occur through intentional or incidental signaling and, in either case, causes an exchange of information between the two parties about the environment and each other’s mental states. In this article, we give an overview of what is known about the building blocks of shared attention (gaze perception and joint attention) and focus on bringing to bear new findings on the initiation of shared attention that complement knowledge about gaze following and incorporate new insights from research into the sense of agency. We also present a neurocognitive model, incorporating first-, second-, and third-order social cognitive processes (the shared-attention system, or SAS), building on previous models and approaches. The SAS model aims to encompass perceptual, cognitive, and affective processes that contribute to and follow on f...
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.