Human Technology Centre (HumTec)

... In C. Forceville and E. Urios-Aparisi (eds.), Multimodal Metaphor. Mouton de Gruyter, Berlin (2009), 329-358. 13. ... O'Reilly Media, 2008. 16. Wobbrock, JO, Morris, MR and Wilson, AD User-defined gestures for surface computing. Proc. of the 27th Int. Conf. ...

This paper explores if people perceive and perform touchless gestures differently when communicating with technology vs. with humans. Qualitative reports from a lab study of 10 participants revealed that people perceive differences in the speed of performing gestures, sense of enjoyment, feedback from the communication target. Preliminary analysis of 1200 gesture trials of motion capture data showed that hand shapes were less taut when communicating to technology. These differences provide implications for the design of gestural user interfaces that use symbolic gestures borrowed from human multimodal communication.

In "Two heads are better than one," "head" stands for people and focuses the message on the intelligence of people. This is an example of figurative language through metonymy, where substituting a whole entity by one of its parts focuses attention on a specific aspect of the entity. Whereas metaphors, another figurative language device, are substitutions based on similarity, metonymy involves substitutions based on associations. Both are figures of speech but are also expressed in coverbal gestures during multimodal communication. The closest neuropsychological studies of metonymy in gestures have been nonlinguistic tool-use, illustrated by the classic apraxic problem of body-part-as-object (BPO, equivalent to an internal metonymy representation of the tool) vs. pantomimed action (external metonymy representation of the absent object/tool). Combining these research domains with concepts in cognitive linguistic research on gestures, we conducted an fMRI study to investigate metonymy resolution in coverbal gestures. Given the greater difficulty in developmental and apraxia studies, perhaps explained by the more complex semantic inferencing involved for external metonymy than for internal metonymy representations, we hypothesized that external metonymy resolution requires greater processing demands and that the neural resources supporting metonymy resolution would modulate regions involved in semantic processing. We found that there are indeed greater activations for external than for internal metonymy resolution in the temporoparietal junction (TPJ). This area is posterior to the lateral temporal regions recruited by metaphor processing. Effective connectivity analysis confirmed our hypothesis that metonymy resolution modulates areas implicated in semantic processing. We interpret our results in an interdisciplinary view of what metonymy in action can reveal about abstract cognition.

This chapter starts from the observation that metaphoric understandings expressed monomodally through gesture tend to rely on "primary metaphors" (Grady 1997a). Asserting that gestures draw on basic, experientially motivated, embodied construal operations, we detail how primary scenes and subscenes (Grady & Johnson 2002), image and force schemas, metonymy, and frames (Fillmore 1982) interact in situated meaning-making. We propose that by shifting the focus from object-oriented schemas, source domains, and mappings to what we call "source actions" and "embodied action frames," we can account for the pragmatically minded nature and specific mediality of communicative gestural acts integrated in natural multimodal discourse. We argue that coverbal gestures recruit frame structures metonymically, singling out elements of "scenes" (Fillmore 1977), especially those underpinning correlated metaphoric meanings. We back up our theoretical claims with evidence from neuroscientific studies and outline a frame-based approach that helps trace avenues for further research into embodied cognition and multimodal discourse processes.