Intersemiotic translation: meaning-making in film and musical art

Proceedings of the 10th World Congress of the International Association For Semiotic Studies Recurso Electronico Culture of Communication Communication of Culture Culture De La Communication Communication De La Culture Cultura De La Comunicacion Comunicacion De La Cultura 2012 Isbn 978 84 9749 522 6, 2012

Humans-and artists-are capable of complex semiotic and experiential translations: we can translate reality, be it nature or culture, into an experience of sound, image, touch, smell, mime, emotion. We speak about 'semiotic' translation when we translate artistic work into another semiotic medium, when we translate the outer conditions of the experience-e.g. translating a novel into a movie, a painting into a song, a performance into a narrative. But before experiencing semiotic translation, humans experience 'synesthetic' translation, moving beyond different possibilities of inner responses towards an artistic setting: for example by way of feeling, smelling, or hearing what can be seen. One holistic perception of artistic expression can be transposed into different modes, each one enriching the other. Both translations originate in the human possibilities of multimodal experience (e.g. blending theory of Turner and Fauconnier), cognitive fluidity (Mythen 1998)-both theories referring to complex neurological responses. Since long, semiotics has analyzed translation on the level of the sign or sign system, neglecting the (neurological) origins/counterparts of these processes. This presentation will consider the importance of the human synesthetic possibilities and integrate these into a broader account of semiotic theory (Kress 1998, p76). It will analyse the complex experience of an artistic manifestation realized by way of an 'outer' and an 'inner' semiotic translation: at one hand an 'outer' complex comprehension which encompasses memory, cultural and aesthetic conventions, personal narratives, knowledge and expectations (Zeki & Lamb 1994), at the other hand the 'inner', rich synesthetic and multimodal cognitive and kinesthetic responses in body and brain acknowledged by recent neuropsychological and cognitive research (Spence 2001, Edelman & Tononi 2000, Thibault 2004).

Cognitive Semiotics, 2009

Integrative Psychological and Behavioral Science, 2010

Taking into account that feeling is “the critical mediating process of the person-world relationships” (Josephs, Theory & Psychology 10(6):815, 2000), this article focuses on the artistic symbolic object as constraints that direct someone’s feelings. Johansen (2010) states that the literary discourse “is designed to arousing and forming the feelings of listeners and readers” (p. 185). Distancing from strict literary production, I’ve used the testimony of the Brazilian songwriter, composer and performer, Tom Zé (2003), in order to discuss the intersubjective aspect of feelings articulation in his artistic work. Is proposed that the creative process of a symbolic object, which can be considered art, is a circumstance of a most general intersubjective-cultural process in which novel objects are built. If the specificity of art is to give a symbolic shape to human feeling (cf. Langer 1953), I argue that it is a sort of mediation which allows otherness to elaborate their affections through its objective guidance. In contrast with the scientific method of objective creation that is an effort for silencing contradictions (cf. Stengers 2002), the object of art remains open to multiple interpretations, stimulating the other to recursively speak and feel through it.

Frontiers in communication, 2024

Condillac's (1754) "Traité des sensations" is the philosophical background of modern discussions on the relationship between perception and multimodal communication. The differences between perception and communication and the transitions between them are discussed with a focus on odor and color. It becomes clear that even at this primary level, the complex interactions of different modalities are the precondition for effective and rich communication. The second part discusses Cassirer's "Philosophy of Symbolic Forms" as a relevant framework for multimodality studies. Basic aspects are first commented on with a focus on music and visual art. The interaction is even more complex and rich in the case of language; the difficulty of large symbolic forms is mainly due to semantic composition and only to a lesser degree to syntactic concatenation. The first must merge/blend different semantic spaces. It must allow for the plurality of levels of integration from the lexical level, the level of phrases and sentences, up to texts and discourse. The third part focuses on multimodality in film. It treats the representation of movement and action in (film) narratives, the visual perception and representation/communication of movement and action, and the integration of music, moving images, and language.

The intermedial and meta-referential concepts are theoretically more intensely developed during the latest decades. The term “meta-reference” was defined by Werner Wolf and it is meant to designate the variety of meta-phenomena, and, as the author states, to provide „heuristically motivated umbrella term for all meta-phenomena occurring in the arts and media.”(Wolf: 2009, 135). The artistic and applied phenomenon exists since a distant historic epoch. Frequently met and commented upon in diverse arts (literature, painting, theatre, sculpture, music, photography, architecture), they have appeared in the filmic or theatrical art. Recent studies are devoted to this area. Metareferenciality is a theoretic aspect widely debated and analyzed. From a semiotic point of view, it was established that verbal signs can be explicitly and implicitly meta-referential, while non-verbal signs have only implicitly this quality. This contribution focuses on films, playwrights and screened novels. Some of these creations deal explicitly the theoretic issues mentioned above, but they also contain default elements with a symbolic value that the viewer must decipher. The contribution highlights and comments on other poetic, rhetorical and stylistic instruments of the artistic communication: exergue, parergon, anaphore, correctio, antorism. These items are subject of our analysis and interpretation. The attraction to blur aesthetic border between fiction and reality can be detected in all the commented works. The creators embrace additional insights in order to prove their modern artistic intentions. They also turn the audience into a more active and elevated communication-partner. The deliberate discontinuity of the representation level and that of reality is striking, the classical norms and standards being abolished. The screen or the stage acts like a modern mirror or a specular surface that highlights the relationship between illusion and reality, between fiction and reality. We deal with the awakening of self-consciousness of a new artistic paradigm focused on itself and its plethora of meanings besides three convergent features and artistic impulses: the illusion of representation, the vanity-theme applied to a creative act and the meta-referential artistic representation. The consequences of all these processes are materialized in a new perspective which calls for further meditation and deeper interpretation.

Transcultural studies, 2012

2009

Social Semiotics

Music and sound contributions of interpersonal meaning to film narratives may be different from or similar to meanings made by language and image, and dynamic interactions between several modalities may generate new story messages. Such interpretive potentials of music and voice sound in motion pictures are rarely considered in social semiotic investigations of intermodality. This paper therefore shares two semiotic studies of distinct and combined music, English speech and image systems in an animated short film and a promotional filmtrailer. The paper considers the impact of music and voice sound on interpretations of film narrative meanings. A music system relevant to the analysis of filmic emotion is proposed. Examples show how music and intonation contribute meaning to lexical, visual and gestural elements of the cinematic spaces. Also described are relations of divergence and resonance between emotion types in various couplings of music, intonation, words and images across story phases. The research is relevant to educational knowledge about sound, and semiotic studies of multimodality.

Cognitive and Intermedial Semiotics, 2020

, and a member of the research groups Studies on Intermediality and Intercultural Mediation (SIIM-UCM) and Biopoetics, Cognitive Semiotics and Neuroesthetics (IUIBS-ULPGC). Her academic interests revolve around the new cognitive studies in neuroesthetics, semiotics, and narratology. In particular, her research is based on investigating the importance of affective neuroscience in the invisible and visible mechanisms of human creativity. For information on her activities and publications, please visit https://www.ucm.es/ siim/marta-silvera-roig. Asunción López-Varela Azcárate is Professor at Universidad Complutense de Madrid. Her research interests are comparative literature, cultural and education studies, and cognitive and intermedial semiotics. In 2007 she established the Research Program Studies on Intermediality and Intercultural Mediation (SIIM), which has received funding from various sources for several projects. She is an external evaluator in various research programs of the European Commission and other national agencies (the Netherlands Organisation for Scientific Research (NWO) and the German Academic Exchange Service (DAAD), etc.). She is also editor in several journals in her research areas. She has served as President of the European Society of Comparative Literature and Deputy Head of the English Department at Complutense. For information on her activities and publications, please visit https://www.ucm.es/siim/asun-lopez-varela.

IASS 2010 Proceedings, 2010

wrote a canonical text for semiotics as well as for the upcoming �eld of translation studies in 1959 with the title "On linguistic aspects of translation". 1 �e text is a semiotical answer to Bertrand Russell's claim that one may only understand a word like "cheese" if one has a non-linguistic acquaintance with cheese. According to Jakobson, it should be added that one cannot understand the word "cheese" if one does not know the particular meaning the word is ascribed in English, that is, in "the lexical code of English". �ere is no signatum without signum. �e meaning of the word "'cheese'" cannot be inferred from a nonlingustic acquaintance with cheddar or with camembert without assistance of the verbal code. 2 �e meaning of the word "cheese"-or any other word-is de�nitely a linguistic fact, or to be precise, Jakobson continues, a semiotic fact. A huge amount of linguistic signs is needed to introduce a foreign word. It is not enough that someone points at the cheese in front of us, because that does not teach us whether the word "cheese" only refers to the particular one at hand or to milk products in general and so forth. �is is a problem that the philosopher W. V. Quine discusses at length in another canonical text, namely "Ontological Relativity", to which we will return. 3 �e exploration will then take a turn and move into Peircean pragmatics (a common denominator for Jakobson and Quine, indirect, it would seem, through Dewey, who was a student of Peirce) and the notion of the iconic sign in the context of the study of differences between verbal language and in this case moving images, the main topic of this presentation. Jakobson writes (with reference in fact to John Dewey's text "Peirce's theory of linguistic signs, thought, and meaning". 4 For us, both as linguists and as ordinary word-users, the meaning of any linguistic sign is its translation into some further, alternative sign,