Emotion resonance and divergence: a semiotic analysis of music and sound in 'The Lost Thing', an animated short film and 'Elizabeth' a film trailer

Music and sound contributions of interpersonal meaning to film narratives may be different from or similar to meanings made by language and image, and dynamic interactions between several modalities may generate new story messages. Such interpretive potentials of music and voice sound in motion pictures are rarely considered in social semiotic investigations of intermodality. This paper therefore shares two semiotic studies of distinct and combined music, English speech and image systems in an animated short film and a promotional filmtrailer. The paper considers the impact of music and voice sound on interpretations of film narrative meanings. A music system relevant to the analysis of filmic emotion is proposed. Examples show how music and intonation contribute meaning to lexical, visual and gestural elements of the cinematic spaces. Also described are relations of divergence and resonance between emotion types in various couplings of music, intonation, words and images across story phases. The research is relevant to educational knowledge about sound, and semiotic studies of multimodality

Deconstructing Speech: new tools for speech manipulation

My research at the London College of Communication is concerned with archives of recorded speech, what new tools need to be devised for its manipulation and how to go about this process of invention. Research into available forms of analysis of speech is discussed below with regard to two specific areas, feature vectors from linear predictive coding (LPC) analysis and hidden Markov-model-based automatic speech recognition (ASR) systems. These are discussed in order to demonstrate that whilst aspects of each may be useful in devising a system of speech-archive manipulation for artistic use. Their drawbacks and deficiencies for use in art – consequent of the reasons for their invention – necessitate the creation of tools with artistic, rather than engineering agendas in mind. It is through the initial process of devising conceptual tools for understanding speech as sound objects that I have been confronted with issues of semiotics and semantics of the voice and of the relationship between sound and meaning in speech, and of the role of analysis in mediating existing methods of communication. This is discussed with reference to Jean-Jacques Nattiez’s Music and Discourse: Towards a Semiology of Music (Nattiez 1987). The ‘trace’ – a neutral level of semiotic analysis proposed by Nattiez, far from being hypothetical as suggested by Hatten (1992: 88–98) and others, is present by analogy to many forms of mediation in modern spoken communication and the reproduction of music, and it is precisely this neutrality with regards to meaning that tools for manipulation of speech must possess, since the relationships between the sound of speech and its meaning are ‘intense’ (after Deleuze 1968

The Role of Accent in Popular Music: An Interdisciplinary Approach

In the thirty years that have passed since Peter Trudgill first published his study of British pop-song pronunciation, and fifteen years since Paul Simpson published his follow-up study of accents in pop and rock singing (1999), there have been several changes in the way linguists approach the sociolinguistics of singing. These changes include Franz Andres Morissey's introduction of sonority as a factor behind choosing particular phonological features, and the ongoing and evolving criticism of Trudgill's original assertion that singers were (and possibly still are) trying to 'imitate' Americans. The present study argues that existing theories are insufficient, and proposes a new framework for dealing with phonological choice in song, centred around three separate but unavoidably interrelated values that influence style choice – aesthetic, sonority, and indexicality. Unlike many related studies, it places emphasis on the interdisciplinary nature of the subject, drawing upon the work of musicologists, philosophers and linguists, in an attempt to bring a fresh perspective on the phenomenon. Special attention is given to the notion that singers use accents to create (or be appropriate to) a particular aesthetic. The view is taken that music scenes act as unique speech communities that possess both socially and musically derived linguistic norms that all members accept (both performers and audience), but only few actively utilise in their language use (the singers)

Nodes and Arcs: Concept Map, Semiotics, and Knowledge Organization.

Purpose – The purpose of the research reported here is to improve comprehension of the socially-negotiated identity of concepts in the domain of knowledge organization. Because knowledge organization as a domain has as its focus the order of concepts, both from a theoretical perspective and from an applied perspective, it is important to understand how the domain itself understands the meaning of a concept. Design/methodology/approach – The paper provides an empirical demonstration of how the domain itself understands the meaning of a concept. The paper employs content analysis to demonstrate the ways in which concepts are portrayed in KO concept maps as signs, and they are subjected to evaluative semiotic analysis as a way to understand their meaning. The frame was the entire population of formal proceedings in knowledge organization – all proceedings of the International Society for Knowledge Organization’s international conferences (1990-2010) and those of the annual classification workshops of the Special Interest Group for Classification Research of the American Society for Information Science and Technology (SIG/CR). Findings – A total of 344 concept maps were analyzed. There was no discernible chronological pattern. Most concept maps were created by authors who were professors from the USA, Germany, France, or Canada. Roughly half were judged to contain semiotic content. Peirceian semiotics predominated, and tended to convey greater granularity and complexity in conceptual terminology. Nodes could be identified as anchors of conceptual clusters in the domain; the arcs were identifiable as verbal relationship indicators. Saussurian concept maps were more applied than theoretical; Peirceian concept maps had more theoretical content. Originality/value – The paper demonstrates important empirical evidence about the coherence of the domain of knowledge organization. Core values are conveyed across time through the concept maps in this population of conference paper

Symbol Emergence in Robotics: A Survey

Humans can learn the use of language through physical interaction with their environment and semiotic communication with other people. It is very important to obtain a computational understanding of how humans can form a symbol system and obtain semiotic skills through their autonomous mental development. Recently, many studies have been conducted on the construction of robotic systems and machine-learning methods that can learn the use of language through embodied multimodal interaction with their environment and other systems. Understanding human social interactions and developing a robot that can smoothly communicate with human users in the long term, requires an understanding of the dynamics of symbol systems and is crucially important. The embodied cognition and social interaction of participants gradually change a symbol system in a constructive manner. In this paper, we introduce a field of research called symbol emergence in robotics (SER). SER is a constructive approach towards an emergent symbol system. The emergent symbol system is socially self-organized through both semiotic communications and physical interactions with autonomous cognitive developmental agents, i.e., humans and developmental robots. Specifically, we describe some state-of-art research topics concerning SER, e.g., multimodal categorization, word discovery, and a double articulation analysis, that enable a robot to obtain words and their embodied meanings from raw sensory--motor information, including visual information, haptic information, auditory information, and acoustic speech signals, in a totally unsupervised manner. Finally, we suggest future directions of research in SER.Comment: submitted to Advanced Robotic

Ambient Gestures

We present Ambient Gestures, a novel gesture-based system designed to support ubiquitous ‘in the environment’ interactions with everyday computing technology. Hand gestures and audio feedback allow users to control computer applications without reliance on a graphical user interface, and without having to switch from the context of a non-computer task to the context of the computer. The Ambient Gestures system is composed of a vision recognition software application, a set of gestures to be processed by a scripting application and a navigation and selection application that is controlled by the gestures. This system allows us to explore gestures as the primary means of interaction within a multimodal, multimedia environment. In this paper we describe the Ambient Gestures system, define the gestures and the interactions that can be achieved in this environment and present a formative study of the system. We conclude with a discussion of our findings and future applications of Ambient Gestures in ubiquitous computing

An exploration of semiotics of new auditory displays: A comparative analysis with visual displays

Communicability is an important factor of user interfaces. To address communicability, extensive research has been done on visual displays, whereas relatively little research has been done on auditory displays. The present paper attempts to analyze semiotics of novel auditory displays (spearcon, spindex, and lyricon) using Peirce’s classification of signs: icon, symbol, and index. After the aesthetic developmental patterns of the visual counterparts are presented, semiotics of auditory cues is discussed with future design directions

That Doesn\u27t Sound Like Me: Student Perceptions of Semiotic Resources in Written-Aural Remediation Practices

This dissertation examines students\u27 composing practices when working with unfamiliar modalities, attending to students\u27 messy material and cognitive negotiations prior to their production of a polished multimodal project. Working from a conceptual vocabulary from composition studies and semiotics, I frame composing as an act of semiotic remediation, attending to students\u27 repurposing and understanding of written and aural materials in composition and their impact on their learning. Specifically, this research uses a grounded theory methodology to examine the attitudes, experiences, and composing practices of first-year writing students enrolled in a composition II course at a private, liberal arts institution in the South who were tasked with revising their writing into–and through–sound editing software to complete an audio revision project. This study examines the practices and evolving attitudes of seven students using various materials and the impact of their composing process on learning and interpersonal development. Findings from this study are used to develop a body of concepts that work together to theorize about the impact of semiotic remediation on students\u27 composing practices and their learning