Symbol Emergence in Robotics: A Survey

Humans can learn the use of language through physical interaction with their environment and semiotic communication with other people. It is very important to obtain a computational understanding of how humans can form a symbol system and obtain semiotic skills through their autonomous mental development. Recently, many studies have been conducted on the construction of robotic systems and machine-learning methods that can learn the use of language through embodied multimodal interaction with their environment and other systems. Understanding human social interactions and developing a robot that can smoothly communicate with human users in the long term, requires an understanding of the dynamics of symbol systems and is crucially important. The embodied cognition and social interaction of participants gradually change a symbol system in a constructive manner. In this paper, we introduce a field of research called symbol emergence in robotics (SER). SER is a constructive approach towards an emergent symbol system. The emergent symbol system is socially self-organized through both semiotic communications and physical interactions with autonomous cognitive developmental agents, i.e., humans and developmental robots. Specifically, we describe some state-of-art research topics concerning SER, e.g., multimodal categorization, word discovery, and a double articulation analysis, that enable a robot to obtain words and their embodied meanings from raw sensory--motor information, including visual information, haptic information, auditory information, and acoustic speech signals, in a totally unsupervised manner. Finally, we suggest future directions of research in SER.Comment: submitted to Advanced Robotic

Narratives of ocular experience in interactive 360° environments

The purpose of this research project was to examine how immersive digital virtual technologies have the potential to expand the genre of interactive film into new forms of audience engagement and narrative production. Aside from addressing the limitations of interactive film, I have explored how interactive digital narratives can be reconfigured in the wake of immersive media. My contribution to knowledge stems from using a transdisciplinary synthesis of the interactive systems in film and digital media art, which is embodied in the research framework and theoretical focal point that I have titled Cynematics (chapter 2). Using a methodology that promotes iterative experimentation I developed a series of works that allowed me to practically explore the limitations of interactive film systems that involve non-haptic user interaction. This is evidenced in the following series of works: Virtual Embodiment, Narrative Maze, Eye Artefact Interactions and Routine Error - all of which are discussed in chapter 4 of this thesis. Each of these lab experiments collectively build towards the development of novel interactive 360° film practices. Funneling my research towards these underexplored processes I focused on virtual gaze interaction (chapters 4-6), aiming to define and historically contextualise this system of interaction, whilst critically engaging with it through my practice. It is here that gaze interaction is cemented as the key focus of this thesis. The potential of interactive 360° film is explored through the creation of three core pieces of practice, which are titled as follows: Systems of Seeing (chapter 5), Mimesis (chapter 6), Vanishing Point (chapter 7). Alongside the close readings in these chapters and the theoretical developments explored in each are the interaction designs included in the appendix of the thesis. These provide useful context for readers unable to experience these site-specific installations as virtual reality applications. After creating these systems, I established terms to theoretically unpack some of the processes occurring within them. These include Datascape Mediation (chapter 2), which frames agency as a complex entanglement built on the constantly evolving relationships between human and machine - and Live-Editing Practice (chapter 7), which aims to elucidate how the interactive 360° film practice designed for this research leads to new way of thinking about how we design, shoot and interact with 360° film. Reflecting on feedback from exhibiting Mimesis I decided to define and evaluate the key modes of virtual gaze interaction, which led to the development of a chapter and concept referred to as The Reticle Effect (chapter 6). This refers to how a visual overlay that is used to represent a user's line of sight not only shapes their experience of the work, but also dictates their perception of genre. To navigate this, I combined qualitative and quantitative analysis to explore user responses to four different types of gaze interaction. In preparing to collect this data I had to articulate these different types of interaction, which served to demarcate the difference between each of these types of gaze interaction. Stemming from this I used questionnaires, thematic analysis and data visualisation to explore the use and response to these systems. The results of this not only supports the idea of the reticle effect, but also gives insight into how these different types of virtual gaze interaction shape whether these works are viewed as games or as types of interactive film. The output of this allowed me to further expand on interactive 360° film as a genre of immersive media and move beyond the realm of interactive film into new technological discourses, which serves to validate the nascent, yet expansive reach of interactive 360° film as a form of practice. The thesis is concluded by framing this research within the wider discourse of posthuman theory as given that the technologies of immersive media perpetuate a state of extended human experience - how we interact and consider the theories that surround these mediums needs to be considered in the same way. The practice and theory developed throughout this thesis contribute to this discourse and allow for new ways of considering filmic language in the wake of interactive 360° film practice

Gestures as Mimetic Forms of Identity in Post-Secondary Italian as a Foreign Language Classrooms: A Sociocultural Perspective

This study investigates the use of mimetic gestures of identity by foreign language teachers of Italian and their students in college classes as a form of meaning-making. All four of the teachers were found to use a variety of Italian gestures as a regular aspect of their teaching and presentation of self. Students and teachers also were found to mirror each other’s gestures. None of the teachers had been video-recorded before the study and all were surprised to see the degree to which they appeared to be Italian, although at the same time all believed this to be an important and positive aspect of their teaching. Students had similar views on the significant role of authentic embodiment of the Italian languaculture for instructors and students alike, particularly in relation to renting a new identity. In offering an explanation for these findings we consider the role of gesture as a social semiotic in learning another language, how teachers perform their identity in the classroom and use gesture to prolept students into a possible future as embodied communicators of the language (thus encouraging language learners to acquire their own identity within a figured world), the process of communicative actuation as it relates to learning a language across different timescales and environments, and how all of the above relates to the zone of proximal development and its application to frontier regions of development

Livegraphy performance art, language, and the multiplicity of sense

PhDThis thesis is constructed in three parts. Each one of them offers a reflection on the common ideas disseminated about Live Art, conceptual dance and postdramatic theatre, i.e. that these practices reject the notion of mimesis as it is supposed to represent reality, they reject text in favour of a phenomenological language and they produce a form of non-sense which should be translated into meaning. Each of these statements will be problematized. I will argue that Live Art is producing mimesis even if it works against representation and although its actions are performed for real. It does not represent reality, but neither does it present the Real. It is producing a version of the "Real", which is the definition of mimesis. I will then argue that if these practices create a phenomenological language, it relies on a form of writing that is being produced live by the work. Finally, I will propose that the non-sense constructed by this writing process should not be forced into a meaning, but should be read as a fluid linguistics, which in some instances will be concretely a linguistics of fluids. By this I intend to point out that the meaning of the constructed non-sense will never be fixed nor unique. The work only becomes meaningful because it remains permeable to meanings. These three steps all participate in the "undoing of meaning"; relying on a process involving destruction within construction to then allow reconstruction. Mimesis, logos and sense need to be taken apart before these concepts can be thought anew. It is the rigidity of the conventional systems of apprehension which has to become permeable to allow a fluid multiplicity of meanings. In conclusion I will draw some parallels between performance art and feminism in their appropriation of the concept of mimesis and their approach to language outside the structure of logos and I will suggest that the performances which explore and expose these concepts adopt a feminist philosophical strategy. 1 I chose to use this spelling closer to the French spelling of “non-sens”, which does not have in French the colloquial use it has in English and is more directly related to the philosophical concept. The hyphenated word better translates the idea of a reverse image of the word “sense”

Embodiment in Evolution and Culture

From its beginnings, the theory of evolution has unsettled fundamental anthropological assumptions about the place of human beings in nature. The integration of human origins into natural history by Darwinism was countered by the philosophical anthropologies of the 20th century. Their attempts were to hold on even more resolutely to the special status of humans as beings 'open towards the world'. Today, evolutionary and philosophical anthropology have moved closer together via the paradigm of embodiment. Building on embodied cognitive science, this volume aims to establish how far the human mind and human cultural cognition can be attributed to the structures of human existence, structures which have emerged in the course of evolution and have in turn been affected by culture. The traditional dualism of nature and culture is transformed into an explanation of an evolutionary process in which body and mind are understood to be intertwined and mutually constitutive

Machine Performers: Agents in a Multiple Ontological State

In this thesis, the author explores and develops new attributes for machine performers and merges the trans-disciplinary fields of the performing arts and artificial intelligence. The main aim is to redefine the term “embodiment” for robots on the stage and to demonstrate that this term requires broadening in various fields of research. This redefining has required a multifaceted theoretical analysis of embodiment in the field of artificial intelligence (e.g. the uncanny valley), as well as the construction of new robots for the stage by the author. It is hoped that these practical experimental examples will generate more research by others in similar fields. Even though the historical lineage of robotics is engraved with theatrical strategies and dramaturgy, further application of constructive principles from the performing arts and evidence from psychology and neurology can shift the perception of robotic agents both on stage and in other cultural environments. In this light, the relation between representation, movement and behaviour of bodies has been further explored to establish links between constructed bodies (as in artificial intelligence) and perceived bodies (as performers on the theatrical stage). In the course of this research, several practical works have been designed and built, and subsequently presented to live audiences and research communities. Audience reactions have been analysed with surveys and discussions. Interviews have also been conducted with choreographers, curators and scientists about the value of machine performers. The main conclusions from this study are that fakery and mystification can be used as persuasive elements to enhance agency. Morphologies can also be applied that tightly couple brain and sensorimotor actions and lead to a stronger stage presence. In fact, if this lack of presence is left out of human replicants, it causes an “uncanny” lack of agency. Furthermore, the addition of stage presence leads to stronger identification from audiences, even for bodies dissimilar to their own. The author demonstrates that audience reactions are enhanced by building these effects into machine body structures: rather than identification through mimicry, this causes them to have more unambiguously biological associations. Alongside these traits, atmospheres such as those created by a cast of machine performers tend to cause even more intensely visceral responses. In this thesis, “embodiment” has emerged as a paradigm shift – as well as within this shift – and morphological computing has been explored as a method to deepen this visceral immersion. Therefore, this dissertation considers and builds machine performers as “true” performers for the stage, rather than mere objects with an aura. Their singular and customized embodiment can enable the development of non-anthropocentric performances that encompass the abstract and conceptual patterns in motion and generate – as from human performers – empathy, identification and experiential reactions in live audiences

Conversational Movement Dynamics and Nonverbal Indicators of Second Language Development: A Microgenetic Approach

This dissertation study extends on current understandings of gesture and embodied interaction with the eco-social environment in second language development (SLD) while introducing new aspects of movement analysis through dynamical modeling. To understand the role of embodiment during learning activities, a second language learning task has been selected. Dyads consisting of a non-native English-speaking student and a native English-speaking tutor were video recorded during writing consultations centered on class assignments provided by the student. Cross-recurrence quantification analysis was used to measure interactional movement synchrony between the members of each dyad. Results indicate that students with varied English proficiency levels synchronize movements with their tutors over brief, frequent periods of time. Synchronous movement pattern complexity is highly variable across and within the dyads. Additionally, co-speech gesture and gesture independent of speech were analyzed qualitatively to identify the role of gesture as related to SLD events. A range of movement types were used during developmental events by the students and tutors to interact with their partner. The results indicated that language development occurs within a movement rich context through negotiated interaction which depends on a combination of synchronized and synergistic movements. Synchronized movements exhibited complex, dynamical behaviors including variability, self-organization, and emergent properties. Synergistic movement emergence revealed how the dualistic presence of the self/other in each dyad creates a functioning intersubjective space. Overall, the dyads demonstrated that movement is a salient factor in the writing consultation activity

Keys to Play: Music as a Ludic Medium from Apollo to Nintendo

How do keyboards make music playable? Drawing on theories of media, systems, and cultural techniques, Keys to Play spans Greek myth and contemporary Japanese digital games to chart a genealogy of musical play and its animation via improvisation, performance, and recreation. As a paradigmatic digital interface, the keyboard forms a field of play on which the book’s diverse objects of inquiry—from clavichords to PCs and eighteenth-century musical dice games to the latest rhythm-action titles—enter into analogical relations. Remapping the keyboard’s topography by way of Mozart and Super Mario, who head an expansive cast of historical and virtual actors, Keys to Play invites readers to unlock ludic dimensions of music that are at once old and new