Landscapes of Affective Interaction: Young Children's Enactive Engagement with Body Metaphors

Empirical research into embodied meaning making suggests specific sensorimotor experiences can support children’s understanding of abstract science ideas. This view is aligned with enactive and grounded cognition perspectives, both centred in the view that our ability to conceptualise emerges from our experiences of interaction with our environment. While much of this research has focused on understanding action and action processes in individual children or children in pairs, less attention has been paid to affective dimensions of young children’s group interaction, and how this relates to meaning making with body metaphors. Indeed, Gallagher describes how no action exists in a vacuum, but rather revolves around a complex web of affective-pragmatic features comprising a ‘Landscape of Interaction’ (2020, p.42). This research project addresses gaps in research in understanding young children’s affective engagement from an enactivist cognition perspective. It takes a Design-Based Research approach with an iterative design orientation to examine young children’s interaction with multisensory body-based metaphors through an embodied participation framework. A series of empirical studies with young children, aged 2-7 years, comprising of experiential workshops, build iteratively upon each other. A novel theoretically informed method, Affective Imagination in Motion, is developed involving several purpose-built multisensory body metaphors prompts to enable access to dimensions of young children’s affective engagement. This research makes theoretical and methodological contributions. It extends the theoretical notion of ‘affect’ from enactive and grounded cognition perspectives through identifying key interactive processes in young children’s engagement with multisensory action metaphors. In addition, the novel method offers a contribution as a way of ‘looking’ at affect within a group situation from affective-pragmatic and social embodiment perspectives. Finally, the research contributes to embodied learning design frameworks offering a guideline for designers wishing to inform their work from enactive cognition perspective

Governing Confidence: Rhetoric, Affect, and Post-Crisis Financial Education

The 2007-08 financial crisis has been characterized as a “crisis of confidence” (Akerlof & Shiller, 2009), a span of time during which the non-discursive energy needed to compel Americans towards profit-producing decisions evaporated. Amidst this decline, the US lost its competitive edge in the global marketplace. Initial responses to the crisis by national leaders failed, triggering a revision to reasoning that resulted in a new argument taken up by central government: the lack of financial knowledge experienced by the majority of US citizens led to a population of ignorant decision makers lacking the confidence needed to take the risks necessary to propel the country’s economy forward. According to the problematization, the solution necessary was the revision of the country’s financial education curriculum. Embracing this argument, both Presidents Bush and Obama called for the development of councils that would evaluate the status of financial education and use that information to recommend changes to the discourse of financial education in the future. This dissertation uses Foucault’s (1991, 2007, 2008) theory of governmentality alongside the affect scholarship of Brennan (2004) and Grusin’s (2010) work on the digital mediation of affect to examine these arguments and the technologies of governance they produced that would motivate US citizens to take control of their financial situations through actions made with confidence that would benefit these individual decision makers as well as the US economy

A PHENOMENOLOGY OF MIMETIC LEARNING AND MULTIMODAL COGNITION: INTEGRATING EXPERIENTIAL KNOWLEDGE INTO PROGRAMS IN RHETORIC, COMPOSITION, AND TECHNICAL COMMUNICATION

My dissertation emphasizes a cognitive account of multimodality that explicitly integrates experiential knowledge work into the rhetorical pedagogy that informs so many composition and technical communication programs. In these disciplines, multimodality is widely conceived in terms of what Gunther Kress calls “socialsemiotic” modes of communication shaped primarily by culture. In the cognitive and neurolinguistic theories of Vittorio Gallese and George Lakoff, however, multimodality is described as a key characteristic of our bodies’ sensory-motor systems which link perception to action and action to meaning, grounding all communicative acts in knowledge shaped through body-engaged experience. I argue that this “situated” account of cognition – which closely approximates Maurice Merleau-Ponty’s phenomenology of perception, a major framework for my study – has pedagogical precedence in the mimetic pedagogy that informed ancient Sophistic rhetorical training, and I reveal that training’s multimodal dimensions through a phenomenological exegesis of the concept mimesis. Plato’s denigration of the mimetic tradition and his elevation of conceptual contemplation through reason, out of which developed the classic Cartesian separation of mind from body, resulted in a general degradation of experiential knowledge in Western education. But with the recent introduction into college classrooms of digital technologies and multimedia communication tools, renewed emphasis is being placed on the “hands-on” nature of inventive and productive praxis, necessitating a revision of methods of instruction and assessment that have traditionally privileged the acquisition of conceptual over experiential knowledge. The model of multimodality I construct from Merleau-Ponty’s phenomenology, ancient Sophistic rhetorical pedagogy, and current neuroscientific accounts of situated cognition insists on recognizing the significant role knowledges we acquire experientially play in our reading and writing, speaking and listening, discerning and designing practices

The interactive brain hypothesis

Enactive approaches foreground the role of interpersonal interaction in explanations of social understanding. This motivates, in combination with a recent interest in neuroscientific studies involving actual interactions, the question of how interactive processes relate to neural mechanisms involved in social understanding. We introduce the Interactive Brain Hypothesis (IBH) in order to help map the spectrum of possible relations between social interaction and neural processes. The hypothesis states that interactive experience and skills play enabling roles in both the development and current function of social brain mechanisms, even in cases where social understanding happens in the absence of immediate interaction. We examine the plausibility of this hypothesis against developmental and neurobiological evidence and contrast it with the widespread assumption that mindreading is crucial to all social cognition. We describe the elements of social interaction that bear most directly on this hypothesis and discuss the empirical possibilities open to social neuroscience. We propose that the link between coordination dynamics and social understanding can be best grasped by studying transitions between states of coordination. These transitions form part of the self-organization of interaction processes that characterize the dynamics of social engagement. The patterns and synergies of this self-organization help explain how individuals understand each other. Various possibilities for role-taking emerge during interaction, determining a spectrum of participation. This view contrasts sharply with the observational stance that has guided research in social neuroscience until recently. We also introduce the concept of readiness to interact to describe the practices and dispositions that are summoned in situations of social significance (even if not interactive). This latter idea links interactive factors to more classical observational scenarios

Multisensory learning in adaptive interactive systems

The main purpose of my work is to investigate multisensory perceptual learning and sensory integration in the design and development of adaptive user interfaces for educational purposes. To this aim, starting from renewed understanding from neuroscience and cognitive science on multisensory perceptual learning and sensory integration, I developed a theoretical computational model for designing multimodal learning technologies that take into account these results. Main theoretical foundations of my research are multisensory perceptual learning theories and the research on sensory processing and integration, embodied cognition theories, computational models of non-verbal and emotion communication in full-body movement, and human-computer interaction models. Finally, a computational model was applied in two case studies, based on two EU ICT-H2020 Projects, "weDRAW" and "TELMI", on which I worked during the PhD

Moving sounds and sonic moves : exploring interaction quality of embodied music mediation technologies through a user-centered perspective

This research project deals with the user-experience related to embodied music mediation technologies. More specifically, adoption and policy problems surrounding new media (art) are considered, which arise from the usability issues that to date pervade new interfaces for musical expression. Since the emergence of new wireless mediators and control devices for musical expression, there is an explicit aspiration of the creative industries and various research centers to embed such technologies into different areas of the cultural industries. The number of applications and their uses have exponentially increased over the last decade. Conversely, many of the applications to date still suffer from severe usability problems, which not only hinder the adoption by the cultural sector, but also make culture participants take a rather cautious, hesitant, or even downright negative stance towards these technologies. Therefore, this thesis takes a vantage point that is in part sociological in nature, yet has a link to cultural studies as well. It combines this with a musicological frame of reference to which it introduces empirical user-oriented approaches, predominantly taken from the field of human-computer-interaction studies. This interdisciplinary strategy is adopted to cope with the complex nature of digital embodied music controlling technologies. Within the Flanders cultural (and creative) industries, opportunities of systems affiliated with embodied interaction are created and examined. This constitutes an epistemological jigsaw that looks into 1) “which stakeholders require what various levels of involvement, what interactive means and what artistic possibilities?”, 2) “the way in which artistic aspirations, cultural prerequisites and operational necessities of (prospective) users can be defined?”, 3) “how functional, artistic and aesthetic requirements can be accommodated?”, and 4) “how quality of use and quality of experience can be achieved, quantified, evaluated and, eventually, improved?”. Within this multi-facetted problem, the eventual aim is to assess the applicability of the foresaid technology, both from a theoretically and empirically sound basis, and to facilitate widening and enhancing the adoption of said technologies. Methodologically, this is achieved by 1) applied experimentation, 2) interview techniques, 3) self-reporting and survey research, 4) usability evaluation of existing devices, and 5) human-computer interaction methods applied – and attuned – to the specific case of embodied music mediation technologies. Within that scope, concepts related to usability, flow, presence, goal assessment and game enjoyment are scrutinized and applied, and both task- and experience-oriented heuristics and metrics are developed and tested. In the first part, covering three chapters, the general context of the thesis is given. In the first chapter, an introduction to the topic is offered and the current problems are enumerated. In the second chapter, a broader theoretical background is presented of the concepts that underpin the project, namely 1) the paradigm of embodiment and its connection to musicology, 2) a state of the arts concerning new interfaces for musical expression, 3) an introduction into HCI-usability and its application domain in systematic musicology, 4) an insight into user-centered digital design procedures, and 5) the challenges brought about by e-culture and digitization for the cultural-creative industries. In the third chapter, the state of the arts concerning the available methodologies related to the thesis’ endeavor is discussed, a set of literature-based design guidelines are enumerated and from this a conceptual model is deduced which is gradually presented throughout the thesis, and fully deployed in the “SoundField”-project (as described in Chapter 9). The following chapters, contained in the second part of the thesis, give a quasi-chronological overview of how methodological concepts have been applied throughout the empirical case studies, aimed specifically at the exploration of the various aspects of the complex status quaestionis. In the fourth chapter, a series of application-based tests, predominantly revolving around interface evaluation, illustrate the complex relation between gestural interfaces and meaningful musical expression, advocating a more user-centered development approach to be adopted. In the fifth chapter, a multi-purpose questionnaire dubbed “What Moves You” is discussed, which aimed at creating a survey of the (prospective) end-users of embodied music mediation technologies. Therefore, it primarily focused on cultural background, musical profile and preferences, views on embodied interaction, literacy of and attitudes towards new technology and participation in digital culture. In the sixth chapter, the ethnographical studies that accompanied the exhibition of two interactive art pieces, entitled "Heart as an Ocean" & "Lament", are discussed. In these studies, the use of interview and questionnaire methodologies together with the presentation and reception of interactive art pieces, are probed. In the seventh chapter, the development of the collaboratively controlled music-game “Sync-In-Team” is presented, in which interface evaluation, presence, game enjoyment and goal assessment are the pivotal topics. In the eighth chapter, two usability studies are considered, that were conducted on prototype systems/interfaces, namely a heuristic evaluation of the “Virtual String” and a usability metrics evaluation on the “Multi-Level Sonification Tool”. The findings of these two studies in conjunction with the exploratory studies performed in association with the interactive art pieces, finally gave rise to the “SoundField”-project, which is recounted in full throughout the ninth chapter. The integrated participatory design and evaluation method, presented in the conceptual model is fully applied over the course of the “SoundField”-project, in which technological opportunities and ecological validity and applicability are investigated through user-informed development of numerous use cases. The third and last part of the thesis renders the final conclusions of this research project. The tenth chapter sets out with an epilogue in which a brief overview is given on how the state of the arts has evolved since the end of the project (as the research ended in 2012, but the research field has obviously moved on), and attempts to consolidate the implications of the research studies with some of the realities of the Flemish cultural-creative industries. Chapter eleven continues by discussing the strengths and weaknesses of the conceptual model throughout the various stages of the project. Also, it comprises the evaluation of the hypotheses, how the assumptions that were made held up, and how the research questions eventually could be assessed. Finally, the twelfth and last chapter concludes with the most important findings of the project. Also, it discusses some of the implications on cultural production, artistic research policy and offers an outlook on future research beyond the scope of the “SoundField” project