Moving sounds and sonic moves : exploring interaction quality of embodied music mediation technologies through a user-centered perspective

This research project deals with the user-experience related to embodied music mediation technologies. More specifically, adoption and policy problems surrounding new media (art) are considered, which arise from the usability issues that to date pervade new interfaces for musical expression. Since the emergence of new wireless mediators and control devices for musical expression, there is an explicit aspiration of the creative industries and various research centers to embed such technologies into different areas of the cultural industries. The number of applications and their uses have exponentially increased over the last decade. Conversely, many of the applications to date still suffer from severe usability problems, which not only hinder the adoption by the cultural sector, but also make culture participants take a rather cautious, hesitant, or even downright negative stance towards these technologies. Therefore, this thesis takes a vantage point that is in part sociological in nature, yet has a link to cultural studies as well. It combines this with a musicological frame of reference to which it introduces empirical user-oriented approaches, predominantly taken from the field of human-computer-interaction studies. This interdisciplinary strategy is adopted to cope with the complex nature of digital embodied music controlling technologies. Within the Flanders cultural (and creative) industries, opportunities of systems affiliated with embodied interaction are created and examined. This constitutes an epistemological jigsaw that looks into 1) “which stakeholders require what various levels of involvement, what interactive means and what artistic possibilities?”, 2) “the way in which artistic aspirations, cultural prerequisites and operational necessities of (prospective) users can be defined?”, 3) “how functional, artistic and aesthetic requirements can be accommodated?”, and 4) “how quality of use and quality of experience can be achieved, quantified, evaluated and, eventually, improved?”. Within this multi-facetted problem, the eventual aim is to assess the applicability of the foresaid technology, both from a theoretically and empirically sound basis, and to facilitate widening and enhancing the adoption of said technologies. Methodologically, this is achieved by 1) applied experimentation, 2) interview techniques, 3) self-reporting and survey research, 4) usability evaluation of existing devices, and 5) human-computer interaction methods applied – and attuned – to the specific case of embodied music mediation technologies. Within that scope, concepts related to usability, flow, presence, goal assessment and game enjoyment are scrutinized and applied, and both task- and experience-oriented heuristics and metrics are developed and tested. In the first part, covering three chapters, the general context of the thesis is given. In the first chapter, an introduction to the topic is offered and the current problems are enumerated. In the second chapter, a broader theoretical background is presented of the concepts that underpin the project, namely 1) the paradigm of embodiment and its connection to musicology, 2) a state of the arts concerning new interfaces for musical expression, 3) an introduction into HCI-usability and its application domain in systematic musicology, 4) an insight into user-centered digital design procedures, and 5) the challenges brought about by e-culture and digitization for the cultural-creative industries. In the third chapter, the state of the arts concerning the available methodologies related to the thesis’ endeavor is discussed, a set of literature-based design guidelines are enumerated and from this a conceptual model is deduced which is gradually presented throughout the thesis, and fully deployed in the “SoundField”-project (as described in Chapter 9). The following chapters, contained in the second part of the thesis, give a quasi-chronological overview of how methodological concepts have been applied throughout the empirical case studies, aimed specifically at the exploration of the various aspects of the complex status quaestionis. In the fourth chapter, a series of application-based tests, predominantly revolving around interface evaluation, illustrate the complex relation between gestural interfaces and meaningful musical expression, advocating a more user-centered development approach to be adopted. In the fifth chapter, a multi-purpose questionnaire dubbed “What Moves You” is discussed, which aimed at creating a survey of the (prospective) end-users of embodied music mediation technologies. Therefore, it primarily focused on cultural background, musical profile and preferences, views on embodied interaction, literacy of and attitudes towards new technology and participation in digital culture. In the sixth chapter, the ethnographical studies that accompanied the exhibition of two interactive art pieces, entitled "Heart as an Ocean" & "Lament", are discussed. In these studies, the use of interview and questionnaire methodologies together with the presentation and reception of interactive art pieces, are probed. In the seventh chapter, the development of the collaboratively controlled music-game “Sync-In-Team” is presented, in which interface evaluation, presence, game enjoyment and goal assessment are the pivotal topics. In the eighth chapter, two usability studies are considered, that were conducted on prototype systems/interfaces, namely a heuristic evaluation of the “Virtual String” and a usability metrics evaluation on the “Multi-Level Sonification Tool”. The findings of these two studies in conjunction with the exploratory studies performed in association with the interactive art pieces, finally gave rise to the “SoundField”-project, which is recounted in full throughout the ninth chapter. The integrated participatory design and evaluation method, presented in the conceptual model is fully applied over the course of the “SoundField”-project, in which technological opportunities and ecological validity and applicability are investigated through user-informed development of numerous use cases. The third and last part of the thesis renders the final conclusions of this research project. The tenth chapter sets out with an epilogue in which a brief overview is given on how the state of the arts has evolved since the end of the project (as the research ended in 2012, but the research field has obviously moved on), and attempts to consolidate the implications of the research studies with some of the realities of the Flemish cultural-creative industries. Chapter eleven continues by discussing the strengths and weaknesses of the conceptual model throughout the various stages of the project. Also, it comprises the evaluation of the hypotheses, how the assumptions that were made held up, and how the research questions eventually could be assessed. Finally, the twelfth and last chapter concludes with the most important findings of the project. Also, it discusses some of the implications on cultural production, artistic research policy and offers an outlook on future research beyond the scope of the “SoundField” project

Machinima And Video-based Soft Skills Training

Multimedia training methods have traditionally relied heavily on video based technologies and significant research has shown these to be very effective training tools. However production of video is time and resource intensive. Machinima (pronounced \u27muh-sheen-eh-mah\u27) technologies are based on video gaming technology. Machinima technology allows video game technology to be manipulated into unique scenarios based on entertainment or training and practice applications. Machinima is the converting of these unique scenarios into video vignettes that tell a story. These vignettes can be interconnected with branching points in much the same way that education videos are interconnected as vignettes between decision points. This study addressed the effectiveness of machinima based soft-skills education using avatar actors versus the traditional video teaching application using human actors. This research also investigated the difference between presence reactions when using avatar actor produced video vignettes as compared to human actor produced video vignettes. Results indicated that the difference in training and/or practice effectiveness is statistically insignificant for presence, interactivity, quality and the skill of assertiveness. The skill of active listening presented a mixed result indicating the need for careful attention to detail in situations where body language and facial expressions are critical to communication. This study demonstrates that a significant opportunity exists for the exploitation of avatar actors in video based instruction

Understanding and training emotion regulation in children and adolescents

Emotion regulation (ER)—the ability to change an emotional experience in relation to a desired emotional goal is linked to broad psychosocial outcomes. In addition, early adolescence presents a sensitive period in the malleability of ER processes and is a period of particular risk for ER difficulties and the development of psychopathology. Utilising a mixed methods approach, this thesis explores the use of ER skills through childhood and adolescence within the context of social functioning (study 1, chapter 2); and leading from this, the training of ER skills via digital intervention approaches (study 2, chapter 3). Chapter 4 applied the findings of chapters 2 and 3 by presenting the evidence and codesign informed development of a prototype novel digital game for training specific ER strategies in early adolescence. Results demonstrate the importance of the development and use of adaptive ER skills through childhood and adolescence, and that issues around engagement, access, acceptability, and stigma in traditional and wider-reaching preventative intervention frameworks may be addressed by training ER via codesigned digital games. The applied implications of the thesis centre around the importance of training ER via appropriately codesigned digital technology in broad samples of early adolescents to address negative social experiences and linked psychological outcomes

The Experience of a Lifetime: Interactive Digital Experience Beyond the Screen

Screen-based digital experience design is blooming among the local businesses in Metro Vancouver along with the increased pervasiveness of information technologies, new digital products in contemporary society. However, there are significantly fewer cases and related businesses around tangible interactive digital experience in which tangible objects and physical spaces replace the screen as the site of interaction. This thesis project aims to explore the specialties of the tangible interactive experience compared to the digital experience on the screen or in the virtual space. Additionally, the author investigates how to leverage user experience design methodologies in the process of designing an experimental interactive experience. In this practice-based exploration, the author prototyped four interactive digital experiences using different interactive technologies and tools tailored to different use case scenarios: 1. an interactive offline retail experience, 2. a “magical” and playful painting, 3. a room-scale interactive installation, and 4. an immersive meditation activity. These projects illustrate and explore the implementation of tangible interactions into digital experience design. During the development process, the author applied several user experience design methodologies in the projects – including field research, interviews, questionnaires, and design probes – to develop a workable framework designing tangible interactive experiences throughout the research project. The author aims to outline key implications of applying principles of user experience design to the field of tangible interactive environments. In the process, the author argues that tangible interactive design is indispensable in a successful and engaging digital experience, and thus worth investing in and exploring further in Vancouver’s marketplace

Anonymous Panda: preserving anonymity and expressiveness in online mental health platforms

Digital solutions that allow people to seek treatment, such as online psychological interventions and other technology-mediated therapies, have been developed to assist individuals with mental health disorders. Such approaches may raise privacy concerns about the use of people’s data and the safety of their mental health information. This work uses cutting-edge computer graphics technology to develop a novel system capable of increasing anonymity while maintaining expressiveness in computer-mediated mental health interventions. According to our preliminary findings, we were able to customize a realistic avatar using Live Link, Metahumans, and Unreal Engine 4 (UE4) with the same emotional depth as a real person. Furthermore, these findings showed that the virtual avatars’ inability to express themselves through hand motion gave the impression that they were acting in an unnatural way. By including the hand tracking feature using the Leap Motion Controller, we were able to improve our comprehension of the prospective use of ultra-realistic virtual human avatars in video conferencing therapy, i.e., both studies helped us understand how vital facial and body expressions are and how problematic their absence is in communicating with others.Soluções digitais que permitem às pessoas procurar tratamento, tais como terapias psicológicas online e outras terapias com recurso à tecnologia, foram desenvolvidas para ajudar indivíduos com distúrbios de saúde mental. Tais abordagens podem suscitar preocupações sobre a privacidade na utilização dos dados das pessoas e a segurança da informação sobre a sua saúde mental. Este trabalho utiliza tecnologia de ponta em computação gráfica para desenvolver um sistema inovador capaz de aumentar o anonimato, mantendo simultaneamente a expressividade nas inter venções de saúde mental mediadas por computador. Segundo os nossos resultados preliminares, conseguimos personalizar um avatar realista usando Live Link, Metahumans, e Unreal Engine 4 (UE4) com a mesma profundidade emocional que uma pessoa real. Além disso, os resultados mostraram que a incapacidade dos avatares virtuais de se expressarem através do movimento das mãos deu a impressão de que estavam a agir de uma forma pouco natural. Ao incluir a função de rastreio das mãos utilizando o Leap Motion Controller, conseguimos melhorar a nossa compreensão do uso prospetivo de avatares humanos virtuais e ultrarrealistas na terapia de videoconferência, ou seja, os estudos realizados ajudaram-nos a compreender como as expressões faciais e corporais são vitais e como a sua ausência é problemática na comunicação com os outros

A Reflection on Virtual Reality Design for Psychological, Cognitive & Behavioral Interventions: Design Needs, Opportunities & Challenges

Despite the substantial research interest in using Virtual Reality (VR) in healthcare in general and in Psychological, Cognitive, and Behavioral (PC&B) interventions in specific, as well as emerging research supporting the efficacy of VR in healthcare, the design process of translating therapies into VR to meet the needs of critical stakeholders such as users and clinicians is rarely addressed. In this paper, we aim to shed light onto the design needs, opportunities and challenges in designing efficient and effective PC&B-VR interventions. Through analyzing the co-design processes of four user-centered PC&B-VR interventions, we examined how therapies were adapted into VR to meet stakeholders’ requirements, explored design elements for meaningful experiences, and investigated how the understanding of healthcare contexts contribute to the VR intervention design. This paper presents the HCI research community with design opportunities and challenges as well as future directions for PC&B-VR intervention design