Sampling the past:a tactile approach to interactive musical instrument exhibits in the heritage sector

In the last decade, the heritage sector has had to adapt to a shifting cultural landscape of public expectations and attitudes towards ownership and intellectual property. One way it has done this is to focus on each visitor’s encounter and provide them with a sense of experiential authenticity.There is a clear desire by the public to engage with music collections in this way, and a sound museological rationale for providing such access, but the approach raises particular curatorial problems, specifically how do we meaningfully balance access with the duty to preserve objects for future generations?This paper charts the development of one such project. Based at Fenton House in Hampstead, and running since 2008, the project seeks to model digitally the keyboard instruments in the Benton Fletcher Collection and provide a dedicated interactive exhibit, which allows visitors to view all of the instruments in situ, and then play them through a custom-built two-manual MIDI controller with touch-screen interface.We discuss the approach to modelling, which uses high-definition sampling, and highlight the strengths and weaknesses of the exhibit as it currently stands, with particular focus on its key shortcoming: at present, there is no way to effectively model the key feel of a historic keyboard instrument.This issue is of profound importance, since the feel of any instrument is fundamental to its character, and shapes the way performers relate to it. The issue is further compounded if we are to consider a single dedicated keyboard as being the primary mode of interface for several instrument models of different classes, each with its own characteristic feel.We conclude by proposing an outline solution to this problem, detailing early work on a real-time adaptive haptic keyboard interface that changes its action in response to sampled resistance curves, measured on a key-by-key basis from the original instruments

A critical assessment of the animator's artistic ownership over motion captured performances

The intention of this research report is to critically assess, as well as theoretically expand upon, the increasingly contentious area regarding character performance in computer generated (CG) animation for feature films that utilize motion capture technology. This paper specifically aims to investigate whether or not the use of motion capture in live-action visual effects, in the pursuit of creating CG characters that are as realistic as possible, has eroded the artistic autonomy of character animators and their artistic ownership of the performances of these characters. Through the analysis and comparison of pertinent case studies, it should become apparent that this perception is not absolute, and is largely dependent on the kinds of characters intended to be portrayed and the kind of film that they are to be portrayed in. It will be shown that motion capture can be a very effective collaborative tool not only in the relationship between directors and actors, but also between animators and actors under the creative supervision of directors

The Rocketbox Library and the Utility of Freely Available Rigged Avatars

As part of the open sourcing of the Microsoft Rocketbox avatar library for research and academic purposes, here we discuss the importance of rigged avatars for the Virtual and Augmented Reality (VR, AR) research community. Avatars, virtual representations of humans, are widely used in VR applications. Furthermore many research areas ranging from crowd simulation to neuroscience, psychology, or sociology have used avatars to investigate new theories or to demonstrate how they influence human performance and interactions. We divide this paper in two main parts: the first one gives an overview of the different methods available to create and animate avatars. We cover the current main alternatives for face and body animation as well introduce upcoming capture methods. The second part presents the scientific evidence of the utility of using rigged avatars for embodiment but also for applications such as crowd simulation and entertainment. All in all this paper attempts to convey why rigged avatars will be key to the future of VR and its wide adoption

Nice is Different than Good: Longitudinal Communicative Effects of Realistic and Cartoon Avatars in Real Mixed Reality Work Meetings

We report a within-subjects study of the effect of realistic and cartoon avatars on communication, task satisfaction, and perceived sense of presence in mixed reality meetings. For 2 − 3 weeks, six groups of co-workers (14 people) held a recurring real work meeting using Microsoft HoloLens2 devices. Each person embodied a personalised full-body avatar with a realistic face and another with a cartoon face. Half the groups started in the realistic condition and the other half started in the cartoon condition; all groups switched conditions half-way. Initial results show that, overall, participants found the realistic avatars’ nonverbal behaviour more appropriate for the interaction and more useful for understanding their colleagues compared to the cartoon one. Regarding the results over time, we identify different insights for cartoon and realistic avatars based on the type of avatar was embodied first. We discuss the implications of these results for mixed and virtual reality meetings

Walking with virtual humans : understanding human response to virtual humanoids' appearance and behaviour while navigating in immersive VR

In this thesis, we present a set of studies whose results have allowed us to analyze how to improve the realism, navigation, and behaviour of the avatars in an immersive virtual reality environment. In our simulations, participants must perform a series of tasks and we have analyzed perceptual and behavioural data. The results of the studies have allowed us to deduce what improvements are needed to be incorporated to the original simulations, in order to enhance the perception of realism, the navigation technique, the rendering of the avatars, their behaviour or their animations. The most reliable technique for simulating avatars’ behaviour in a virtual reality environment should be based on the study of how humans behave within the environment. For this purpose, it is necessary to build virtual environments where participants can navigate safely and comfortably with a proper metaphor and, if the environment is populated with avatars, simulate their behaviour accurately. All these aspects together will make the participants behave in a way that is closer to how they would behave in the real world. Besides, the integration of these concepts could provide an ideal platform to develop different types of applications with and without collaborative virtual reality such as emergency simulations, teaching, architecture, or designing. In the first contribution of this thesis, we carried out an experiment to study human decision making during an evacuation. We were interested to evaluate to what extent the behaviour of a virtual crowd can affect individuals' decisions. From the second contribution, in which we studied the perception of realism with bots and humans performing just locomotion or varied animations, we can conclude that the combination of having human-like avatars with animation variety can increase the overall realism of a crowd simulation, trajectories and animation. The preliminary study presented in the third contribution of this thesis showed that realistic rendering of the environment and the avatars do not appear to increase the perception of realism in the participants, which is consistent with works presented previously. The preliminary results in our walk-in-place contribution showed a seamless and natural transition between walk-in-place and normal walk. Our system provided a velocity mapping function that closely resembles natural walk. We observed through a pilot study that the system successfully reduces motion sickness and enhances immersion. Finally, the results of the contribution related to locomotion in collaborative virtual reality showed that animation synchronism and footstep sound of the avatars representing the participants do not seem to have a strong impact in terms of presence and feeling of avatar control. However, in our experiment, incorporating natural animations and footstep sound resulted in smaller clearance values in VR than previous work in the literature. The main objective of this thesis was to improve different factors related to virtual reality experiences to make the participants feel more comfortable in the virtual environment. These factors include the behaviour and appearance of the virtual avatars and the navigation through the simulated space in the experience. By increasing the realism of the avatars and facilitating navigation, high scores in presence are achieved during the simulations. This provides an ideal framework for developing collaborative virtual reality applications or emergency simulations that require participants to feel as if they were in real life.En aquesta tesi, es presenta un conjunt d'estudis els resultats dels quals ens han permès analitzar com millorar el realisme, la navegació i el comportament dels avatars en un entorn de realitat virtual immersiu. En les nostres simulacions, els participants han de realitzar una sèrie de tasques i hem analitzat dades perceptives i de comportament mentre les feien. Els resultats dels estudis ens han permès deduir quines millores són necessàries per a ser incorporades a les simulacions originals, amb la finalitat de millorar la percepció del realisme, la tècnica de navegació, la representació dels avatars, el seu comportament o les seves animacions. La tècnica més fiable per simular el comportament dels avatars en un entorn de realitat virtual hauria de basar-se en l'estudi de com es comporten els humans dins de l¿entorn virtual. Per a aquest propòsit, és necessari construir entorns virtuals on els participants poden navegar amb seguretat i comoditat amb una metàfora adequada i, si l¿entorn està poblat amb avatars, simular el seu comportament amb precisió. Tots aquests aspectes junts fan que els participants es comportin d'una manera més pròxima a com es comportarien en el món real. A més, la integració d'aquests conceptes podria proporcionar una plataforma ideal per desenvolupar diferents tipus d'aplicacions amb i sense realitat virtual col·laborativa com simulacions d'emergència, ensenyament, arquitectura o disseny. En la primera contribució d'aquesta tesi, vam realitzar un experiment per estudiar la presa de decisions durant una evacuació. Estàvem interessats a avaluar en quina mesura el comportament d'una multitud virtual pot afectar les decisions dels participants. A partir de la segona contribució, en la qual estudiem la percepció del realisme amb robots i humans que realitzen només una animació de caminar o bé realitzen diverses animacions, vam arribar a la conclusió que la combinació de tenir avatars semblants als humans amb animacions variades pot augmentar la percepció del realisme general de la simulació de la multitud, les seves trajectòries i animacions. L'estudi preliminar presentat en la tercera contribució d'aquesta tesi va demostrar que la representació realista de l¿entorn i dels avatars no semblen augmentar la percepció del realisme en els participants, que és coherent amb treballs presentats anteriorment. Els resultats preliminars de la nostra contribució de walk-in-place van mostrar una transició suau i natural entre les metàfores de walk-in-place i caminar normal. El nostre sistema va proporcionar una funció de mapatge de velocitat que s'assembla molt al caminar natural. Hem observat a través d'un estudi pilot que el sistema redueix amb èxit el motion sickness i millora la immersió. Finalment, els resultats de la contribució relacionada amb locomoció en realitat virtual col·laborativa van mostrar que el sincronisme de l'animació i el so dels avatars que representen els participants no semblen tenir un fort impacte en termes de presència i sensació de control de l'avatar. No obstant això, en el nostre experiment, la incorporació d'animacions naturals i el so de passos va donar lloc a valors de clearance més petits en RV que treballs anteriors ja publicats. L'objectiu principal d'aquesta tesi ha estat millorar els diferents factors relacionats amb experiències de realitat virtual immersiva per fer que els participants se sentin més còmodes en l'entorn virtual. Aquests factors inclouen el comportament i l'aparença dels avatars i la navegació a través de l'entorn virtual. En augmentar el realisme dels avatars i facilitar la navegació, s'aconsegueixen altes puntuacions en presència durant les simulacions. Això proporciona un marc ideal per desenvolupar aplicacions col·laboratives de realitat virtual o simulacions d'emergència que requereixen que els participants se sentin com si estiguessin en la vida realPostprint (published version

Examining the Effects of a Virtual Character on Learning and Engagement in Serious Games

Virtual characters have been employed for many purposes including interacting with players of serious games, with a purpose to increase engagement. These characters are often embodied conversational agents playing diverse roles, such as demonstrators, guides, teachers or interviewers. Recently, much research has been conducted into properties that affect the realism and plausibility of virtual characters, but it is less clear whether the inclusion of interactive agents in serious applications can enhance a user’s engagement with the application, or indeed increase efficacy. In a first step towards answering these questions, we conducted a study where a Virtual Learning Environment was used to examine the effect of employing a virtual character to deliver a lesso

In God’s Land: Cinematic Affect, Animation and the Perceptual Dilemmas of Slow Violence

In this paper, I argue that Indian independent filmmaker Pankaj Rishi Kumar\u27s documentary In God’s Land (2012) blends animation and live-action to illuminate the destructive nuances of postcolonial literary scholar, Rob Nixon\u27s notion of slow violence. In turning to cinema, I also suggest that In God’s Land’s “aesthetic strategies” further eco-film scholarship’s recent interests in animation, which have tended to highlight the mode\u27s feel good affect. I draw attention to In God\u27s Land\u27s hybrid of dark, discordant animation spectacle interspliced in the documentary live-action to articulate the potential of eco-animation outside of this affect. Ultimately, the film not only draws attention to animation’s non-playful affect—its potentials and dilemmas, but I also suggest that reading such a film adds postcolonial understandings of cinema beyond the Western/Japanese center on with eco-animation scholars have so far focused

Exploring Virtual Reality and Doppelganger Avatars for the Treatment of Chronic Back Pain

Cognitive-behavioral models of chronic pain assume that fear of pain and subsequent avoidance behavior contribute to pain chronicity and the maintenance of chronic pain. In chronic back pain (CBP), avoidance of movements often plays a major role in pain perseverance and interference with daily life activities. In treatment, avoidance is often addressed by teaching patients to reduce pain behaviors and increase healthy behaviors. The current project explored the use of personalized virtual characters (doppelganger avatars) in virtual reality (VR), to influence motor imitation and avoidance, fear of pain and experienced pain in CBP. We developed a method to create virtual doppelgangers, to animate them with movements captured from real-world models, and to present them to participants in an immersive cave virtual environment (CAVE) as autonomous movement models for imitation. Study 1 investigated interactions between model and observer characteristics in imitation behavior of healthy participants. We tested the hypothesis that perceived affiliative characteristics of a virtual model, such as similarity to the observer and likeability, would facilitate observers’ engagement in voluntary motor imitation. In a within-subject design (N=33), participants were exposed to four virtual characters of different degrees of realism and observer similarity, ranging from an abstract stickperson to a personalized doppelganger avatar designed from 3d scans of the observer. The characters performed different trunk movements and participants were asked to imitate these. We defined functional ranges of motion (ROM) for spinal extension (bending backward, BB), lateral flexion (bending sideward, BS) and rotation in the horizontal plane (RH) based on shoulder marker trajectories as behavioral indicators of imitation. Participants’ ratings on perceived avatar appearance were recorded in an Autonomous Avatar Questionnaire (AAQ), based on an explorative factor analysis. Linear mixed effects models revealed that for lateral flexion (BS), a facilitating influence of avatar type on ROM was mediated by perceived identification with the avatar including avatar likeability, avatar-observer-similarity and other affiliative characteristics. These findings suggest that maximizing model-observer similarity may indeed be useful to stimulate observational modeling. Study 2 employed the techniques developed in study 1 with participants who suffered from CBP and extended the setup with real-world elements, creating an immersive mixed reality. The research question was whether virtual doppelgangers could modify motor behaviors, pain expectancy and pain. In a randomized controlled between-subject design, participants observed and imitated an avatar (AVA, N=17) or a videotaped model (VID, N=16) over three sessions, during which the movements BS and RH as well as a new movement (moving a beverage crate) were shown. Again, self-reports and ROMs were used as measures. The AVA group reported reduced avoidance with no significant group differences in ROM. Pain expectancy increased in AVA but not VID over the sessions. Pain and limitations did not significantly differ. We observed a moderation effect of group, with prior pain expectancy predicting pain and avoidance in the VID but not in the AVA group. This can be interpreted as an effect of personalized movement models decoupling pain behavior from movement-related fear and pain expectancy by increasing pain tolerance and task persistence. Our findings suggest that personalized virtual movement models can stimulate observational modeling in general, and that they can increase pain tolerance and persistence in chronic pain conditions. Thus, they may provide a tool for exposure and exercise treatments in cognitive behavioral treatment approaches to CBP

To Affinity and Beyond: Interactive Digital Humans as a Human Computer Interface

The field of human computer interaction is increasingly exploring the use of more natural, human-like user interfaces to build intelligent agents to aid in everyday life. This is coupled with a move to people using ever more realistic avatars to represent themselves in their digital lives. As the ability to produce emotionally engaging digital human representations is only just now becoming technically possible, there is little research into how to approach such tasks. This is due to both technical complexity and operational implementation cost. This is now changing as we are at a nexus point with new approaches, faster graphics processing and enabling new technologies in machine learning and computer vision becoming available. I articulate the issues required for such digital humans to be considered successfully located on the other side of the phenomenon known as the Uncanny Valley. My results show that a complex mix of perceived and contextual aspects affect the sense making on digital humans and highlights previously undocumented effects of interactivity on the affinity. Users are willing to accept digital humans as a new form of user interface and they react to them emotionally in previously unanticipated ways. My research shows that it is possible to build an effective interactive digital human that crosses the Uncanny Valley. I directly explore what is required to build a visually realistic digital human as a primary research question and I explore if such a realistic face provides sufficient benefit to justify the challenges involved in building it. I conducted a Delphi study to inform the research approaches and then produced a complex digital human character based on these insights. This interactive and realistic digital human avatar represents a major technical undertaking involving multiple teams around the world. Finally, I explored a framework for examining the ethical implications and signpost future research areas

Anonymous Panda: preserving anonymity and expressiveness in online mental health platforms

Digital solutions that allow people to seek treatment, such as online psychological interventions and other technology-mediated therapies, have been developed to assist individuals with mental health disorders. Such approaches may raise privacy concerns about the use of people’s data and the safety of their mental health information. This work uses cutting-edge computer graphics technology to develop a novel system capable of increasing anonymity while maintaining expressiveness in computer-mediated mental health interventions. According to our preliminary findings, we were able to customize a realistic avatar using Live Link, Metahumans, and Unreal Engine 4 (UE4) with the same emotional depth as a real person. Furthermore, these findings showed that the virtual avatars’ inability to express themselves through hand motion gave the impression that they were acting in an unnatural way. By including the hand tracking feature using the Leap Motion Controller, we were able to improve our comprehension of the prospective use of ultra-realistic virtual human avatars in video conferencing therapy, i.e., both studies helped us understand how vital facial and body expressions are and how problematic their absence is in communicating with others.Soluções digitais que permitem às pessoas procurar tratamento, tais como terapias psicológicas online e outras terapias com recurso à tecnologia, foram desenvolvidas para ajudar indivíduos com distúrbios de saúde mental. Tais abordagens podem suscitar preocupações sobre a privacidade na utilização dos dados das pessoas e a segurança da informação sobre a sua saúde mental. Este trabalho utiliza tecnologia de ponta em computação gráfica para desenvolver um sistema inovador capaz de aumentar o anonimato, mantendo simultaneamente a expressividade nas inter venções de saúde mental mediadas por computador. Segundo os nossos resultados preliminares, conseguimos personalizar um avatar realista usando Live Link, Metahumans, e Unreal Engine 4 (UE4) com a mesma profundidade emocional que uma pessoa real. Além disso, os resultados mostraram que a incapacidade dos avatares virtuais de se expressarem através do movimento das mãos deu a impressão de que estavam a agir de uma forma pouco natural. Ao incluir a função de rastreio das mãos utilizando o Leap Motion Controller, conseguimos melhorar a nossa compreensão do uso prospetivo de avatares humanos virtuais e ultrarrealistas na terapia de videoconferência, ou seja, os estudos realizados ajudaram-nos a compreender como as expressões faciais e corporais são vitais e como a sua ausência é problemática na comunicação com os outros