ADAPT: The Agent Development and Prototyping Testbed

We present ADAPT, a flexible platform for designing and authoring functional, purposeful human characters in a rich virtual environment. Our framework incorporates character animation, navigation, and behavior with modular interchangeable components to produce narrative scenes. Our animation system provides locomotion, reaching, gaze tracking, gesturing, sitting, and reactions to external physical forces, and can easily be extended with more functionality due to a decoupled, modular structure. Additionally, our navigation component allows characters to maneuver through a complex environment with predictive steering for dynamic obstacle avoidance. Finally, our behavior framework allows a user to fully leverage a character’s animation and navigation capabilities when authoring both individual decision-making and complex interactions between actors using a centralized, event-driven model

Automatic Replication of Teleoperator Head Movements and Facial Expressions on a Humanoid Robot

Robotic telepresence aims to create a physical presence for a remotely located human (teleoperator) by reproducing their verbal and nonverbal behaviours (e.g. speech, gestures, facial expressions) on a robotic platform. In this work, we propose a novel teleoperation system that combines the replication of facial expressions of emotions (neutral, disgust, happiness, and surprise) and head movements on the fly on the humanoid robot Nao. Robots' expression of emotions is constrained by their physical and behavioural capabilities. As the Nao robot has a static face, we use the LEDs located around its eyes to reproduce the teleoperator expressions of emotions. Using a web camera, we computationally detect the facial action units and measure the head pose of the operator. The emotion to be replicated is inferred from the detected action units by a neural network. Simultaneously, the measured head motion is smoothed and bounded to the robot's physical limits by applying a constrained-state Kalman filter. In order to evaluate the proposed system, we conducted a user study by asking 28 participants to use the replication system by displaying facial expressions and head movements while being recorded by a web camera. Subsequently, 18 external observers viewed the recorded clips via an online survey and assessed the quality of the robot's replication of the participants' behaviours. Our results show that the proposed teleoperation system can successfully communicate emotions and head movements, resulting in a high agreement among the external observers (ICC_E = 0.91, ICC_HP = 0.72).This work was funded by the EPSRC under its IDEAS Factory Sandpits call on Digital Personhood (Grant Ref· EP/L00416X/1)

Interfaces for human-centered production and use of computer graphics assets

L'abstract è presente nell'allegato / the abstract is in the attachmen

심층 강화학습을 이용한 사람의 모션을 통한 이형적 캐릭터 제어기 개발

학위논문(석사) -- 서울대학교대학원 : 공과대학 컴퓨터공학부, 2022. 8. 서진욱.사람의 모션을 이용한 로봇 컨트롤 인터페이스는 사용자의 직관과 로봇의 모터 능력을 합하여 위험한 환경에서 로봇의 유연한 작동을 만들어낸다. 하지만, 휴머노이드 외의 사족보행 로봇이나 육족보행 로봇을 위한 모션 인터페이스를 디자인 하는 것은 쉬운일이 아니다. 이것은 사람과 로봇 사이의 형태 차이로 오는 다이나믹스 차이와 제어 전략이 크게 차이나기 때문이다. 우리는 사람 사용자가 움직임을 통하여 사족보행 로봇에서 부드럽게 여러 과제를 수행할 수 있게끔 하는 새로운 모션 제어 시스템을 제안한다. 우리는 우선 캡쳐한 사람의 모션을 상응하는 로봇의 모션으로 리타겟 시킨다. 이때 상응하는 로봇의 모션은 유저가 의도한 의미를 내포하게 되며, 우리는 이를 지도학습 방법과 후처리 기술을 이용하여 가능케 하였다. 그 뒤 우리는 모션을 모사하는 학습을 커리큘럼 학습과 병행하여 주어진 리타겟된 참조 모션을 따라가는 제어 정책을 생성하였다. 우리는 "전문가 집단"을 학습함으로 모션 리타게팅 모듈과 모션 모사 모듈의 성능을 크게 증가시켰다. 결과에서 볼 수 있듯, 우리의 시스템을 이용하여 사용자가 사족보행 로봇의 서있기, 앉기, 기울이기, 팔 뻗기, 걷기, 돌기와 같은 다양한 모터 과제들을 시뮬레이션 환경과 현실에서 둘 다 수행할 수 있었다. 우리는 연구의 성능을 평가하기 위하여 다양한 분석을 하였으며, 특히 우리 시스템의 각각의 요소들의 중요성을 보여줄 수 있는 실험들을 진행하였다.A human motion-based interface fuses operator intuitions with the motor capabilities of robots, enabling adaptable robot operations in dangerous environments. However, the challenge of designing a motion interface for non-humanoid robots, such as quadrupeds or hexapods, is emerged from the different morphology and dynamics of a human controller, leading to an ambiguity of control strategy. We propose a novel control framework that allows human operators to execute various motor skills on a quadrupedal robot by their motion. Our system first retargets the captured human motion into the corresponding robot motion with the operator's intended semantics. The supervised learning and post-processing techniques allow this retargeting skill which is ambiguity-free and suitable for control policy training. To enable a robot to track a given retargeted motion, we then obtain the control policy from reinforcement learning that imitates the given reference motion with designed curriculums. We additionally enhance the system's performance by introducing a set of experts. Finally, we randomize the domain parameters to adapt the physically simulated motor skills to real-world tasks. We demonstrate that a human operator can perform various motor tasks using our system including standing, tilting, manipulating, sitting, walking, and steering on both physically simulated and real quadruped robots. We also analyze the performance of each system component ablation study.1 Introduction 1 2 Related Work 5 2.1 Legged Robot Control 5 2.2 Motion Imitation 6 2.3 Motion-based Control 7 3 Overview 9 4 Motion Retargeting Module 11 4.1 Motion Retargeting Network 12 4.2 Post-processing for Consistency 14 4.3 A Set of Experts for Multi-task Support 15 5 Motion Imitation Module 17 5.1 Background: Reinforcement Learning 18 5.2 Formulation of Motion Imitation 18 5.3 Curriculum Learning over Tasks and Difficulties 21 5.4 Hierarchical Control with States 21 5.5 Domain Randomization 22 6 Results and Analysis 23 6.1 Experimental Setup 23 6.2 Motion Performance 24 6.3 Analysis 28 6.4 Comparison to Other Methods 31 7 Conclusion And Future Work 32 Bibliography 34 Abstract (In Korean) 44 감사의 글 45석

Dancing Media: The Contagious Movement of Posthuman Bodies (or Towards A Posthuman Theory of Dance)

My dissertation seeks to define a posthuman theory of dance through a historical study of the dancer as an instrument or technology for exploring emergent visual media, and by positioning screendance as an experimental technique for animating posthuman relation and thought. Commonly understood as ephemeral, dance is produced by assemblages that include bodies but are not limited to them. In this way, dance exceeds the human body. There is a central tension in the practice of dance, between the persistent presumption of the dancing body as a channel for human expression, and dance as a technicity of the body—a discipline and a practice of repeated gesture—that calls into question categories of the human. A posthuman theory of dance invites examination of such tensions and interrogates traditional notions of authenticity, ownership and commodification, as well as the bounded, individual subject who can assess the surrounding world with precise clarity, certain of where the human begins and ends. The guiding historical question for my dissertation is: if it is possible to describe both a modern form of posthuman dance (turn of the 19th-20th century), and a more recent form of posthuman dance (turn of the 20th-21st century), are they part of the same assemblage or are they constituted differently, and if so, how? Throughout my four chapters, I explore an array of case studies from early modernism to advanced capitalism, including Loie Fuller’s otherworldly stage dances; the scientific motion studies of Muybridge and Marey; Fritz Lang’s dancing maschinenmensch (or the first on-screen dancing machine) in the 1927 film Metropolis; the performances of singer-dancer hologram pop star, Hatsune Miku; and American engineering firm Boston Dynamics’ dancing military robots. The figure of the “dancing machine” (McCarren) is central to my project, especially given that dance has historically been used as a means of testing machines—from automata to robots to CGI images animated with MoCap—in their capacity to be lively or human-like. In each case, I am interested in how dance continues to be productive of some kind of subjectivity (or interiority, or “soul”), even in the absence of the human body, and how technique and gesture passes between bodies, both virtual and organic, dispersing agency often attributed to the human alone. I propose that a posthuman theory of dance is a necessary intervention to the broad and contradictory field of posthumanism because dance returns us to questions about bodies that are often suspiciously ignored in theories of posthumanism, especially regarding race (and historically racist categories of non/inhumanity), thereby exposing many of posthumanism’s biases, appropriations, dispossessions and erasures. Throughout my dissertation, I look to dance as both a concrete example and as a method of thinking through the potentials and limitations of posthumanism

Expressive movement generation with machine learning

Movement is an essential aspect of our lives. Not only do we move to interact with our physical environment, but we also express ourselves and communicate with others through our movements. In an increasingly computerized world where various technologies and devices surround us, our movements are essential parts of our interaction with and consumption of computational devices and artifacts. In this context, incorporating an understanding of our movements within the design of the technologies surrounding us can significantly improve our daily experiences. This need has given rise to the field of movement computing – developing computational models of movement that can perceive, manipulate, and generate movements. In this thesis, we contribute to the field of movement computing by building machine-learning-based solutions for automatic movement generation. In particular, we focus on using machine learning techniques and motion capture data to create controllable, generative movement models. We also contribute to the field by creating datasets, tools, and libraries that we have developed during our research. We start our research by reviewing the works on building automatic movement generation systems using machine learning techniques and motion capture data. Our review covers background topics such as high-level movement characterization, training data, features representation, machine learning models, and evaluation methods. Building on our literature review, we present WalkNet, an interactive agent walking movement controller based on neural networks. The expressivity of virtual, animated agents plays an essential role in their believability. Therefore, WalkNet integrates controlling the expressive qualities of movement with the goal-oriented behaviour of an animated virtual agent. It allows us to control the generation based on the valence and arousal levels of affect, the movement’s walking direction, and the mover’s movement signature in real-time. Following WalkNet, we look at controlling movement generation using more complex stimuli such as music represented by audio signals (i.e., non-symbolic music). Music-driven dance generation involves a highly non-linear mapping between temporally dense stimuli (i.e., the audio signal) and movements, which renders a more challenging modelling movement problem. To this end, we present GrooveNet, a real-time machine learning model for music-driven dance generation

The Turning, Stretching and Boxing Technique: a Direction Worth Looking Towards

3D avatar user interfaces (UI) are now used for many applications, a growing area for their use is serving location sensitive information to users as they need it while visiting or touring a building. Users communicate directly with an avatar rendered to a display in order to ask a question, get directions or partake in a guided tour and as a result of this kind of interaction with avatar UI, they have become a familiar part of modern human-computer interaction (HCI). However, if the viewer is not in the sweet spot (defined by Raskar et al. (1999) as a stationary viewing position at the optimal 90° angle to a 2D display) of the 2D display, the 3D illusion of the avatar deteriorates, which becomes evident as the user’s ability to interpret the avatar’s gaze direction towards points of interests (PoI) in the user’s real-world surroundings deteriorates also. This thesis combats the above problem by allowing the user to view the 3D avatar UI from outside the sweet spot, without any deterioration in the 3D illusion. The user does not lose their ability to interpret the avatar’s gaze direction and thus, the user experiences no loss in the perceived corporeal presence (Holz et al., 2011) for the avatar. This is facilitated by a three pronged graphical process called the Turning, Stretching and Boxing (TSB) technique, which maintains the avatar’s 3D illusion regardless of the user’s viewing angle and is achieved by using head-tracking data from the user captured by a Microsoft Kinect. The TSB technique is a contribution of this thesis because of how it is used with an avatar UI, where the user is free to move outside of the sweet spot without losing the 3D illusion of the rendered avatar. Then each consecutive empirical study evaluates the claims of the TSB Technique are also contributions of this thesis, those claims are as follows: (1) increase interpretability of the avatar’s gaze direction and (2) increase perception of corporeal presence for the avatar. The last of the empirical studies evaluates the use of 3D display technology in conjunction with the TSB technique. The results of Study 1 and Study 2 indicate that there is a significant increase in the participants’ abilities to interpret the avatar’s gaze direction when the TSB technique is switched on. The survey from Study 1 shows a significant increase in the perceived corporeal presence of the avatar when the TSB technique is switched on. The results from Study 3 indicate that there is no significant benefit for participants’ when interpreting the avatar’s gaze direction with 3D display technology turned on or off when the TSB technique is switched on

Designing Touchless Gestural Interfaces for Public Displays

Nell\u2019ultimo decennio, molti autori hanno studiato la possibilit\ue0 di utilizzare le interfacce a gesti come strumento innovativo per supportare l\u2019interazione con i computer. Inoltre, le recenti innovazioni tecnologiche hanno permesso di installare display interattivi in ambienti privati e pubblici. Tuttavia, l\u2019interattivit\ue0 di tali display \ue8 spesso basata sull\u2019uso di touchscreen, mentre tecnologie come i dispositivi Kinect-like vengono adottate molto pi\uf9 raramente, soprattutto se si considera l\u2019ambito dei display pubblici. Al giorno d\u2019oggi, l\u2019opportunit\ue0 di studiare le interfacce touchless per i display pubblici \ue8 diventata concreta, e rappresenta il campo di studio di diversi ricercatori. L\u2019obiettivo principale di questa tesi \ue8 quello di descrivere e studiare i problemi legati alla progettazione e all\u2019implementazione di un\u2019interfaccia grafica dedicata all\u2019interazione touchless a gesti con display pubblici. Ci\uf2 implica la necessit\ue0 di superare alcuni problemi tipici, sia dei display pubblici (ad esempio, l\u2019interaction blindness e l\u2019usabilit\ue0 immediata), che delle interfacce touchless (per esempio, comunicare che l\u2019interattivit\ue0 \ue8 gestuale). La tesi, inoltre, include uno studio che analizza quanto la presenza dell\u2019Avatar possa influire sulle interazioni degli utenti, in termini di carico di lavoro percepito, e quanto essa sia in grado di incoraggiare le interazioni a due mani. Poich\ue9 ABaToGI \ue8 stata progettata per i display pubblici, l\u2019interfaccia \ue8 stata anche inclusa in un\u2019installazione pubblica per essere valutata sul campo. I risultati di questo studio (e di quelli precedenti) sono stati quindi riassunti al fine di sviluppare una serie di linee guida per lo sviluppo di nuove interfacce touchless a gesti basata sull\u2019uso di un Avatar. La tesi si conclude con alcuni spunti di ricerca per il futuro.In the last decade, many authors have investigated and studied touchless and gestural interactions as a novel tool for interacting with computers. Moreover, technological innovations have allowed for installations of interactive displays in private and public places. However, interactivity is usually implemented by touchscreens, whereas technologies able to recognize body gestures are more rarely adopted, especially in integration with commercial public displays. Nowadays, the opportunity to investigate touchless interfaces for such systems has become concrete and studied by many researchers. Indeed, this interaction modality offers the possibility to overcome several issues that cannot be solved by touch-based solutions, e.g. keeping a high hygiene level of the screen surface, as well as providing big displays with interactive capabilities. The main goal of this thesis is to describe the design process for implementing touchless gestural interfaces for public displays. This implies the need for overcoming several typical issues of both public displays (e.g. interaction blindness, immediate usability) and touchless interfaces (e.g. communicating touchless interactivity). To this end, a novel Avatar-based Touchless Gestural Interface (or ABaToGI) has been developed, and its design process is described in the thesis, along with the user studies conducted for its evaluation. Moreover, the thesis analyzes how the presence of the Avatar may affect user interactions in terms of perceived cognitive workload, and if it may be able to foster bimanual interactions. Then, as ABaToGI was designed for public displays, it has been installed in an actual deployment in order to be evaluated in-the-wild (i.e. not in a lab setting). The resulting outcomes, along with the previously described studies, have been used to introduce a set of design guidelines for developing future touchless gestural interfaces, with a particular focus on Avatar-based ones. The results of this thesis provide also a basis for future research, which concludes this work

Responding to human full-body gestures embedded in motion data streams.

 This research created a neural-network enabled artificially intelligent performing agent that was able to learn to dance and recognise movement through a rehearsal and performance process with a human dancer. The agent exhibited emergent dance behaviour and successfully engaged in a live, semi-improvised dance performance with the human dancer