Generating, animating, and rendering varied individuals for real-time crowds

To simulate realistic crowds of virtual humans in real time, three main requirements need satisfaction. First of all, quantity, i.e., the ability to simulate thousands of characters. Secondly, quality, because each virtual human composing a crowd needs to look unique in its appearance and animation. Finally, efficiency is paramount, for an operation usually efficient on a single virtual human, becomes extremely costly when applied on large crowds. Developing an architecture able to manage all three aspects is a challenging problem that we have addressed in our research. Our first contribution is an efficient and versatile architecture called YaQ, able to simulate thousands of characters in real time. This platform, developed at EPFL-VRLab, results from several years of research and integrates state-of-the-art techniques at all levels: YaQ aims at providing efficient algorithms and real-time solutions for populating globally and massively large-scale empty environments. YaQ thus fits various application domains, such as video games and virtual reality. Our architecture is especially efficient in managing the large quantity of data that is used to simulate crowds. In order to simulate large crowds, many instances of a small set of human templates have to be generated. From this starting point, if no care is taken to vary each character individually, many clones appear in the crowd. We present several algorithms to make each individual unique in the crowd. Firstly, we introduce a new method to distinguish body parts of a human and apply detailed color variety and patterns to each one of them. Secondly, we present two techniques to modify the shape and profile of a virtual human: a simple and efficient method for attaching accessories to individuals, and efficient tools to scale the skeleton and mesh of an instance. Finally, we also contribute to varying individuals' animation by introducing variations to the upper body movements, thus allowing characters to make a phone call, have a hand in their pocket, or carry heavy accessories, etc. To achieve quantity in a crowd, levels of detail need to be used. We explore the most adequate solutions to simulate large crowds with levels of detail, while avoiding disturbing switches between two different representations of a virtual human. To do so, we develop solutions to make most variety techniques scalable to all levels of detail

An Information-Theoretic Framework for Consistency Maintenance in Distributed Interactive Applications

Distributed Interactive Applications (DIAs) enable geographically dispersed users to interact with each other in a virtual environment. A key factor to the success of a DIA is the maintenance of a consistent view of the shared virtual world for all the participants. However, maintaining consistent states in DIAs is difficult under real networks. State changes communicated by messages over such networks suffer latency leading to inconsistency across the application. Predictive Contract Mechanisms (PCMs) combat this problem through reducing the number of messages transmitted in return for perceptually tolerable inconsistency. This thesis examines the operation of PCMs using concepts and methods derived from information theory. This information theory perspective results in a novel information model of PCMs that quantifies and analyzes the efficiency of such methods in communicating the reduced state information, and a new adaptive multiple-model-based framework for improving consistency in DIAs. The first part of this thesis introduces information measurements of user behavior in DIAs and formalizes the information model for PCM operation. In presenting the information model, the statistical dependence in the entity state, which makes using extrapolation models to predict future user behavior possible, is evaluated. The efficiency of a PCM to exploit such predictability to reduce the amount of network resources required to maintain consistency is also investigated. It is demonstrated that from the information theory perspective, PCMs can be interpreted as a form of information reduction and compression. The second part of this thesis proposes an Information-Based Dynamic Extrapolation Model for dynamically selecting between extrapolation algorithms based on information evaluation and inferred network conditions. This model adapts PCM configurations to both user behavior and network conditions, and makes the most information-efficient use of the available network resources. In doing so, it improves PCM performance and consistency in DIAs

Emotion and Stress Recognition Related Sensors and Machine Learning Technologies

This book includes impactful chapters which present scientific concepts, frameworks, architectures and ideas on sensing technologies and machine learning techniques. These are relevant in tackling the following challenges: (i) the field readiness and use of intrusive sensor systems and devices for capturing biosignals, including EEG sensor systems, ECG sensor systems and electrodermal activity sensor systems; (ii) the quality assessment and management of sensor data; (iii) data preprocessing, noise filtering and calibration concepts for biosignals; (iv) the field readiness and use of nonintrusive sensor technologies, including visual sensors, acoustic sensors, vibration sensors and piezoelectric sensors; (v) emotion recognition using mobile phones and smartwatches; (vi) body area sensor networks for emotion and stress studies; (vii) the use of experimental datasets in emotion recognition, including dataset generation principles and concepts, quality insurance and emotion elicitation material and concepts; (viii) machine learning techniques for robust emotion recognition, including graphical models, neural network methods, deep learning methods, statistical learning and multivariate empirical mode decomposition; (ix) subject-independent emotion and stress recognition concepts and systems, including facial expression-based systems, speech-based systems, EEG-based systems, ECG-based systems, electrodermal activity-based systems, multimodal recognition systems and sensor fusion concepts and (x) emotion and stress estimation and forecasting from a nonlinear dynamical system perspective

Social Touch

Interpersonal or social touch is an intuitive and powerful way to express and communicate emotions, comfort a friend, bond with teammates, comfort a child in pain, and soothe someone who is stressed. If there is one thing that the current pandemic is showing us, it is that social distancing can make some people crave physical interaction through social touch. The notion of “skin-hunger” has become tangible for many.Social touch differs at a functional and anatomical level from discriminative touch, and has clear effects at physiological, emotional, and behavioural levels. Social touch is a topic in psychology (perception, emotion, behaviour), neuroscience (neurophysiological pathways), computer science (mediated touch communication), engineering (haptic devices), robotics (social robots that can touch), humanities (science and technology studies), and sociology (the social implications of touch). Our current scientific knowledge of social touch is scattered across disciplines and not yet adequate for the purpose of meeting today's challenges of connecting human beings through the mediating channel of technology

Immersive Participation:Futuring, Training Simulation and Dance and Virtual Reality

Dance knowledge can inform the development of scenario design in immersive digital simulation environments by strengthening a participant’s capacity to learn through the body. This study engages with processes of participatory practice that question how the transmission and transfer of dance knowledge/embodied knowledge in immersive digital environments is activated and applied in new contexts. These questions are relevant in both arts and industry and have the potential to add value and knowledge through crossdisciplinary collaboration and exchange. This thesis consists of three different research projects all focused on observation, participation, and interviews with experts on embodiment in digital simulation. The projects were chosen to provide a range of perspectives across dance, industry and futures studies. Theories of embodied cognition, in particular the notions of the extended body, distributed cognition, enactment and mindfulness, offer critical lenses through which to explore the relationship of embodied integration and participation within immersive digital environments. These areas of inquiry lead to the consideration of how language from the field of computer science can assist in describing somatic experience in digital worlds through a discussion of the emerging concepts of mindfulness, wayfinding, guided movement and digital kinship. These terms serve as an example of how the mutability of language became part of the process as terms applied in disparate disciplines were understood within varying contexts. The analytic tools focus on applying a posthuman view, speculation through a futures ethnography, and a cognitive ethnographical approach to my research project. These approaches allowed me to examine an ecology of practices in order to identify methods and processes that can facilitate the transmission and transfer of embodied knowledge within a community of practice. The ecological components include dance, healthcare, transport, education and human/computer interaction. These fields drove the data collection from a range of sources including academic papers, texts, specialists’ reports, scientific papers, interviews and conversations with experts and artists.The aim of my research is to contribute both a theoretical and a speculative understanding of processes, as well as tools applicable in the transmission of embodied knowledge in virtual dance and arts environments as well as digital simulation across industry. Processes were understood theoretically through established studies in embodied cognition applied to workbased training, reinterpreted through my own movement study. Futures methodologies paved the way for speculative processes and analysis. Tools to choreograph scenario design in immersive digital environments were identified through the recognition of cross purpose language such as mindfulness, wayfinding, guided movement and digital kinship. Put together, the major contribution of this research is a greater understanding of the value of dance knowledge applied to simulation developed through theoretical and transformational processes and creative tools