Learning and Mining Player Motion Profiles in Physically Interactive Robogames

Physically-Interactive RoboGames (PIRG) are an emerging application whose aim is to develop robotic agents able to interact and engage humans in a game situation. In this framework, learning a model of players’ activity is relevant both to understand their engagement, as well as to understand specific strategies they adopted, which in turn can foster game adaptation. Following such directions and given the lack of quantitative methods for player modeling in PIRG, we propose a methodology for representing players as a mixture of existing player’s types uncovered from data. This is done by dealing both with the intrinsic uncertainty associated with the setting and with the agent necessity to act in real time to support the game interaction. Our methodology first focuses on encoding time series data generated from player-robot interaction into images, in particular Gramian angular field images, to represent continuous data. To these, we apply latent Dirichlet allocation to summarize the player’s motion style as a probabilistic mixture of different styles discovered from data. This approach has been tested in a dataset collected from a real, physical robot game, where activity patterns are extracted by using a custom three-axis accelerometer sensor module. The obtained results suggest that the proposed system is able to provide a robust description for the player interaction

Le mouvement expressif du corps entier : variabilités intra-individuelles dans des contextes affectifs et interactifs

Movement is a major component of our daily life. Every day we use it to both perform simple and essential task and to communicate. Intentionnal or not our movements sign our intraindividual and interindividual differences liked to our status, intentions and affects. In the same day, the cinematic of our movements evolve and adapt according of our social environnement. Distinction between movements of robot – virtual character and human movement is that the latter can vary. Indeed an identical movement made twice by a human will not be perfectly the same. However this specificity tends to change. From Darwin’s first works studying the impact of affect on movement to recent studies about movement expressivity in various interactive context (e.g. man-woman, student-professor interaction) and various applications (e.g. Autism, Exergames) researchers and entreprises seek to implement this human specificity in human computer interaction (HCI). Based on social science, movement science and computer science, this multidisciplinary work contributes to the understanding of action and perception of expressive movements thanks to three studies in coach-student sport context interaction. The first study aims at understanding how the perceived affect impact the expressivity of human movement. In the second study we examine dyadic interactions involving different status of participants and social set-up. Finaly we desgined an expressive full-body virtual agent and used it in an interactive trask. The main contribution of this PhD thesis is to show that expressivity features computed from different time-series (Energy, Caractère franc,rigidité and sptatial extent) are relevant to discriminate participants’ affects, status and thoughts. One goal and possible application of this work is the design of a virtual trainer allowing credible and dynamic full-body interactions thanks to its expressive movements.Le mouvement est une composante primordiale et nécessaire de notre existence. Nous l’utilisons tous les jours pour accomplir des tâches simples et essentielles, mais également pour communiquer. Que cela soit intentionnel ou non, nos mouvements signent nos différences interindividuelles, mais aussi intra-individuelles liées à nos états émotionnels, notre status, et nos intentions. Au cours d’une même journée, la cinématique de nos mouvements tend inéluctablement à évoluer et à s’adapter en fonction de notre environnement social. Ce qui différencie le mouvement des robots et des personnages virtuels de celui des humains est sa capacité à varier, un humain ne reproduisant jamais deux mouvements identiques. Néanmoins, ce contraste est de moins en moins évident. Depuis Darwin et ses travaux sur l’impact des émotions dans le mouvement jusqu’aux plus récentes études sur l’expressivité du mouvement dans des contextes d’interactions (p. ex. homme femme, interaction professeur-élève) et d’applications variées (p. ex. autisme, exergame), les chercheurs et entreprises cherchent à implémenter la variabilité du mouvement biologique humain dans les interactions homme-machine (IHM). En s’appuyant sur les Sciences sociales, du mouvement et de l’informatique, ce travail doctoral multidisciplinaire contribue à la compréhension de l’action et de la perception de mouvement expressif à travers trois études dans un contexte sportif d’interaction coach-élève. La première étude a pour objectif de comprendre comment l’expressivité du mouvement humain signe l’émotion perçue. Dans la seconde étude, nous envisageons plusieurs dyades composées de participants dont les status de passations et les conditions expérimentales évoluaient. Enfin, une dernière expérience orientée IHM a été réalisée. Au cours de celle-ci, des personnages virtuels expressifs ont été conçus pour interagir non verbalement avec les participants. Les résultats de ces travaux permettent de mettre en évidence que certains paramètres de l’expressivité du mouvement issue de séries temporelles (ST) (Energie, Caratère direct, Rigidité et l’étendue) sont nécessaires pour discriminer les affects, les status et les ressentis des participants au sein des interactions. La visée applicative de ce travail doctoral est la création d’un coach virtuel qui, au moyen de ces mouvements expressifs, permet une interaction dynamique et crédible

How Shall I Count the Ways? A Method for Quantifying the Qualitative Aspects of Unscripted Movement With Laban Movement Analysis

There is significant clinical evidence showing that creative and expressive movement processes involved in dance/movement therapy (DMT) enhance psycho-social well-being. Yet, because movement is a complex phenomenon, statistically validating which aspects of movement change during interventions or lead to significant positive therapeutic outcomes is challenging because movement has multiple, overlapping variables appearing in unique patterns in different individuals and situations. One factor contributing to the therapeutic effects of DMT is movement’s effect on clients’ emotional states. Our previous study identified sets of movement variables which, when executed, enhanced specific emotions. In this paper, we describe how we selected movement variables for statistical analysis in that study, using a multi-stage methodology to identify, reduce, code, and quantify the multitude of variables present in unscripted movement. We suggest a set of procedures for using Laban Movement Analysis (LMA)-described movement variables as research data. Our study used LMA, an internationally accepted comprehensive system for movement analysis, and a primary DMT clinical assessment tool for describing movement. We began with Davis’s (1970) three-stepped protocol for analyzing movement patterns and identifying the most important variables: (1) We repeatedly observed video samples of validated (Atkinson et al., 2004) emotional expressions to identify prevalent movement variables, eliminating variables appearing minimally or absent. (2) We use the criteria repetition, frequency, duration and emphasis to eliminate additional variables. (3) For each emotion, we analyzed motor expression variations to discover how variables cluster: first, by observing ten movement samples of each emotion to identify variables common to all samples; second, by qualitative analysis of the two best-recognized samples to determine if phrasing, duration or relationship among variables was significant. We added three new steps to this protocol: (4) we created Motifs (LMA symbols) combining movement variables extracted in steps 1–3; (5) we asked participants in the pilot study to move these combinations and quantify their emotional experience. Based on the results of the pilot study, we eliminated more variables; (6) we quantified the remaining variables’ prevalence in each Motif for statistical analysis that examined which variables enhanced each emotion. We posit that our method successfully quantified unscripted movement data for statistical analysis

Automated Analysis of Synchronization in Human Full-body Expressive Movement

The research presented in this thesis is focused on the creation of computational models for the study of human full-body movement in order to investigate human behavior and non-verbal communication. In particular, the research concerns the analysis of synchronization of expressive movements and gestures. Synchronization can be computed both on a single user (intra-personal), e.g., to measure the degree of coordination between the joints\u2019 velocities of a dancer, and on multiple users (inter-personal), e.g., to detect the level of coordination between multiple users in a group. The thesis, through a set of experiments and results, contributes to the investigation of both intra-personal and inter-personal synchronization applied to support the study of movement expressivity, and improve the state-of-art of the available methods by presenting a new algorithm to perform the analysis of synchronization

Towards Balancing Fun and Exertion in Exergames: Exploring the Impact of Movement-Based Controller Devices, Exercise Concepts, Game Adaptivity and Player Modes on Player Experience and Training Intensity in Different Exergame Settings

Physical inactivity remains one of the biggest societal challenges of the 21st century. The gaming industry and the fitness sector have responded to this alarming fact by introducing game-based or gamified training scenarios and thus established the promising trend of exergaming. Exergames – games controlled by active (whole) body movements – have been extolled as potential attractive and effective training tools. However, the majority of the exergames do not meet the required intensity or effectiveness, nor do they induce the intended training adherence or long-term motivation. One reason for this is that the evaluated exergames were often not co-designed with the user group to meet their specific needs and preferences, nor were they co-designed with an interdisciplinary expert team of game designers (to ensure a good gaming experience) and sports scientists (for a great training experience). Accordingly, the research results from studies with these exergames are rather limited. To fully exploit the potential of these innovative movement tools and to establish them as attractive and effective training approach, it is necessary to understand and explore both the underlying interdisciplinary theories and concepts as well as possible design approaches and their impact on the game and training experience. This dissertation aims to contribute to a better understanding of well-balanced exergame design. It explores and evaluates how different movement-based control devices, exercise concepts, game adaptations, and player modes influence the attractiveness and effectiveness of exergames. The work provides theoretical and practical contributions to the problem area of effective and attractive exergames. For this purpose, a research and development (R&D) approach with iterative phases was followed. As preliminary work for the contributions of this dissertation, exergames were approached from a theoretical perspective. Underlying multidisciplinary theories and concepts of exergames from relevant fields were analyzed and a generic framework was built, which structured the findings based on three interdependent dimensions: the player, the game controller, and the virtual game scenario. Some commercially available exergames were explored to verify the theory-based assumption that the interposition of technology brings specific transformations in the coupling of perception and action that do not occur in real sports situations. Among other things, the comparative pilot study showed that two different controllers (one gesture-based and one haptic device), which allowed for different physical input, were likely to induce diverse gameplay experiences (e.g., higher feeling of flow and self-location when playing with the haptic device) with differently skilled players. However, certain design-specific differences in the two exergame conditions meant that these results could only be interpreted as a first trend. To overcome the limitations of this preliminary study approach (e.g., unequal game design of the commercial exergames and very sports-specific movement concept), Plunder Planet, an adaptive exergame environment, was iteratively designed with and for children and allowed for a single- and cooperative multiplayer experience with two different controller devices. The user-centered design was further informed by insights from the growing body of related R&D work in the field of exergames. The first study presented in this dissertation compared the subjectively experienced attractiveness and effectiveness of Plunder Planet when played with different motion-based controllers. Besides a generally great acceptance of the exergame, it was found that the haptic full-body motion controller provided physical guidance and a more cognitively and coordinatively challenging workout, which was more highly rated by experienced gamers with fewer athletic skills. The gesture-based Kinect sensor felt more natural, allowed more freedom of movement, and provided a rather physically intense but cognitively less challenging workout, which was more highly rated by athletic players with less gameplay experience. Furthermore, experiments were made with an exploratory adaptive algorithm that enabled the cognitive and the physical challenge of the exergame to be manually adapted in real-time based on the player’s fitness and gaming skills. The first and the second study also compared an adaptive with a non-adaptive single player version of Plunder Planet. It could be shown that the (well-balanced) adaptive version of the exergame was better valued than the non-adaptive version with regard to the experienced and measured attractiveness (motivation, game flow, spatial presence experience, balance of cognitive and physical challenge) and effectiveness (heart rate, physical exertion, balance of cognitive and physical challenge) by differently skilled players. Finally, and contrary to the findings from related work, the results of the third study proved that the specifically designed controller technology could be used as an “enabler”, “supporter” and “shaper” of bodily interplay in social exergaming. Based on these promising findings, the goal became to further explore the effectiveness of exergames, refine the adaptive game difficulty algorithm, and explore further attractiveness- and motivation-boosting design approaches. Therefore, the ExerCube, a physically immersive and adaptive fitness game setting, was developed. It was iteratively designed with and for adults and allowed for cooperatively and competitive exergame experiences. With its physically immersive game setup, the ExerCube combines a mixed version of the advantages of both previously tested controllers. A coordinatively and cognitively challenging functional workout protocol with scalable intensity (moderate to high) was developed and the subjective experience of the ExerCube training was compared with a conventional functional training with a personal trainer. The fourth study showed that the game-based training gave signs of reaching a similar intensity to the personal training, but was more highly rated for flow, motivation, and enjoyment. Based on this exploratory comparison of the ExerCube with a personal trainer session, valuable avenues for further design could be identified. Among other things, it could be proved that the player’s focus during the ExerCube session was more on the game than on the own body. Players experienced stronger physical exertion and social pressure with the personal trainer and a stronger cognitive exertion and involvement with the ExerCube. Furthermore, a refined version of the previously tested adaptive game difficulty algorithm was implemented and automated for the first time for purpose of this study. Again it was shown that the adaptive version had benefits with regard to subjectively experienced attractiveness (motivation, game flow, balance of cognitive and physical challenge) and effectiveness (physical exertion, balance of cognitive and physical challenge) compared to the non-adaptive version. In order to further enhance the gaming experience, experiments were also conducted with sound designs and an adaptive audio design with adaptive background music and sound feedback was implemented. It was found to be a promising and beneficial add-on for a user-centered attractive exergame design. To inform the design of a multiplayer version of the ExerCube, different social play mechanics were explored in the fifth study. This resulted in differently balanced experiences of fun, and in physical as well as cognitive exertion. As the preliminary comparative evaluation of the subjectively experienced effectiveness and attractiveness of an ExerCube session and a personal trainer session could prove the general feasibility of the concept and revealed the first indications of the intensity of the ExerCube’s training concept, the objectively measured effectiveness of a single ExerCube session with a functional high-intensity interval training (fHIIT) with a personal trainer was compared in a final sixth study, and after another design iteration. Again, the subjectively experienced attractiveness of both conditions was assessed. It could be shown that the ExerCube is a feasible training device for training at fHIIT-level. While physical exertion was slightly lower than in the conventional fHIIT condition, the ExerCube condition’s average heart rate values reached the fHIIT threshold and also yielded significantly better results for flow, enjoyment, and motivation. The ExerCube training also resulted in a subjectively experienced higher cognitive load (dual-domain training). To sum up, it can be stated that this dissertation provides valuable and fundamental research contributions to the promising field of exergames as attractive and effective training tools. Furthermore, important contributions to design questions in this field could be developed. Since this field is still relatively unexplored, the work presented creates a sound basis for future R&D work in this area

Social Perception of Pedestrians and Virtual Agents Using Movement Features

In many tasks such as navigation in a shared space, humans explicitly or implicitly estimate social information related to the emotions, dominance, and friendliness of other humans around them. This social perception is critical in predicting others’ motions or actions and deciding how to interact with them. Therefore, modeling social perception is an important problem for robotics, autonomous vehicle navigation, and VR and AR applications. In this thesis, we present novel, data-driven models for the social perception of pedestrians and virtual agents based on their movement cues, including gaits, gestures, gazing, and trajectories. We use deep learning techniques (e.g., LSTMs) along with biomechanics to compute the gait features and combine them with local motion models to compute the trajectory features. Furthermore, we compute the gesture and gaze representations using psychological characteristics. We describe novel mappings between these computed gaits, gestures, gazing, and trajectory features and the various components (emotions, dominance, friendliness, approachability, and deception) of social perception. Our resulting data-driven models can identify the dominance, deception, and emotion of pedestrians from videos with an accuracy of more than 80%. We also release new datasets to evaluate these methods. We apply our data-driven models to socially-aware robot navigation and the navigation of autonomous vehicles among pedestrians. Our method generates robot movement based on pedestrians’ dominance levels, resulting in higher rapport and comfort. We also apply our data-driven models to simulate virtual agents with desired emotions, dominance, and friendliness. We perform user studies and show that our data-driven models significantly increase the user’s sense of social presence in VR and AR environments compared to the baseline methods.Doctor of Philosoph

Multisensory learning in adaptive interactive systems

The main purpose of my work is to investigate multisensory perceptual learning and sensory integration in the design and development of adaptive user interfaces for educational purposes. To this aim, starting from renewed understanding from neuroscience and cognitive science on multisensory perceptual learning and sensory integration, I developed a theoretical computational model for designing multimodal learning technologies that take into account these results. Main theoretical foundations of my research are multisensory perceptual learning theories and the research on sensory processing and integration, embodied cognition theories, computational models of non-verbal and emotion communication in full-body movement, and human-computer interaction models. Finally, a computational model was applied in two case studies, based on two EU ICT-H2020 Projects, "weDRAW" and "TELMI", on which I worked during the PhD