Faces and hands : modeling and animating anatomical and photorealistic models with regard to the communicative competence of virtual humans

In order to be believable, virtual human characters must be able to communicate in a human-like fashion realistically. This dissertation contributes to improving and automating several aspects of virtual conversations. We have proposed techniques to add non-verbal speech-related facial expressions to audiovisual speech, such as head nods for of emphasis. During conversation, humans experience shades of emotions much more frequently than the strong Ekmanian basic emotions. This prompted us to develop a method that interpolates between facial expressions of emotions to create new ones based on an emotion model. In the area of facial modeling, we have presented a system to generate plausible 3D face models from vague mental images. It makes use of a morphable model of faces and exploits correlations among facial features. The hands also play a major role in human communication. Since the basis for every realistic animation of gestures must be a convincing model of the hand, we devised a physics-based anatomical hand model, where a hybrid muscle model drives the animations. The model was used to visualize complex hand movement captured using multi-exposure photography.Um überzeugend zu wirken, müssen virtuelle Figuren auf dieselbe Art wie lebende Menschen kommunizieren können. Diese Dissertation hat das Ziel, verschiedene Aspekte virtueller Unterhaltungen zu verbessern und zu automatisieren. Wir führten eine Technik ein, die es erlaubt, audiovisuelle Sprache durch nichtverbale sprachbezogene Gesichtsausdrücke zu bereichern, wie z.B. Kopfnicken zur Betonung. Während einer Unterhaltung empfinden Menschen weitaus öfter Emotionsnuancen als die ausgeprägten Ekmanschen Basisemotionen. Dies bewog uns, eine Methode zu entwickeln, die Gesichtsausdrücke für neue Emotionen erzeugt, indem sie, ausgehend von einem Emotionsmodell, zwischen bereits bekannten Gesichtsausdrücken interpoliert. Auf dem Gebiet der Gesichtsmodellierung stellten wir ein System vor, um plausible 3D-Gesichtsmodelle aus vagen geistigen Bildern zu erzeugen. Dieses System basiert auf einem Morphable Model von Gesichtern und nutzt Korrelationen zwischen Gesichtszügen aus. Auch die Hände spielen ein große Rolle in der menschlichen Kommunikation. Da der Ausgangspunkt für jede realistische Animation von Gestik ein überzeugendes Handmodell sein muß, entwikkelten wir ein physikbasiertes anatomisches Handmodell, bei dem ein hybrides Muskelmodell die Animationen antreibt. Das Modell wurde verwendet, um komplexe Handbewegungen zu visualisieren, die aus mehrfach belichteten Photographien extrahiert worden waren

Towards a Smart Drone Cinematographer for Filming Human Motion

Affordable consumer drones have made capturing aerial footage more convenient and accessible. However, shooting cinematic motion videos using a drone is challenging because it requires users to analyze dynamic scenarios while operating the controller. In this thesis, our task is to develop an autonomous drone cinematography system to capture cinematic videos of human motion. We understand the system's filming performance to be influenced by three key components: 1) video quality metric, which measures the aesthetic quality -- the angle, the distance, the image composition -- of the captured video, 2) visual feature, which encapsulates the visual elements that influence the filming style, and 3) camera planning, which is a decision-making model that predicts the next best movement. By analyzing these three components, we designed two autonomous drone cinematography systems using both heuristic-based methods and learning-based methods.For the first system, we designed an Autonomous CinemaTography system -- "ACT" by proposing a viewpoint quality metric focusing on the visibility of the 3D human skeleton of the subject. We expanded the application of human motion analysis and simplified manual control by assisting viewpoint selection using a through-the-lens method. For the second system, we designed an imitation-based system that learns the artistic intention of the cameramen through watching professional aerial videos. We designed a camera planner that analyzes the video contents and previous camera motion to predict future camera motion. Furthermore, we propose a planning framework, which can imitate a filming style by ``seeing" only one single demonstration video of such style. We named it ``one-shot imitation filming." To the best of our knowledge, this is the first work that extends imitation learning to autonomous filming. Experimental results in both simulation and field test exhibit significant improvements over existing techniques and our approach managed to help inexperienced pilots capture cinematic videos

Relational agents : effecting change through human-computer relationships

Thesis (Ph. D.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2003.Includes bibliographical references (p. 205-219).What kinds of social relationships can people have with computers? Are there activities that computers can engage in that actively draw people into relationships with them? What are the potential benefits to the people who participate in these human-computer relationships? To address these questions this work introduces a theory of Relational Agents, which are computational artifacts designed to build and maintain long-term, social-emotional relationships with their users. These can be purely software humanoid animated agents--as developed in this work--but they can also be non-humanoid or embodied in various physical forms, from robots, to pets, to jewelry, clothing, hand-helds, and other interactive devices. Central to the notion of relationship is that it is a persistent construct, spanning multiple interactions; thus, Relational Agents are explicitly designed to remember past history and manage future expectations in their interactions with users. Finally, relationships are fundamentally social and emotional, and detailed knowledge of human social psychology--with a particular emphasis on the role of affect--must be incorporated into these agents if they are to effectively leverage the mechanisms of human social cognition in order to build relationships in the most natural manner possible. People build relationships primarily through the use of language, and primarily within the context of face-to-face conversation. Embodied Conversational Agents--anthropomorphic computer characters that emulate the experience of face-to-face conversation--thus provide the substrate for this work, and so the relational activities provided by the theory will primarily be specific types of verbal and nonverbal conversational behaviors used by people to negotiate and maintain relationships.(cont.) This work also provides an analysis of the types of applications in which having a human-computer relationship is advantageous to the human participant. In addition to applications in which the relationship is an end in itself (e.g., in entertainment systems), human-computer relationships are important in tasks in which the human is attempting to undergo some change in behavior or cognitive or emotional state. One such application is explored here: a system for assisting the user through a month-long health behavior change program in the area of exercise adoption. This application involves the research, design and implementation of relational agents as well as empirical evaluation of their ability to build relationships and effect change over a series of interactions with users.by Timothy Wallace Bickmore.Ph.D

New Game Physics - Added Value for Transdisciplinary Teams

This study focused on game physics, an area of computer game design where physics is applied in interactive computer software. The purpose of the research was a fresh analysis of game physics in order to prove that its current usage is limited and requires advancement. The investigations presented in this dissertation establish constructive principles to advance game physics design. The main premise was that transdisciplinary approaches provide significant value. The resulting designs reflected combined goals of game developers, artists and physicists and provide novel ways to incorporate physics into games. The applicability and user impact of such new game physics across several target audiences was thoroughly examined. In order to explore the transdisciplinary nature of the premise, valid evidence was gathered using a broad range of theoretical and practical methodologies. The research established a clear definition of game physics within the context of historical, technological, practical, scientific, and artistic considerations. Game analysis, literature reviews and seminal surveys of game players, game developers and scientists were conducted. A heuristic categorization of game types was defined to create an extensive database of computer games and carry out a statistical analysis of game physics usage. Results were then combined to define core principles for the design of unconventional new game physics elements. Software implementations of several elements were developed to examine the practical feasibility of the proposed principles. This research prototype was exposed to practitioners (artists, game developers and scientists) in field studies, documented on video and subsequently analyzed to evaluate the effectiveness of the elements on the audiences. The findings from this research demonstrated that standard game physics is a common but limited design element in computer games. It was discovered that the entertainment driven design goals of game developers interfere with the needs of educators and scientists. Game reviews exemplified the exaggerated and incorrect physics present in many commercial computer games. This “pseudo physics” was shown to have potentially undesired effects on game players. Art reviews also indicated that game physics technology remains largely inaccessible to artists. The principal conclusion drawn from this study was that the proposed new game physics advances game design and creates value by expanding the choices available to game developers and designers, enabling artists to create more scientifically robust artworks, and encouraging scientists to consider games as a viable tool for education and research. The practical portion generated tangible evidence that the isolated “silos” of engineering, art and science can be bridged when game physics is designed in a transdisciplinary way. This dissertation recommends that scientific and artistic perspectives should always be considered when game physics is used in computer-based media, because significant value for a broad range of practitioners in succinctly different fields can be achieved. The study has thereby established a state of the art research into game physics, which not only offers other researchers constructive principles for future investigations, but also provides much-needed new material to address the observed discrepancies in game theory and digital media design

Models of ICT Innovation. A Focus on the Cinema Sector

The report starts by looking at the competing and overlapping definitions of creative industries, media and content industries. Chapter 1 investigates the fate of R&D and innovation in the creative industries and in the broader Telecom Media and Technology sectors. Chapter 2 summarizes past studies on innovation in distinct media and content industries (videogames, music recording and newspapers publishing) and draws some lessons from them. Chapter 3 delves more deeply into the specific case of cinema. This chapter investigates the film industry's complex and evolving relationship with technologies and technological inventions. Chapter 4 offers a short cross-comparison with R&D in the book publishing industry. Chapter 5 deals with policy issues triggered by the observed digital changes. Chapter 6 concludes with a brief assessment of EU strengths and weaknesses, and offers some recommendations.JRC.J.3-Information Societ

MediaSync: Handbook on Multimedia Synchronization

This book provides an approachable overview of the most recent advances in the fascinating field of media synchronization (mediasync), gathering contributions from the most representative and influential experts. Understanding the challenges of this field in the current multi-sensory, multi-device, and multi-protocol world is not an easy task. The book revisits the foundations of mediasync, including theoretical frameworks and models, highlights ongoing research efforts, like hybrid broadband broadcast (HBB) delivery and users' perception modeling (i.e., Quality of Experience or QoE), and paves the way for the future (e.g., towards the deployment of multi-sensory and ultra-realistic experiences). Although many advances around mediasync have been devised and deployed, this area of research is getting renewed attention to overcome remaining challenges in the next-generation (heterogeneous and ubiquitous) media ecosystem. Given the significant advances in this research area, its current relevance and the multiple disciplines it involves, the availability of a reference book on mediasync becomes necessary. This book fills the gap in this context. In particular, it addresses key aspects and reviews the most relevant contributions within the mediasync research space, from different perspectives. Mediasync: Handbook on Multimedia Synchronization is the perfect companion for scholars and practitioners that want to acquire strong knowledge about this research area, and also approach the challenges behind ensuring the best mediated experiences, by providing the adequate synchronization between the media elements that constitute these experiences

Autonomous characters in virtual environments: The technologies involved in artificial life and their affects on perceived intelligence and playability of computer games

Computer games are viewed by academics as un֊grounded hack and patch experiments. "The industry lacks the formalism and requirement for a "perfect" solution often necessary in the academic world " [Woob]. Academic Artifical Intelligence (AI) is often viewed as un-implementable and narrow minded by the majority of ทon-AI programmer. "Historically, AI tended to be focused, containing detailed problems and domain-specific techniques. This focus makes for easier study - or engineering - of particular solutions. " [СһаОЗ .By implementing several well known AI techniques into the same gaming environment and judging users reactions this project aims to make links between the academic nature of AI, as well as investigate the nature of practical implementation in a gaming environment. An online Java implemented version of the 1970'ร classic Space Invaders has been developed and tested, with the Aliens being controlled by 6 different approaches to modelling AI functions. In total information from 334 individuals games was recorded. Different types of games AI can create highly varied gaming experience as highlighted by the range of values and high standard deviation values seen in the results. The link between complex behaviour, complex control systems and perceived intelligence was not supported. A positive correlation identified between how fun the users found the game and how intelligent they perceived the Aliens to be, would seem to be logical. As games get visually more and more impressive, the need for intelligent characters cannot be denied because it is one of the few way in which games can set themselves apart from the competition. Conclusions identified that computer games must remain focussed on their end- goal, that of producing a fun game. Whilst complex and clever AI can help to achieve it, the AI itself can never overshadow the end result

Affective Computing

This book provides an overview of state of the art research in Affective Computing. It presents new ideas, original results and practical experiences in this increasingly important research field. The book consists of 23 chapters categorized into four sections. Since one of the most important means of human communication is facial expression, the first section of this book (Chapters 1 to 7) presents a research on synthesis and recognition of facial expressions. Given that we not only use the face but also body movements to express ourselves, in the second section (Chapters 8 to 11) we present a research on perception and generation of emotional expressions by using full-body motions. The third section of the book (Chapters 12 to 16) presents computational models on emotion, as well as findings from neuroscience research. In the last section of the book (Chapters 17 to 22) we present applications related to affective computing

Proceedings of the 5th international conference on disability, virtual reality and associated technologies (ICDVRAT 2004)

The proceedings of the conferenc