Analysis and Simulation of Dynamic Vision in the City: A Computer-Aided Cinematic Approach

This paper proposes a computer-aided Dynamic Visual Research and Design Protocol for environmental designers to analyze humans' dynamic visual experiences in the city and to simulate dynamic vision in the design process. The Protocol recommends using action cameras to collect massive dynamic visual data from participants' first-person perspectives. It prescribes a computer-aided visual analysis approach to produce cinematic charts and storyboards, which further afford qualitative interpretations for aesthetic assessment and discussion. Employing real-time 3D simulation technologies, the Protocol enables the simulation of people's dynamic vision in designed urban environments to support evaluation in design. Detailed contents and merits of the Protocol were demonstrated by its application in the Urbanscape Studio, a community participatory design course based at Watertown, South Dakota

Storytelling in the Metaverse: From Desktop to Immersive Virtual Reality Storyboarding

Creatives from the animation and film industries have always been experimenting with innovative tools and methodologies to improve the creation of prototypes of their visual sequences before bringing them to life. In recent years, as realistic real-time rendering techniques have emerged, the increasing popularity of virtual reality (VR) can lead to new approaches and solutions, leveraging the immersive and interactive features provided by 3D immersive experiences. A 3D desktop application and a novel storyboarding pipeline, which can automatically generate a storyboard including camera details and a textual description of the actions performed in three-dimensional environments, have already been investigated in previous work. The aim was to exploit new technologies to improve existing 3D storytelling approaches, thus providing a software solution for expert and novice storyboarders. This research investigates 3D storyboarding in immersive virtual reality (IVR) to move toward a new storyboarding paradigm. IVR systems provide peculiarities such as body-controlled exploration of the 3D scene and a head-dependant camera view that can extend features of traditional storyboarding tools. The proposed system enables users to set up the virtual stage, adding elements to the scene and exploring the environment as they build it. After that, users can select the available characters or the camera, control them in first person, position them in the scene, and perform actions selecting from a list of options, each paired with a corresponding animation. Relying on the concept of state-machine, the system can automatically generate the list of available actions depending on the context. Finally, the descriptions for each storyboard panel are automatically generated based on the history of activities performed. The proposed application maintains all the functionalities of the desktop version and can be effectively used to create storyboards in immersive virtual environments

From Screenwriting to Space-Writing

In the past ten years, audiovisual creators have been working on the development of narrative experiences for extended reality (XR) technologies, especially virtual reality (VR). The evolution of this practice has led to the creation of a technical language and processes. The transfer of knowledge from cinematography and videography has been the basis for the creative practice of “immersive narratives,” very often carrying with it jargon and practices that do not fit entirely with XR’s spatial nature. In this essay, I ref lect on whether we are still writing for a screen or writing for space from a practitioner's perspective. Such a change of perspective starts with the recognition of the perceptual sphere and how to compose scenes in it. In this regard, a review of storyboarding for VR, followed by my own experience in creating an interactive VR movie, allowed me to ref lect on the concept of framing, camera positions, and authorial intentions. Finally, I argue that we can move from screen- writing to space-writing in relation to the technologies and immersive power of XR

As cinemáticas e narrativas de jogos digitais: implicações para o design de jogos

Game cinematics integrating filming techniques, cutscenes, and animations are likely to have a pivotal role in the player experience, emotional attachment, and relatedness to the game narrative content. In recent years, the avoidance of cuts to generate a seamless and connected game experience has challenged older patterns on storytelling in games and find new ways of intertwining gameplay with the narrative. However, a one-size-fits-all strategy may not be applied when considering different storytelling purposes, structures, and language used in diversified game genres. The purpose of this research is to examine the contributions of cinematics to affect game story comprehension, especially in young adults. A multi-stage development research method is applied, encompassing the following activities: (1) identification of requirements to develop game cinematics based on the literature review and interviews with 16 scholars and industry professionals in Game UX, and cinematic development; (2) Development of a Game Cinematics experiment for the game Mutation Madness, using first- and third- perspectives; and a (3) Comparative evaluation of game cinematics in terms of experience and narrative comprehension. The cinematics developed are tested with 46 young adults to assess the effect of the Point Of View (POV) camera on visual attention and story comprehension in Mutation Madness, using an eye- tracking experiment. The results suggest that third- perspective in game cinematics constitute omniscient knowledge of the story, evoking the sense of time and focusing on the story agents, while first- perspective visually guides the player to the game events happening in the time of gameplay. This research contributes to the Communication Sciences and Technologies field by presenting a set of best practices developing game cinematics.As cinemáticas de jogos que integram técnicas de filmagem, cutscenes, e animações, tendem a ter um papel central na experiência do jogador, envolvimento emocional, e relação com a narrativa. Nos últimos anos, os padrões mais antigos inerentes ao processo de contar histórias nos jogos têm sido desafiados pela ausência de cortes e interrupções na experiência, de modo a garantir a interconexão e o entrelaçar da jogabilidade com a narrativa. No entanto, a adoção de uma estratégia de dimensão única pode não atender a diferentes propósitos da narrativa, estruturas e linguagem utilizadas em diferentes géneros de jogos. O objetivo desta investigação é compreender o modo como as cinemáticas podem afetar a compreensão de histórias de jogos, especialmente em jovens adultos. O método de investigação de desenvolvimento é aplicado, subdividido nas seguintes etapas: (1) Identificação de requisitos para desenvolver cinemáticas de jogos com base na revisão da literatura e entrevista a 16 académicos e profissionais da indústria em experiência de jogo, e desenvolvimento de cinemáticas; (2) Desenvolvimento de cinemáticas do jogo Mutation Madness, com recurso à primeira e terceira perspetiva; e (3) Avaliação comparativa das cinemáticas de jogo em termos de experiência e compreensão da narrativa. As cinemáticas desenvolvidas são testadas por 46 jovens adultos, recorrendo à metodologia eye-tracking, para avaliar o efeito da perspetiva da câmara [Point Of View (POV)] na atenção visual e na compreensão da narrativa do jogo Mutation Madness. Os resultados sugerem que a cinemática em que é adotada a terceira perspetiva no jogo proposto contribui para um conhecimento omnisciente da história, evocando o sentido do tempo e concentrando-se nos agentes da história, enquanto a primeira perspetiva orienta visualmente o jogador para os eventos do jogo que acontecem no tempo da jogabilidade. Esta investigação contribui para a área das Ciências da Tecnologia da Comunicação ao apresentar um conjunto de melhores práticas para desenvolver as cinemáticas nos jogos.Mestrado em Comunicação Multimédi

Interactive Fiction in Cinematic Virtual Reality: Epistemology, Creation and Evaluation

This dissertation presents the Interactive Fiction in Cinematic Virtual Reality (IFcVR), an interactive digital narrative (IDN) that brings together the cinematic virtual reality (cVR) and the creation of virtual environments through 360\ub0 video within an interactive fiction (IF) structure. This work is structured in three components: an epistemological approach to this kind of narrative and media hybrid; the creation process of IFcVR, from development to postproduction; and user evaluation of IFcVR. In order to set the foundations for the creation of interactive VR fiction films, I dissect the IFcVR by investigating the aesthetics, narratological and interactive notions that converge and diverge in it, proposing a medium-conscious narratology for this kind of artefact. This analysis led to the production of an IFcVR functional prototype: \u201cZENA\u201d, the first interactive VR film shot in Genoa. ZENA\u2019s creation process is reported proposing some guidelines for interactive and immersive film-makers. In order to evaluate the effectiveness of the IFcVR as an entertaining narrative form and a vehicle for diverse types of messages, this study also proposes a methodology to measure User Experience (UX) on IFcVR. The full evaluation protocol gathers both qualitative and quantitative data through ad hoc instruments. The proposed protocol is illustrated through its pilot application on ZENA. Findings show interactors' positive acceptance of IFcVR as an entertaining experience

Techniques de mise en scène pour le jeu vidéo et l'animation

Eurographics State of the Art Report (STAR).International audienceOver the last forty years, researchers in computer graphics have proposed a large variety of theoretical models and computer implementations of a virtual film director, capable of creating movies from minimal input such as a screenplay or storyboard. The underlying film directing techniques are also in high demand to assist and automate the generation of movies in computer games and animation. The goal of this survey is to characterize the spectrum of applications that require film directing, to present a historical and up-to-date summary of research in algorithmic film directing, and to identify promising avenues and hot topics for future research.Depuis quarante ans, les chercheurs en informatique graphique ont proposé une grande variété de modèles théoriques et d'implémentations de réalisateurs virtuels, capables de créer des films automatiquement à partir de scénarios ou de storyboards. Les techniques de mise en scène sous-jacentes peuvent également être très utiles pour assister et automatiser la création de films dans le jeu vidéo et l'animation. Le but de cet état de l'art est de caractériser le spectre des applications qui peuvent bénéficier des techniques de mise en scène, de donner un compte rendu historique de la recherche en mise en scène algorithmique, et d'identifier les tendances et perspectives du domaine

Brain-controlled cinematic interactions

Interactive films have been around for almost a century, yet they have suffered repeatedly from critical, commercial and interactional failings. We propose that brain-computer interfaces can offer interactions with narratives and encourage cinematic engagement by minimising active control. We ask what are the problems inherent to interactive cinema? Can real-time interactions via a brain-computer interface (BCI) construct cinematic content? And how do groups of individuals experience brain-controlled cinema designed for individual, shared or distributed control? Our review of related work motivates the interactional choice of using Passive BCI with real-time cinematic construction to synchronise rhythms of the viewers blinking, Attention and Meditation to the rhythms of cinema. We use the Performance Led Research in-the-Wild methodology to probe public deployments of our films, and we describe user interactions in-the-Wild during screenings of multiple designs of two interactive films: three single user, three multi user, and a live score performance. Our descriptions of BCI mappings to cinematic techniques and production strategies to produce interactive content efficiently contributes to the understanding of practical interactive cinema production. In our results we define 1) different stages of control; discovery, conscious and unconscious, 2) awareness of the affective loop, 3) a shifting prominence of engagement between the narrative, the visual qualities and the agency of users’ interactions. We offer a dynamic view of control; people’s experiences are shifting from awareness of their self, the film, and their control. Our hyper-scanning multi-user study introduces the concept of effects moving across groups, working together to produce engaging experiences, and instances of group members disrupting other’s experience by deciding to unilaterally take control of the film. Our discussion contributes to our understanding of passive interactions with narrative systems. Our research contributions include our insights into seven designs of two brain-controlled films. We define two taxonomies, of control and group control, and produce insights into value to audiences of brain-controlled films. We show the development of affective loops of physiological response and cinematic content, and provide new design directions and practical implications for interactive filmmakers

Gesture, Rhetoric, and Digital Storytelling

This proposal seeks to develop new theory and technology for digital interactive narratives based on cross-cultural storytelling traditions that utilize non-verbal communication to convey meaning. Digital interactive narratives are of increasing cultural relevance including: new genres of electronic literature, virtual museums and memorials, educational computer games, and computer-assisted generation of scholarly documents. We propose a new form of storytelling that changes emotional tone, theme, perspective, other subjective elements based upon gestural input via touch-screen and motion-sensitive devices. This proposal builds upon the Principal Investigator's significant body of work in developing humanistically grounded interactive narrative and artificial intelligence. The Principal Investigator has found that literary, cultural, and performance studies theories of orature (oral literature) and accounts of non-verbal communication from animation studies provide models that can be adapted to interactive narratives

Brain-controlled cinematic interactions

Interactive films have been around for almost a century, yet they have suffered repeatedly from critical, commercial and interactional failings. We propose that brain-computer interfaces can offer interactions with narratives and encourage cinematic engagement by minimising active control. We ask what are the problems inherent to interactive cinema? Can real-time interactions via a brain-computer interface (BCI) construct cinematic content? And how do groups of individuals experience brain-controlled cinema designed for individual, shared or distributed control? Our review of related work motivates the interactional choice of using Passive BCI with real-time cinematic construction to synchronise rhythms of the viewers blinking, Attention and Meditation to the rhythms of cinema. We use the Performance Led Research in-the-Wild methodology to probe public deployments of our films, and we describe user interactions in-the-Wild during screenings of multiple designs of two interactive films: three single user, three multi user, and a live score performance. Our descriptions of BCI mappings to cinematic techniques and production strategies to produce interactive content efficiently contributes to the understanding of practical interactive cinema production. In our results we define 1) different stages of control; discovery, conscious and unconscious, 2) awareness of the affective loop, 3) a shifting prominence of engagement between the narrative, the visual qualities and the agency of users’ interactions. We offer a dynamic view of control; people’s experiences are shifting from awareness of their self, the film, and their control. Our hyper-scanning multi-user study introduces the concept of effects moving across groups, working together to produce engaging experiences, and instances of group members disrupting other’s experience by deciding to unilaterally take control of the film. Our discussion contributes to our understanding of passive interactions with narrative systems. Our research contributions include our insights into seven designs of two brain-controlled films. We define two taxonomies, of control and group control, and produce insights into value to audiences of brain-controlled films. We show the development of affective loops of physiological response and cinematic content, and provide new design directions and practical implications for interactive filmmakers