Generating story variants with constrained video recombination

We present a novel approach to the automatic generation of filmic variants within an implemented Video-Based Storytelling (VBS) system that successfully integrates video segmentation with stochastically controlled re-ordering techniques and narrative generation via AI planning. We have introduced flexibility into the video recombination process by sequencing video shots in a way that maintains local video consistency and this is combined with exploitation of shot polysemy to enable shot reuse in a range of valid semantic contexts. Results of evaluations on output narratives using a shared set of video data show consistency in terms of local video sequences and global causality with no loss of generative power

Markov Chains Fusion for Video Scene Generation

In this paper we address the general issue of merging Markov chains used to model two instances of a given process with some properties in common. In particular, in this work we apply this scenario to a multimedia application that generates new video scenes mixing the original segments of a given movie. To perform the latter process, it is first necessary to describe the structure of the scenes in some way, which in our case is done through Markov chains. The video scenes are then recombined by fusing their corresponding models using the general method described here. We analyze and validate the proposed methodology only for this specific application, however the solution presented here could be used in a very diverse array of applications where Markov chains are routinely used, ranging from queuing modeling to financial decision processes

Interactive Film Recombination

In this paper we discuss an innovative media entertainment application called Interactive Movietelling. As an offspring of Interactive Storytelling applied to movies, we propose to integrate narrative generation through AI planning with video processing and modeling to construct filmic variants starting from the baseline content. The integration is possible thanks to content description using semantic attributes pertaining to intermediate-level concepts shared between video processing and planning levels. The output is a recombination of segments taken from the input movie performed so as to convey an alternative plot. User tests on the prototype proved how promising Interactive Movietelling might be, even if it was designed at a proof of concept level. Possible improvements that are suggested here lead to many challenging research issues

Investigating collaborative creativity via machine-mediated game blending

Can the creativity of humans be enhanced through mutual cooperation, or is it a detriment to their own individual creativity? Although most artists are known for their artistic individuality, some of the best creative works were achieved through mutual collaborative efforts. This paper proposes the study of a game blending system capable of combining user- And machine-generated content from multiple users and creativity facets (e.g., audio, visuals, narrative) for the creation of complete games. Supported by mixed-initiative design tools and human computation (crowdsourcing), users create facet- specific content, while getting stimulated by other creations on different facets by other users. Our research will investigate the ability for machine input into the collaborative process to yield games of higher novelty and quality for players.peer-reviewe

Improved Face Tracking Thanks to Local Features Correspondence

In this paper, we propose a technique to enhance the quality of detected face tracks in videos. In particular, we present a tracking algorithm that can improve the temporal localization of the tracks, remedying to the unavoidable failures of the face detection algorithms. Local features are extracted and tracked to “fill the gaps” left by missed detections. The principal aim of this work is to provide robust and well localized tracks of faces to a system of Interactive Movietelling, but the concepts can be extended whenever there is the necessity to localize the presence of a determined face even in environments where the face detection is, for any reason, difficult. We test the effectiveness of the proposed algorithm in terms of faces localization both in space and time, first assessing the performance in an ad-hoc simulation scenario and then showing output examples of some real-world video sequences

Expressivity of parameterized and data-driven representations in quality diversity search

Algorithms and the Foundations of Software technolog

Content-Style Decomposition: Representation Discovery and Applications

Content-style decompositions, or CSDs, decompose entities into content, defined by the entity's class, and style, defined as the remaining within-class variation. Content is typically defined in terms of some task. For example, in a face recognition task, identity is the content; in an emotion recognition task, expression is the content. CSDs have many applications: they can provide insight into domains where we have little prior knowledge of the sources of within- and between-class variation, and content-style recombinations are interesting as a creative exercise or for data set augmentation. Our approach is to decompose CSD discovery into two sub-problems: (1) to find useful representations of content that capture the class structure of the domain, and (2) to use those content-representations to discover CSDs. We make contributions to both sub-problems. First, we propose the F-statistic loss, a new method for discovering content representations that uses statistics of class separation on isolated embedding dimensions within a minibatch to determine when to terminate training. In addition to state-of-the-art performance on few-shot learning, we find that the method leads to factorial (also known as disentangled) representations of content when applied with a novel form of weak supervision. Previous work on disentangling is either unsupervised or uses a factor-aware oracle, which provides similar/dissimilar judgments with respect to a named attribute/factor. We explore an intermediate form of supervision, an unnamed-factor oracle, which provides similarity judgments with respect to a random unnamed factor. We demonstrate that the F-statistic loss leads to better disentangling when compared with other deep-embeddings losses and β-VAE, a state-of-the-art unsupervised disentangling method. Second, we introduce a new loss for discovering CSDs that can arbitrarily recombine content and style, called leakage filtering. In contrast to previous research which attempts to separate content and style in two different representation vectors, leakage filtering allows for imperfectly disentangled representations but ensures that residual content information will not leak out of the style representation and vice versa. Leakage filtering is also distinguished by its ability to operate on novel content-classes and by its lack of dependency on style labels for training. The recombined images produced are of high quality and can be used to augment datasets for few-shot learning tasks, yielding significant generalization improvements

Automated iterative game design

Computational systems to model aspects of iterative game design were proposed, encompassing: game generation, sampling behaviors in a game, analyzing game behaviors for patterns, and iteratively altering a game design. Explicit models of the actions in games as planning operators allowed an intelligent system to reason about how actions and action sequences affect gameplay and to create new mechanics. Metrics to analyze differences in player strategies were presented and were able to identify flaws in game designs. An intelligent system learned design knowledge about gameplay and was able to reduce the number of design iterations needed during playtesting a game to achieve a design goal. Implications for how intelligent systems augment and automate human game design practices are discussed.Ph.D

A Creative Exploration of the Use of Intelligent Agents in Spatial Narrative Structures

This thesis is an interdisciplinary study of authoring tools for creating spatial narrative structures– exposing the relationship between artists, the tools they use, and the experiences they create. It is a research-creation enterprise resulting in the creation of a new authoring tool. A prototype collaborative tool for authoring spatial narratives used at the Land|Slide: Possible Futures public art exhibit in Markham, Ontario 2013 is described. Using narrative analysis of biographical information a cultural context for authoring and experiencing spatial narrative structures is discussed. The biographical information of artists using digital technologies is posited as a context framing for usability design heuristics. The intersection of intelligent agents and spatial narrative structures provide a future scenario by which to assess the suitability of the approach outlined in this study