Text-based Editing of Talking-head Video

Editing talking-head video to change the speech content or to remove filler words is challenging. We propose a novel method to edit talking-head video based on its transcript to produce a realistic output video in which the dialogue of the speaker has been modified, while maintaining a seamless audio-visual flow (i.e. no jump cuts). Our method automatically annotates an input talking-head video with phonemes, visemes, 3D face pose and geometry, reflectance, expression and scene illumination per frame. To edit a video, the user has to only edit the transcript, and an optimization strategy then chooses segments of the input corpus as base material. The annotated parameters corresponding to the selected segments are seamlessly stitched together and used to produce an intermediate video representation in which the lower half of the face is rendered with a parametric face model. Finally, a recurrent video generation network transforms this representation to a photorealistic video that matches the edited transcript. We demonstrate a large variety of edits, such as the addition, removal, and alteration of words, as well as convincing language translation and full sentence synthesis

She inches glass to break: conversations between friends

She inches glass to break: conversations between friends is a project that aims to manifest, through research and practice, my own feminist language within the videos I have produced in my final year of my Masters of Fine Arts. My feminist language is Australian and intersectional, invested in combating sexism, racism and in deepening language and representation around sexuality in relation to Asian women. This project discusses my video She inches glass to break (2018) in length, which created intersectional feminist dialogue in response to feminist filmmaker Ulrike Ottinger’s film Ticket of No Return (1979) and Breakfast at Tiffany’s (1961). Additionally, given this project’s investment in language, this body of work is influenced both by aspects of psychoanalysis – in which speech is central to a “therapeutic action” – and by feminist linguistics in which linguistic analysis reveals some of the mechanisms through which language constrains, coerces and represents women, men and non-binary people in oppressive ways

The Specifics of Editing Comedy

This research paper examines the essential knowledge required for editing comedy well. Editing comedy differs from other genres, and can be extremely difficult and time consuming. The paper explores what defines comedy as a genre, editing techniques unique to comedy, the process of editing, and film theories that enable an editor to make better cuts to heighten the hilarity of the story. Sources include academic writings, personal projects completed under guidance from a mentor, movie and scene examples, and three interviews with comedians and in-demand comedy editors

Audeosynth: music-driven video montage

We introduce music-driven video montage, a media format that offers a pleasant way to browse or summarize video clips collected from various occasions, including gatherings and adventures. In music-driven video montage, the music drives the composition of the video content. According to musical movement and beats, video clips are organized to form a montage that visually reflects the experiential properties of the music. Nonetheless, it takes enormous manual work and artistic expertise to create it. In this paper, we develop a framework for automatically generating music-driven video montages. The input is a set of video clips and a piece of background music. By analyzing the music and video content, our system extracts carefully designed temporal features from the input, and casts the synthesis problem as an optimization and solves the parameters through Markov Chain Monte Carlo sampling. The output is a video montage whose visual activities are cut and synchronized with the rhythm of the music, rendering a symphony of audio-visual resonance.postprin

Detecting and removing visual distractors for video aesthetic enhancement

Personal videos often contain visual distractors, which are objects that are accidentally captured that can distract viewers from focusing on the main subjects. We propose a method to automatically detect and localize these distractors through learning from a manually labeled dataset. To achieve spatially and temporally coherent detection, we propose extracting features at the Temporal-Superpixel (TSP) level using a traditional SVM-based learning framework. We also experiment with end-to-end learning using Convolutional Neural Networks (CNNs), which achieves slightly higher performance than other methods. The classification result is further refined in a post-processing step based on graph-cut optimization. Experimental results show that our method achieves an accuracy of 81% and a recall of 86%. We demonstrate several ways of removing the detected distractors to improve the video quality, including video hole filling; video frame replacement; and camera path re-planning. The user study results show that our method can significantly improve the aesthetic quality of videos

Education vs. Entertainment: A Cultural History of Children's Software

Part of the Volume on the Ecology of Games: Connecting Youth, Games, and Learning This chapter draws on ethnographic material to consider the cultural politics and recent history of children's software and reflects on how this past can inform our current efforts to mobilize games for learning. The analysis uses a concept of genre as a way of making linkages across the distributed but interconnected circuit of everyday play, software content, and industry context. Organized through three genres in children's software -- academic, entertainment, and construction -- the body of the chapter describes how these genres play out within a production and advertising context, in the design of particular software titles, and at sites of play in after-school computer centers where the fieldwork was conducted

Movie Editing and Cognitive Event Segmentation in Virtual Reality Video

Traditional cinematography has relied for over a century on a well-established set of editing rules, called continuity editing, to create a sense of situational continuity. Despite massive changes in visual content across cuts, viewers in general experience no trouble perceiving the discontinuous flow of information as a coherent set of events. However, Virtual Reality (VR) movies are intrinsically different from traditional movies in that the viewer controls the camera orientation at all times. As a consequence, common editing techniques that rely on camera orientations, zooms, etc., cannot be used. In this paper we investigate key relevant questions to understand how well traditional movie editing carries over to VR. To do so, we rely on recent cognition studies and the event segmentation theory, which states that our brains segment continuous actions into a series of discrete, meaningful events. We first replicate one of these studies to assess whether the predictions of such theory can be applied to VR. We next gather gaze data from viewers watching VR videos containing different edits with varying parameters, and provide the first systematic analysis of viewers' behavior and the perception of continuity in VR. From this analysis we make a series of relevant findings; for instance, our data suggests that predictions from the cognitive event segmentation theory are useful guides for VR editing; that different types of edits are equally well understood in terms of continuity; and that spatial misalignments between regions of interest at the edit boundaries favor a more exploratory behavior even after viewers have fixated on a new region of interest. In addition, we propose a number of metrics to describe viewers' attentional behavior in VR. We believe the insights derived from our work can be useful as guidelines for VR content creation