Towards a Smart Drone Cinematographer for Filming Human Motion

Affordable consumer drones have made capturing aerial footage more convenient and accessible. However, shooting cinematic motion videos using a drone is challenging because it requires users to analyze dynamic scenarios while operating the controller. In this thesis, our task is to develop an autonomous drone cinematography system to capture cinematic videos of human motion. We understand the system's filming performance to be influenced by three key components: 1) video quality metric, which measures the aesthetic quality -- the angle, the distance, the image composition -- of the captured video, 2) visual feature, which encapsulates the visual elements that influence the filming style, and 3) camera planning, which is a decision-making model that predicts the next best movement. By analyzing these three components, we designed two autonomous drone cinematography systems using both heuristic-based methods and learning-based methods.For the first system, we designed an Autonomous CinemaTography system -- "ACT" by proposing a viewpoint quality metric focusing on the visibility of the 3D human skeleton of the subject. We expanded the application of human motion analysis and simplified manual control by assisting viewpoint selection using a through-the-lens method. For the second system, we designed an imitation-based system that learns the artistic intention of the cameramen through watching professional aerial videos. We designed a camera planner that analyzes the video contents and previous camera motion to predict future camera motion. Furthermore, we propose a planning framework, which can imitate a filming style by ``seeing" only one single demonstration video of such style. We named it ``one-shot imitation filming." To the best of our knowledge, this is the first work that extends imitation learning to autonomous filming. Experimental results in both simulation and field test exhibit significant improvements over existing techniques and our approach managed to help inexperienced pilots capture cinematic videos

Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Production

Amateurs working on mini-films and short-form videos usually spend lots of time and effort on the multi-round complicated process of setting and adjusting scenes, plots, and cameras to deliver satisfying video shots. We present Virtual Dynamic Storyboard (VDS) to allow users storyboarding shots in virtual environments, where the filming staff can easily test the settings of shots before the actual filming. VDS runs on a "propose-simulate-discriminate" mode: Given a formatted story script and a camera script as input, it generates several character animation and camera movement proposals following predefined story and cinematic rules to allow an off-the-shelf simulation engine to render videos. To pick up the top-quality dynamic storyboard from the candidates, we equip it with a shot ranking discriminator based on shot quality criteria learned from professional manual-created data. VDS is comprehensively validated via extensive experiments and user studies, demonstrating its efficiency, effectiveness, and great potential in assisting amateur video production.Comment: Project page: https://virtualfilmstudio.github.io

CineTransfer: Controlling a Robot to Imitate Cinematographic Style from a Single Example

This work presents CineTransfer, an algorithmic framework that drives a robot to record a video sequence that mimics the cinematographic style of an input video. We propose features that abstract the aesthetic style of the input video, so the robot can transfer this style to a scene with visual details that are significantly different from the input video. The framework builds upon CineMPC, a tool that allows users to control cinematographic features, like subjects' position on the image and the depth of field, by manipulating the intrinsics and extrinsics of a cinematographic camera. However, CineMPC requires a human expert to specify the desired style of the shot (composition, camera motion, zoom, focus, etc). CineTransfer bridges this gap, aiming a fully autonomous cinematographic platform. The user chooses a single input video as a style guide. CineTransfer extracts and optimizes two important style features, the composition of the subject in the image and the scene depth of field, and provides instructions for CineMPC to control the robot to record an output sequence that matches these features as closely as possible. In contrast with other style transfer methods, our approach is a lightweight and portable framework which does not require deep network training or extensive datasets. Experiments with real and simulated videos demonstrate the system's ability to analyze and transfer style between recordings, and are available in the supplementary video

Cortical fMRI activation to opponents' body kinematics in sport-related anticipation: Expert-novice differences with normal and point-light video

This is the post-print version of the final paper published in Neuroscience Letters. The published article is available from the link below. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. Copyright @ 2011 Elsevier B.V.Badminton players of varying skill levels viewed normal and point-light video clips of opponents striking the shuttle towards the viewer; their task was to predict in which quadrant of the court the shuttle would land. In a whole-brain fMRI analysis we identified bilateral cortical networks sensitive to the anticipation task relative to control stimuli. This network is more extensive and localised than previously reported. Voxel clusters responding more strongly in experts than novices were associated with all task-sensitive areas, whereas voxels responding more strongly in novices were found outside these areas. Task-sensitive areas for normal and point-light video were very similar, whereas early visual areas responded differentially, indicating the primacy of kinematic information for sport-related anticipation.Research Grants Council of the Hong Kong Special Administrative Region, Chin

Graphic Design and the Cinema: An Application of Graphic Design to the Art of Filmmaking

When the public considers different art forms such as painting, drawing and sculpture, it is easy to understand the common elements that unite them. Each is a non-moving art form that begins at the drawing board. Using line, color and shape to evoke a particular response from audiences is what ties these fine arts together. Graphic design, however, tends to separate itself from the fine arts. Because of its later development in the art world, as well as, its operation within modern technological developments, graphic design is driven by the idea of communicating to large audiences. In this way, although design finds its base in fine art practices, it is more easily related to filmmaking in many aspects. Both graphic design and filmmaking are particularly unique in respect to their reliance on technological innovation, communication and public consumption. Even though the mediums operate in different manners, it is important to note that they share basic design principles such as color, composition and image systems. These elements of design help build a solid foundation for a successful piece of design or film. The research discussed in this paper critically analyzes how design principles are incorporated into the 2015 Best Picture Oscar Nominees. By examining the aforementioned design elements, color, composition and image systems, I was able to break down the visual meaning of each film according to the use of these elements. All of the nominees were successful in their own aspect, however the more experimental the design of the film, the more effective the communication of their respective narratives

Dying in Full Detail

In 'Dying in Full Detail' Jennifer Malkowski explores digital media's impact on one of documentary film's greatest taboos: the recording of death. Despite technological advances that allow for the easy creation and distribution of death footage, digital media often fail to live up to their promise to reveal the world in greater fidelity. Malkowski analyzes a wide range of death footage, from feature films about the terminally ill (Dying, Silverlake Life, Sick), to surreptitiously recorded suicides (The Bridge), to #BlackLivesMatter YouTube videos and their precursors. Contextualizing these recordings in the long history of attempts to capture the moment of death in American culture, Malkowski shows how digital media are unable to deliver death "in full detail," as its metaphysical truth remains beyond representation

Crushing Animals and Crashing Funerals: The Semiotics of Free Expression

With insights from philosophy of language and semiotics, this article addresses judicial choices and semantic errors involved in United States v. Stevens, 130 S.Ct. 1577 (2010) (refusing to read “killing” and “wounding” to include cruelty and thus striking down a federal statute outlawing videos of animal cruelty), and Snyder v. Phelps, 131 S.Ct. 1207 (2011) (finding a First Amendment right to picket military funerals and verbally attack parents of dead soldiers as part of purportedly-public expression). This article maintains that a better understanding of semiotics (the theory of signs) exposes the flaws in both decisions and bolsters the arguments of the lone dissenter in both cases, Justice Alito. Such a better understanding of semiotics involves grasping (a) how expression involves signs, (b) how signs work in general, and (c) the differences between three basic kinds of signs (indexes, icons and symbols). This article maintains that the expression involved in Stevens and in Phelps was a type of indexical or quasi-indexical expression that, for reasons similar to those involved in child pornography cases, should have no First Amendment protection. This article also notes shifting interpretive positions in the Court that cry out for reform. Although Chief Justice Roberts uses a textualist approach in his majority opinion striking down the animal cruelty statute in Stevens, his majority opinion in National Federation of Independent Business v. Sebelius, 132 S.Ct. 2566 (2012), has no trouble finding a “penalty” a “tax,” upholding the Affordable Care Act, and chastising the dissent for voting to strike down a statute simply because “. . . Congress used the wrong labels.” Id. at 2597. This article attempts to expose such gamesmanship in “textualism” and attempts to lay out a better semiotic path for the Court. It calls for more forthright judicial decision-making in constitutional and statutory interpretation; calls for rejecting mechanical notions of law that conceal judicial choice involved in constitutional and statutory interpretation; and calls for rejecting claims that dictionaries can settle constitutional or statutory interpretation issues without reference to constitutional and statutory goals. Keywords: interpretation, construction, meaning, plain meaning, originalism, original intent, canon, semiotics, signs, signals, symbol, icon, index, signifier, signified, ordinary meaning, freedom of speech, freedom of expression, animal cruelty, first amendment, statute

Good enough sculptures : what happens when sculptures are made to be filmed?

This PhD proposes the camera as a tool in the creation of sculpture. Exploring the ways in which the sculptural process is transformed by its relationship to the moment of filming, it aligns itself with artistic practices and theories which foreground material exploration, uncertainty and improvisation, and draws on a number of key artists who have used film and video to extend and explore sculptural practice. It situates fine art practice as a vehicle for exploratory and open-ended research, forging strong links with contemporary art educational theory which sees the creative process as heuristic and immersed within a social context. Using Winnicott’s theory of transitional objects and conception of psychoanalytic practice as a specialised form of play, the PhD forges strong connections between the engaged, responsive and explorative work done by the artist, analyst, teacher and student. The research presents a form of artistic research which facilitates encounters between objects and cameras, through which learning can take place and knowledge can be created - knowledge, which is not discrete or abstracted, but contextualised and embodied. The aim is to involve people in its processes and methods, as opposed to presenting finished works and findings, inscribing the reception of the work into the making process thereby producing active viewers and participants who are thoughtfully and practically involved within the making process. The artistic research method revolves around a collection of objects made to prompt physical, material and imaginative exploration in front of the camera. The camera’s field of vision is re-considered as an arena or situation structured in order to facilitate exploratory activity. ‘Filming sculpture’ becomes the situation/set-up which organises the production of objects-as-sculpture in ways that open up questions around sculpture as a particular category of object, the nature of film experience, and objects more generally. The PhD submission comprises a series of films and gifs, documentation of exhibitions, screenings and discussions undertaken during the research, experimental workshops, and photographs of each of the sculptures. The main written element consists of a series of aphoristic texts and a contextual document, which both draw on ideas and concepts from art and film theory, psychoanalysis, phenomenology, object-oriented ontology and anthropology, outlining the development of the research and situating it within a wider network of practices