Towards a Smart Drone Cinematographer for Filming Human Motion

Affordable consumer drones have made capturing aerial footage more convenient and accessible. However, shooting cinematic motion videos using a drone is challenging because it requires users to analyze dynamic scenarios while operating the controller. In this thesis, our task is to develop an autonomous drone cinematography system to capture cinematic videos of human motion. We understand the system's filming performance to be influenced by three key components: 1) video quality metric, which measures the aesthetic quality -- the angle, the distance, the image composition -- of the captured video, 2) visual feature, which encapsulates the visual elements that influence the filming style, and 3) camera planning, which is a decision-making model that predicts the next best movement. By analyzing these three components, we designed two autonomous drone cinematography systems using both heuristic-based methods and learning-based methods.For the first system, we designed an Autonomous CinemaTography system -- "ACT" by proposing a viewpoint quality metric focusing on the visibility of the 3D human skeleton of the subject. We expanded the application of human motion analysis and simplified manual control by assisting viewpoint selection using a through-the-lens method. For the second system, we designed an imitation-based system that learns the artistic intention of the cameramen through watching professional aerial videos. We designed a camera planner that analyzes the video contents and previous camera motion to predict future camera motion. Furthermore, we propose a planning framework, which can imitate a filming style by ``seeing" only one single demonstration video of such style. We named it ``one-shot imitation filming." To the best of our knowledge, this is the first work that extends imitation learning to autonomous filming. Experimental results in both simulation and field test exhibit significant improvements over existing techniques and our approach managed to help inexperienced pilots capture cinematic videos

Gesture Object Interfaces to enable a world of multiple projections

Thesis (Ph. D.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2010.Cataloged from PDF version of thesis.Includes bibliographical references (p. [209]-226).Tangible Media as an area has not explored how the tangible handle is more than a marker or place-holder for digital data. Tangible Media can do more. It has the power to materialize and redefine our conception of space and content during the creative process. It can vary from an abstract token that represents a movie to an anthropomorphic plush that reflects the behavior of a sibling during play. My work begins by extending tangible concepts of representation and token-based interactions into movie editing and play scenarios. Through several design iterations and research studies, I establish tangible technologies to drive visual and oral perspectives along with finalized creative works, all during a child's play and exploration. I define the framework, Gesture Object Interfaces, expanding on the fields of Tangible User Interaction and Gesture Recognition. Gesture is a mechanism that can reinforce or create the anthropomorphism of an object. It can give the object life. A Gesture Object is an object in hand while doing anthropomorphized gestures. Gesture Object Interfaces engender new visual and narrative perspectives as part of automatic film assembly during children's play. I generated a suite of automatic film assembly tools accessible to diverse users. The tools that I designed allow for capture, editing and performing to be completely indistinguishable from one another. Gestures integrated with objects become a coherent interface on top of natural play. I built a distributed, modular camera environment and gesture interaction to control that environment. The goal of these new technologies is to motivate children to take new visual and narrative perspectives. In this dissertation I present four tangible platforms that I created as alternatives to the usual fragmented and sequential capturing, editing and performing of narratives available to users of current storytelling tools. I developed Play it by Eye, Frame it by hand, a new generation of narrative tools that shift the frame of reference from the eye to the hand, from the viewpoint (where the eye is) to the standpoint (where the hand is). In Play it by Eye, Frame it by Hand environments, children discover atypical perspectives through the lens of everyday objects. When using Picture This!, children imagine how an object would appear relative to the viewpoint of the toy. They iterate between trying and correcting in a world of multiple perspectives. The results are entirely new genres of child-created films, where children finally capture the cherished visual idioms of action and drama. I report my design process over the course of four tangible research projects that I evaluate during qualitative observations with over one hundred 4- to 14-year-old users. Based on these research findings, I propose a class of moviemaking tools that transform the way users interpret the world visually, and through storytelling.by Catherine Nicole Vaucelle.Ph.D

Multi-Robot Systems: Challenges, Trends and Applications

This book is a printed edition of the Special Issue entitled “Multi-Robot Systems: Challenges, Trends, and Applications” that was published in Applied Sciences. This Special Issue collected seventeen high-quality papers that discuss the main challenges of multi-robot systems, present the trends to address these issues, and report various relevant applications. Some of the topics addressed by these papers are robot swarms, mission planning, robot teaming, machine learning, immersive technologies, search and rescue, and social robotics

Swarm intelligence: novel tools for optimization, feature extraction, and multi-agent system modeling

Abstract Animal swarms in nature are able to adapt to dynamic changes in their envi-ronment, and through cooperation they can solve problems that are crucial for their survival. Only by means of local interactions with other members of the swarm and with the environment, they can achieve a common goal more efficiently than it would be done by a single individual. This problem-solving behavior that results from the multiplicity of such interactions is referred to as Swarm Intelligence. The mathematical models of swarming behavior in nature were initially proposed to solve optimization problems. Nevertheless, this decentralized approach can be a valuable tool for a variety of applications, where emerging global patterns represent a solution to the task at hand. Methods for the solution of difficult computational problems based on Swarm Intelligence have been experimentally demonstrated and reported in the literature. However, a general framework that would facilitate their design does not exist yet. In this dissertation, a new general design methodology for Swarm Intelligence tools is proposed. By defining a discrete space in which the members of the swarm can move, and by modifying the rules of local interactions and setting the adequate objective function for solutions evaluation, the proposed methodology is tested in various domains. The dissertation presents a set of case studies, and focuses on two general approaches. One approach is to apply Swarm Intelligence as a tool for optimization and feature extraction, and the other approach is to model multi-agent systems such that they resemble swarms of animals in nature providing them with the ability to autonomously perform a task at hand. Artificial swarms are designed to be autonomous, scalable, robust, and adaptive to the changes in their environment. In this work, the methods that exploit one or more of these features are presented. First, the proposed methodology is validated in a real-world scenario seen as a combinatorial optimization problem. Then a set of novel tools for feature extraction, more precisely the adaptive edge detection and the broken-edge linking in digital images is proposed. A novel data clustering algorithm is also proposed and applied to image segmentation. Finally, a scalable algorithm based on the proposed methodology is developed for distributed task allocation in multi-agent systems, and applied to a swarm of robots. The newly proposed general methodology provides a guideline for future developers of the Swarm Intelligence tools. Los enjambres de animales en la naturaleza son capaces de adaptarse a cambios dinamicos en su entorno y, por medio de la cooperación, pueden resolver problemas ´ cruciales para su supervivencia. Unicamente por medio de interacciones locales con otros miembros del enjambre y con el entorno, pueden lograr un objetivo común de forma más eficiente que lo haría un solo individuo. Este comportamiento problema-resolutivo que es resultado de la multiplicidad de interacciones se denomina Inteligencia de Enjambre. Los modelos matemáticos de comportamiento de enjambres en entornos naturales fueron propuestos inicialmente para resolver problemas de optimización. Sin embargo, esta aproximación descentralizada puede ser una herramienta valiosa en una variedad de aplicaciones donde patrones globales emergentes representan una solución de las tareas actuales. Aunque en la literatura se muestra la utilidad de los métodos de Inteligencia de Enjambre, no existe un entorno de trabajo que facilite su diseño. En esta memoria de tesis proponemos una nueva metodologia general de diseño para herramientas de Inteligencia de Enjambre. Desarrollamos herramientas noveles que representan ejem-plos ilustrativos de su implementación. Probamos la metodología propuesta en varios dominios definiendo un espacio discreto en el que los miembros del enjambre pueden moverse, modificando las reglas de las interacciones locales y fijando la función objetivo adecuada para evaluar las soluciones. La memoria de tesis presenta un conjunto de casos de estudio y se centra en dos aproximaciones generales. Una aproximación es aplicar Inteligencia de Enjambre como herramienta de optimización y extracción de características mientras que la otra es modelar sistemas multi-agente de tal manera que se asemejen a enjambres de animales en la naturaleza a los que se les confiere la habilidad de ejecutar autónomamente la tarea. Los enjambres artificiales están diseñados para ser autónomos, escalables, robustos y adaptables a los cambios en su entorno. En este trabajo, presentamos métodos que explotan una o más de estas características. Primero, validamos la metodología propuesta en un escenario del mundo real visto como un problema de optimización combinatoria. Después, proponemos un conjunto de herramientas noveles para ex-tracción de características, en concreto la detección adaptativa de bordes y el enlazado de bordes rotos en imágenes digitales, y el agrupamiento de datos para segmentación de imágenes. Finalmente, proponemos un algoritmo escalable para la asignación distribuida de tareas en sistemas multi-agente aplicada a enjambres de robots. La metodología general recién propuesta ofrece una guía para futuros desarrolladores deherramientas de Inteligencia de Enjambre

Real-time Ultrasound Signals Processing: Denoising and Super-resolution

Ultrasound acquisition is widespread in the biomedical field, due to its properties of low cost, portability, and non-invasiveness for the patient. The processing and analysis of US signals, such as images, 2D videos, and volumetric images, allows the physician to monitor the evolution of the patient's disease, and support diagnosis, and treatments (e.g., surgery). US images are affected by speckle noise, generated by the overlap of US waves. Furthermore, low-resolution images are acquired when a high acquisition frequency is applied to accurately characterise the behaviour of anatomical features that quickly change over time. Denoising and super-resolution of US signals are relevant to improve the visual evaluation of the physician and the performance and accuracy of processing methods, such as segmentation and classification. The main requirements for the processing and analysis of US signals are real-time execution, preservation of anatomical features, and reduction of artefacts. In this context, we present a novel framework for the real-time denoising of US 2D images based on deep learning and high-performance computing, which reduces noise while preserving anatomical features in real-time execution. We extend our framework to the denoise of arbitrary US signals, such as 2D videos and 3D images, and we apply denoising algorithms that account for spatio-temporal signal properties into an image-to-image deep learning model. As a building block of this framework, we propose a novel denoising method belonging to the class of low-rank approximations, which learns and predicts the optimal thresholds of the Singular Value Decomposition. While previous denoise work compromises the computational cost and effectiveness of the method, the proposed framework achieves the results of the best denoising algorithms in terms of noise removal, anatomical feature preservation, and geometric and texture properties conservation, in a real-time execution that respects industrial constraints. The framework reduces the artefacts (e.g., blurring) and preserves the spatio-temporal consistency among frames/slices; also, it is general to the denoising algorithm, anatomical district, and noise intensity. Then, we introduce a novel framework for the real-time reconstruction of the non-acquired scan lines through an interpolating method; a deep learning model improves the results of the interpolation to match the target image (i.e., the high-resolution image). We improve the accuracy of the prediction of the reconstructed lines through the design of the network architecture and the loss function. %The design of the deep learning architecture and the loss function allow the network to improve the accuracy of the prediction of the reconstructed lines. In the context of signal approximation, we introduce our kernel-based sampling method for the reconstruction of 2D and 3D signals defined on regular and irregular grids, with an application to US 2D and 3D images. Our method improves previous work in terms of sampling quality, approximation accuracy, and geometry reconstruction with a slightly higher computational cost. For both denoising and super-resolution, we evaluate the compliance with the real-time requirement of US applications in the medical domain and provide a quantitative evaluation of denoising and super-resolution methods on US and synthetic images. Finally, we discuss the role of denoising and super-resolution as pre-processing steps for segmentation and predictive analysis of breast pathologies

Embeddedness as Condition and Strategy in Contemporary Art and Cultural Production

This thesis examines the concept of ‘embeddedness’ as condition and strategy in contemporary art and cultural production. Identifying embeddedness as a motif of contextual proximity and a strategy in contemporary art, the thesis proposes immediacy to be the result of intrinsic mediation. The project’s main concern is how embeddedness is contextualised by the current conditions that authors and cultural producers engage with. The primary question is whether and how embeddedness can convey a critical relation to the mediation that it undertakes. These concerns inform and arise from my work as an artist, and my participation in events, some of which I organise. The project claims that embeddedness in art is a critical condition and an editorial concept or a strategic plan that can be set up by the artist. The investigation begins by looking at conditions of embeddedness by focusing on concepts of subjectivity and by elaborating strategies that I call ‘auto-direction’. For example, concepts of subjectivity are taken up in relation to Richard Serra’s video Boomerang (1974), in which the performer Nancy Holt reflects on her own spoken words, which are fed back with a short delay via microphone and headphones into her ears. Auto-direction, introduced with the example of Steven Spielberg’s initiative of a video diary exchange project between Israeli and Palestinian children, describes the activity of the producer, who self-directs his situated presence. Taking up idioms of embeddedness from artists like Phil Collins, Christian Jankowski and Erik van Lieshout the project examines embeddedness through a comparative analysis between contemporary art, visual culture, media theory, sociology, art theory, psychoanalysis and philosophy. These practices lead to an identification of embeddedness as an author’s immanent exposure, a claim taken up through analysis of theoretical texts and literature by Rosalind Krauss, Jacques Lacan, Jacques Derrida, Gregory Bateson, Hal Foster, Bernard Williams and Alfred North Whitehead

Embeddedness as condition and strategy in contemporary art and cultural production

This thesis examines the concept of ‘embeddedness’ as condition and strategy in contemporary art and cultural production. Identifying embeddedness as a motif of contextual proximity and a strategy in contemporary art, the thesis proposes immediacy to be the result of intrinsic mediation. The project’s main concern is how embeddedness is contextualised by the current conditions that authors and cultural producers engage with. The primary question is whether and how embeddedness can convey a critical relation to the mediation that it undertakes. These concerns inform and arise from my work as an artist, and my participation in events, some of which I organise. The project claims that embeddedness in art is a critical condition and an editorial concept or a strategic plan that can be set up by the artist. The investigation begins by looking at conditions of embeddedness by focusing on concepts of subjectivity and by elaborating strategies that I call ‘auto-direction’. For example, concepts of subjectivity are taken up in relation to Richard Serra’s video Boomerang (1974), in which the performer Nancy Holt reflects on her own spoken words, which are fed back with a short delay via microphone and headphones into her ears. Auto-direction, introduced with the example of Steven Spielberg’s initiative of a video diary exchange project between Israeli and Palestinian children, describes the activity of the producer, who self-directs his situated presence. Taking up idioms of embeddedness from artists like Phil Collins, Christian Jankowski and Erik van Lieshout the project examines embeddedness through a comparative analysis between contemporary art, visual culture, media theory, sociology, art theory, psychoanalysis and philosophy. These practices lead to an identification of embeddedness as an author’s immanent exposure, a claim taken up through analysis of theoretical texts and literature by Rosalind Krauss, Jacques Lacan, Jacques Derrida, Gregory Bateson, Hal Foster, Bernard Williams and Alfred North Whitehead.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Audio-coupled video content understanding of unconstrained video sequences

Unconstrained video understanding is a difficult task. The main aim of this thesis is to recognise the nature of objects, activities and environment in a given video clip using both audio and video information. Traditionally, audio and video information has not been applied together for solving such complex task, and for the first time we propose, develop, implement and test a new framework of multi-modal (audio and video) data analysis for context understanding and labelling of unconstrained videos. The framework relies on feature selection techniques and introduces a novel algorithm (PCFS) that is faster than the well-established SFFS algorithm. We use the framework for studying the benefits of combining audio and video information in a number of different problems. We begin by developing two independent content recognition modules. The first one is based on image sequence analysis alone, and uses a range of colour, shape, texture and statistical features from image regions with a trained classifier to recognise the identity of objects, activities and environment present. The second module uses audio information only, and recognises activities and environment. Both of these approaches are preceded by detailed pre-processing to ensure that correct video segments containing both audio and video content are present, and that the developed system can be made robust to changes in camera movement, illumination, random object behaviour etc. For both audio and video analysis, we use a hierarchical approach of multi-stage classification such that difficult classification tasks can be decomposed into simpler and smaller tasks. When combining both modalities, we compare fusion techniques at different levels of integration and propose a novel algorithm that combines advantages of both feature and decision-level fusion. The analysis is evaluated on a large amount of test data comprising unconstrained videos collected for this work. We finally, propose a decision correction algorithm which shows that further steps towards combining multi-modal classification information effectively with semantic knowledge generates the best possible results

I IS ANOTHER: The fabulative filmic collaboration with someone recovering from addiction

The research is a critical exploration of the fabulative filmic collaboration with Petra, a participant recovering from heroin addiction. This fabulative approach articulates a cinema practice that seeks to address the issue of addictive behaviours in a way that has rarely been investigated, with a focus on the recovery process in the long-term. The research lies at the intersection of Film Studies, Performance Studies, Philosophy, Psychiatry and Anthropology. Theoretical insights obtained through primary practice-led film research make contributions to addiction studies by reconsidering biomedical, sociocultural and psychological research on addiction; questioning past and contemporary performative nonfiction filmmaking strategies addressing mental health narratives thereby offering a new model of filmic collaboration in relation to practice-led findings in long durational performance art. The collective filmic enquiry explores alternative safe spaces for people recovering from addiction to current cognitive-behavioural therapeutic models by addressing the crucial issue of hidden or neglected forms of mental health narratives. The doctoral research aims at exploring duration in nonfiction filmmaking and during the recovery process, shifting from rather implicit, anticipated and impressive performances to more explicit, spontaneous, subtle and durational ones. This helps to remain focused on nonverbal and more-than corporeal dimensions of addiction, which also generally remain underresearched. The research hypothesises that recovery from addiction is an explicit performance. Instead of only seeing addiction as an issue to solve, a set of symptoms to address or an urge that needs to be controlled, each new step is also a complex and rich performative experience to understand, cope with and re-enact. The model of working tests the hypothesis with help of performative techniques initially practiced in the context of long durational performance art