Video inpainting for non-repetitive motion

Master'sMASTER OF SCIENC

How Not to Be Seen -- Inpainting Dynamic Objects in Crowded Scenes

Removing dynamic objects from videos is an extremely challenging problem that even visual effects professionals often solve with time-consuming manual frame-by-frame editing. We propose a new approach to video completion that can deal with complex scenes containing dynamic background and non-periodical moving objects. We build upon the idea that the spatio-temporal hole left by a removed object can be filled with data available on other regions of the video where the occluded objects were visible. Video completion is performed by solving a large combinatorial problem that searches for an optimal pattern of pixel offsets from occluded to unoccluded regions. Our contribution includes an energy functional that generalizes well over different scenes with stable parameters, and that has the desirable convergence properties for a graph-cut-based optimization. We provide an interface to guide the completion process that both reduces computation time and allows for efficient correction of small errors in the result. We demonstrate that our approach can effectively complete complex, high-resolution occlusions that are greater in difficulty than what existing methods have shown

Sur la Restauration et l'Edition de Vidéo : Détection de Rayures et Inpainting de Scènes Complexes

The inevitable degradation of visual content such as images and films leads to the goal ofimage and video restoration. In this thesis, we look at two specific restoration problems : the detection ofline scratches in old films and the automatic completion of videos, or video inpainting as it is also known.Line scratches are caused when the film physically rubs against a mechanical part. This origin resultsin the specific characteristics of the defect, such as verticality and temporal persistence. We propose adetection algorithm based on the statistical approach known as a contrario methods. We also proposea temporal filtering step to remove false alarms present in the first detection step. Comparisons withprevious work show improved recall and precision, and robustness with respect to the presence of noiseand clutter in the film.The second part of the thesis concerns video inpainting. We propose an algorithm based on theminimisation of a patch-based functional of the video content. In this framework, we address the followingproblems : extremely high execution times, the correct handling of textures in the video and inpaintingwith moving cameras. We also address some convergence issues in a very simplified inpainting context.La degradation inévitable des contenus visuels (images, films) conduit nécessairementà la tâche de la restauration des images et des vidéos. Dans cetre thèse, nous nous intéresserons àdeux sous-problèmes de restauration : la détection des rayures dans les vieux films, et le remplissageautomatique des vidéos (“inpainting vidéo en anglais).En général, les rayures sont dues aux frottements de la pellicule du film avec un objet lors de laprojection du film. Les origines physiques de ce défaut lui donnent des caractéristiques très particuliers.Les rayures sont des lignes plus ou moins verticales qui peuvent être blanches ou noires (ou parfois encouleur) et qui sont temporellement persistantes, c’est-à-dire qu’elles ont une position qui est continuedans le temps. Afin de détecter ces défauts, nous proposons d’abord un algorithme de détection basésur un ensemble d’approches statistiques appelées les méthodes a contrario. Cet algorithme fournitune détection précise et robuste aux bruits et aux textures présentes dans l’image. Nous proposonségalement une étape de filtrage temporel afin d’écarter les fausses alarmes de la première étape dedétection. Celle-ci améliore la précision de l’algorithme en analysant le mouvement des détections spatiales.L’ensemble de l’algorithme (détection spatiale et filtrage temporel) est comparé à des approchesde la littérature et montre un rappel et une précision grandement améliorés.La deuxième partie de cette thèse est consacrée à l’inpainting vidéo. Le but ici est de remplirune région d’une vidéo avec un contenu qui semble visuellement cohérent et convaincant. Il existeune pléthore de méthodes qui traite ce problème dans le cas des images. La littérature dans le casdes vidéos est plus restreinte, notamment car le temps d’exécution représente un véritable obstacle.Nous proposons un algorithme d’inpainting vidéo qui vise l’optimisation d’une fonctionnelle d’énergiequi intègre la notion de patchs, c’est-à-dire des petits cubes de contenu vidéo. Nous traitons d’abord leprobl’‘eme du temps d’exécution avant d’attaquer celui de l’inpainting satisfaisant des textures dans lesvidéos. Nous traitons également le cas des vidéos dont le fond est en mouvement ou qui ont été prisesavec des caméras en mouvement. Enfin, nous nous intéressons à certaines questions de convergencede l’algorithme dans des cas très simplifiés

Optimizing Apparent Display Resolution Enhancement for Arbitrary Videos

Display resolution is frequently exceeded by available image resolution. Recently, apparent display resolution enhancement techniques (ADRE) have demonstrated how characteristics of the human visual system can be exploited to provide super-resolution on high refresh rate displays. In this paper we address the problem of generalizing the apparent display resolution enhancement technique to conventional videos of arbitrary content. We propose an optimization-based approach to continuously translate the video frames in such a way that the added motion enables apparent resolution enhancement for the salient image region. The optimization takes the optimal velocity, smoothness and similarity into account to compute an appropriate trajectory. Additionally, we provide an intuitive user interface which allows to guide the algorithm interactively and preserve important compositions within the video. We present a user study evaluating apparent rendering quality and demonstrate versatility of our method on a variety of general test scenes.Aktuelle Kameras sind in der Lage, Videos mit sehr hoher Auflösung aufzunehmen (> 4K Pixel). Monitore, Fernseher und Projektoren haben jedoch meist eine deutlich niedrigere Auflösung (FullHD). Bei der Darstellung hochaufgelöster Videos auf diesen Geräten gehen durch das nötige Herrunterrechnen der Videodaten feine Details verloren, z.B. Haare oder die Pigmentierung von Oberflächenmaterialien. Es wird ein Verfahren präsentiert, welches die Darstellung eines beliebigen Videos mit einer Auflösung ermöglicht, die perzeptuell höher ist als die Auflösung des Ausgabegerätes

Image and Video Forensics

Nowadays, images and videos have become the main modalities of information being exchanged in everyday life, and their pervasiveness has led the image forensics community to question their reliability, integrity, confidentiality, and security. Multimedia contents are generated in many different ways through the use of consumer electronics and high-quality digital imaging devices, such as smartphones, digital cameras, tablets, and wearable and IoT devices. The ever-increasing convenience of image acquisition has facilitated instant distribution and sharing of digital images on digital social platforms, determining a great amount of exchange data. Moreover, the pervasiveness of powerful image editing tools has allowed the manipulation of digital images for malicious or criminal ends, up to the creation of synthesized images and videos with the use of deep learning techniques. In response to these threats, the multimedia forensics community has produced major research efforts regarding the identification of the source and the detection of manipulation. In all cases (e.g., forensic investigations, fake news debunking, information warfare, and cyberattacks) where images and videos serve as critical evidence, forensic technologies that help to determine the origin, authenticity, and integrity of multimedia content can become essential tools. This book aims to collect a diverse and complementary set of articles that demonstrate new developments and applications in image and video forensics to tackle new and serious challenges to ensure media authenticity

State of the art in privacy preservation in video data

Active and Assisted Living (AAL) technologies and services are a possible solution to address the crucial challenges regarding health and social care resulting from demographic changes and current economic conditions. AAL systems aim to improve quality of life and support independent and healthy living of older and frail people. AAL monitoring systems are composed of networks of sensors (worn by the users or embedded in their environment) processing elements and actuators that analyse the environment and its occupants to extract knowledge and to detect events, such as anomalous behaviours, launch alarms to tele-care centres, or support activities of daily living, among others. Therefore, innovation in AAL can address healthcare and social demands while generating economic opportunities. Recently, there has been far-reaching advancements in the development of video-based devices with improved processing capabilities, heightened quality, wireless data transfer, and increased interoperability with Internet of Things (IoT) devices. Computer vision gives the possibility to monitor an environment and report on visual information, which is commonly the most straightforward and human-like way of describing an event, a person, an object, interactions and actions. Therefore, cameras can offer more intelligent solutions for AAL but they may be considered intrusive by some end users. The General Data Protection Regulation (GDPR) establishes the obligation for technologies to meet the principles of data protection by design and by default. More specifically, Article 25 of the GDPR requires that organizations must "implement appropriate technical and organizational measures [...] which are designed to implement data protection principles [...] , in an effective manner and to integrate the necessary safeguards into [data] processing.” Thus, AAL solutions must consider privacy-by-design methodologies in order to protect the fundamental rights of those being monitored. Different methods have been proposed in the latest years to preserve visual privacy for identity protection. However, in many AAL applications, where mostly only one person would be present (e.g. an older person living alone), user identification might not be an issue; concerns are more related to the disclosure of appearance (e.g. if the person is dressed/naked) and behaviour, what we called bodily privacy. Visual obfuscation techniques, such as image filters, facial de-identification, body abstraction, and gait anonymization, can be employed to protect privacy and agreed upon by the users ensuring they feel comfortable. Moreover, it is difficult to ensure a high level of security and privacy during the transmission of video data. If data is transmitted over several network domains using different transmission technologies and protocols, and finally processed at a remote location and stored on a server in a data center, it becomes demanding to implement and guarantee the highest level of protection over the entire transmission and storage system and for the whole lifetime of the data. The development of video technologies, increase in data rates and processing speeds, wide use of the Internet and cloud computing as well as highly efficient video compression methods have made video encryption even more challenging. Consequently, efficient and robust encryption of multimedia data together with using efficient compression methods are important prerequisites in achieving secure and efficient video transmission and storage.This publication is based upon work from COST Action GoodBrother - Network on Privacy-Aware Audio- and Video-Based Applications for Active and Assisted Living (CA19121), supported by COST (European Cooperation in Science and Technology). COST (European Cooperation in Science and Technology) is a funding agency for research and innovation networks. Our Actions help connect research initiatives across Europe and enable scientists to grow their ideas by sharing them with their peers. This boosts their research, career and innovation. www.cost.e

Multimedia Forensics

This book is open access. Media forensics has never been more relevant to societal life. Not only media content represents an ever-increasing share of the data traveling on the net and the preferred communications means for most users, it has also become integral part of most innovative applications in the digital information ecosystem that serves various sectors of society, from the entertainment, to journalism, to politics. Undoubtedly, the advances in deep learning and computational imaging contributed significantly to this outcome. The underlying technologies that drive this trend, however, also pose a profound challenge in establishing trust in what we see, hear, and read, and make media content the preferred target of malicious attacks. In this new threat landscape powered by innovative imaging technologies and sophisticated tools, based on autoencoders and generative adversarial networks, this book fills an important gap. It presents a comprehensive review of state-of-the-art forensics capabilities that relate to media attribution, integrity and authenticity verification, and counter forensics. Its content is developed to provide practitioners, researchers, photo and video enthusiasts, and students a holistic view of the field