Detecting Manipulations in Video

This chapter presents the techniques researched and developed within InVID for the forensic analysis of videos, and the detection and localization of forgeries within User-Generated Videos (UGVs). Following an overview of state-of-the-art video tampering detection techniques, we observed that the bulk of current research is mainly dedicated to frame-based tampering analysis or encoding-based inconsistency characterization. We built upon this existing research, by designing forensics filters aimed to highlight any traces left behind by video tampering, with a focus on identifying disruptions in the temporal aspects of a video. As for many other data analysis domains, deep neural networks show very promising results in tampering detection as well. Thus, following the development of a number of analysis filters aimed to help human users in highlighting inconsistencies in video content, we proceeded to develop a deep learning approach aimed to analyze the outputs of these forensics filters and automatically detect tampered videos. In this chapter, we present our survey of the state of the art with respect to its relevance to the goals of InVID, the forensics filters we developed and their potential role in localizing video forgeries, as well as our deep learning approach for automatic tampering detection. We present experimental results on benchmark and real-world data, and analyze the results. We observe that the proposed method yields promising results compared to the state of the art, especially with respect to the algorithm’s ability to generalize to unknown data taken from the real world. We conclude with the research directions that our work in InVID has opened for the future

Morphological Filter Detector for Image Forensics Applications

Mathematical morphology provides a large set of powerful non-linear image operators, widely used for feature extraction, noise removal or image enhancement. Although morphological filters might be used to remove artifacts produced by image manipulations, both on binary and graylevel documents, little effort has been spent towards their forensic identification. In this paper we propose a non-trivial extension of a deterministic approach originally detecting erosion and dilation of binary images. The proposed approach operates on grayscale images and is robust to image compression and other typical attacks. When the image is attacked the method looses its deterministic nature and uses a properly trained SVM classifier, using the original detector as a feature extractor. Extensive tests demonstrate that the proposed method guarantees very high accuracy in filtering detection, providing 100% accuracy in discriminating the presence and the type of morphological filter in raw images of three different datasets. The achieved accuracy is also good after JPEG compression, equal or above 76.8% on all datasets for quality factors above 80. The proposed approach is also able to determine the adopted structuring element for moderate compression factors. Finally, it is robust against noise addition and it can distinguish morphological filter from other filters

Machine learning based digital image forensics and steganalysis

The security and trustworthiness of digital images have become crucial issues due to the simplicity of malicious processing. Therefore, the research on image steganalysis (determining if a given image has secret information hidden inside) and image forensics (determining the origin and authenticity of a given image and revealing the processing history the image has gone through) has become crucial to the digital society. In this dissertation, the steganalysis and forensics of digital images are treated as pattern classification problems so as to make advanced machine learning (ML) methods applicable. Three topics are covered: (1) architectural design of convolutional neural networks (CNNs) for steganalysis, (2) statistical feature extraction for camera model classification, and (3) real-world tampering detection and localization. For covert communications, steganography is used to embed secret messages into images by altering pixel values slightly. Since advanced steganography alters the pixel values in the image regions that are hard to be detected, the traditional ML-based steganalytic methods heavily relied on sophisticated manual feature design have been pushed to the limit. To overcome this difficulty, in-depth studies are conducted and reported in this dissertation so as to move the success achieved by the CNNs in computer vision to steganalysis. The outcomes achieved and reported in this dissertation are: (1) a proposed CNN architecture incorporating the domain knowledge of steganography and steganalysis, and (2) ensemble methods of the CNNs for steganalysis. The proposed CNN is currently one of the best classifiers against steganography. Camera model classification from images aims at assigning a given image to its source capturing camera model based on the statistics of image pixel values. For this, two types of statistical features are designed to capture the traces left by in-camera image processing algorithms. The first is Markov transition probabilities modeling block-DCT coefficients for JPEG images; the second is based on histograms of local binary patterns obtained in both the spatial and wavelet domains. The designed features serve as the input to train support vector machines, which have the best classification performance at the time the features are proposed. The last part of this dissertation documents the solutions delivered by the author’s team to The First Image Forensics Challenge organized by the Information Forensics and Security Technical Committee of the IEEE Signal Processing Society. In the competition, all the fake images involved were doctored by popular image-editing software to simulate the real-world scenario of tampering detection (determine if a given image has been tampered or not) and localization (determine which pixels have been tampered). In Phase-1 of the Challenge, advanced steganalysis features were successfully migrated to tampering detection. In Phase-2 of the Challenge, an efficient copy-move detector equipped with PatchMatch as a fast approximate nearest neighbor searching method were developed to identify duplicated regions within images. With these tools, the author’s team won the runner-up prizes in both the two phases of the Challenge

Digital image forensics via meta-learning and few-shot learning

Digital images are a substantial portion of the information conveyed by social media, the Internet, and television in our daily life. In recent years, digital images have become not only one of the public information carriers, but also a crucial piece of evidence. The widespread availability of low-cost, user-friendly, and potent image editing software and mobile phone applications facilitates altering images without professional expertise. Consequently, safeguarding the originality and integrity of digital images has become a difficulty. Forgers commonly use digital image manipulation to transmit misleading information. Digital image forensics investigates the irregular patterns that might result from image alteration. It is crucial to information security. Over the past several years, machine learning techniques have been effectively used to identify image forgeries. Convolutional Neural Networks(CNN) are a frequent machine learning approach. A standard CNN model could distinguish between original and manipulated images. In this dissertation, two CNN models are introduced to recognize seam carving and Gaussian filtering. Training a conventional CNN model for a new similar image forgery detection task, one must start from scratch. Additionally, many types of tampered image data are challenging to acquire or simulate. Meta-learning is an alternative learning paradigm in which a machine learning model gets experience across numerous related tasks and uses this expertise to improve its future learning performance. Few-shot learning is a method for acquiring knowledge from few data. It can classify images with as few as one or two examples per class. Inspired by meta-learning and few-shot learning, this dissertation proposed a prototypical networks model capable of resolving a collection of related image forgery detection problems. Unlike traditional CNN models, the proposed prototypical networks model does not need to be trained from scratch for a new task. Additionally, it drastically decreases the quantity of training images

Correlated Activity and Corticothalamic Cell Function in the Early Mouse Visual System

Vision has long been the model for understanding cortical function. Great progress has been made in understanding the transformations that occur within some primary visual cortex (V1) layers, like the emergence of orientation selectivity in layer 4. Less is known about other V1 circuit elements, like the shaping of V1 input via corticothalamic projections, or the population structure of the cortico-cortical output in layer 2/3. Here, we use the mouse early visual system to investigate the structure and function of circuit elements in V1. We use two approaches: comparative physiology and optogenetics. We measured the structure of pairwise correlations in the output layer 2/3 using extracellular recordings. We find that despite a lack of organization in mouse V1 seen in other species, the specificity of connections preserves a correlation structure on multiple timescales. To investigate the role of corticogeniculate projections, we utilize a transgenic mouse line to specifically and reversibly manipulate these projections with millisecond precision. We find that activity of these cells results a mix of inhibition and excitation in the thalamus, is not spatiotemporally specific, and can affect correlated activity. Finally, we classify mouse thalamic cells according to stimuli used for cell classification in primates and cats, finding some, but not complete, homology to the processing streams of primate thalamus and further highlighting fundamentals of mammalian visual system organization

Assessing Neuronal Synchrony and Brain Function Through Local Field Potential and Spike Analysis

Studies of neuronal network oscillations and rhythmic neuronal synchronization have led to a number of important insights in recent years, giving us a better understanding of the temporal organization of neuronal activity related to essential brain functions like sensory processing and cognition. Important principles and theories have emerged from these findings, including the communication through coherence hypothesis, which proposes that synchronous oscillations render neuronal communication effective, selective, and precise. The implications of such a theory may be universal for brain function, as the determinants of neuronal communication inextricably shape the neuronal representation of information in the brain. However, the study of communication through coherence is still relatively young. Since its articulation in 2005, the theory has predominantly been applied to assess cortical function and its communication with downstream targets in different sensory and behavioral conditions. The results herein are intended to bolster this hypothesis and explore new ways in which oscillations coordinate neuronal communication in distributed regions. This includes the development of new analytic tools for interpreting electrophysiological patterns, inspired by phase synchronization and spike train analysis. These tools aim to offer fast results with clear statistical and physiological interpretation

Vers l’anti-criminalistique en images numériques via la restauration d’images

Image forensics enjoys its increasing popularity as a powerful image authentication tool, working in a blind passive way without the aid of any a priori embedded information compared to fragile image watermarking. On its opponent side, image anti-forensics attacks forensic algorithms for the future development of more trustworthy forensics. When image coding or processing is involved, we notice that image anti-forensics to some extent shares a similar goal with image restoration. Both of them aim to recover the information lost during the image degradation, yet image anti-forensics has one additional indispensable forensic undetectability requirement. In this thesis, we form a new research line for image anti-forensics, by leveraging on advanced concepts/methods from image restoration meanwhile with integrations of anti-forensic strategies/terms. Under this context, this thesis contributes on the following four aspects for JPEG compression and median filtering anti-forensics: (i) JPEG anti-forensics using Total Variation based deblocking, (ii) improved Total Variation based JPEG anti-forensics with assignment problem based perceptual DCT histogram smoothing, (iii) JPEG anti-forensics using JPEG image quality enhancement based on a sophisticated image prior model and non-parametric DCT histogram smoothing based on calibration, and (iv) median filtered image quality enhancement and anti-forensics via variational deconvolution. Experimental results demonstrate the effectiveness of the proposed anti-forensic methods with a better forensic undetectability against existing forensic detectors as well as a higher visual quality of the processed image, by comparisons with the state-of-the-art methods.La criminalistique en images numériques se développe comme un outil puissant pour l'authentification d'image, en travaillant de manière passive et aveugle sans l'aide d'informations d'authentification pré-intégrées dans l'image (contrairement au tatouage fragile d'image). En parallèle, l'anti-criminalistique se propose d'attaquer les algorithmes de criminalistique afin de maintenir une saine émulation susceptible d'aider à leur amélioration. En images numériques, l'anti-criminalistique partage quelques similitudes avec la restauration d'image : dans les deux cas, l'on souhaite approcher au mieux les informations perdues pendant un processus de dégradation d'image. Cependant, l'anti-criminalistique se doit de remplir au mieux un objectif supplémentaire, extit{i.e.} : être non détectable par la criminalistique actuelle. Dans cette thèse, nous proposons une nouvelle piste de recherche pour la criminalistique en images numériques, en tirant profit des concepts/méthodes avancés de la restauration d'image mais en intégrant des stratégies/termes spécifiquement anti-criminalistiques. Dans ce contexte, cette thèse apporte des contributions sur quatre aspects concernant, en criminalistique JPEG, (i) l'introduction du déblocage basé sur la variation totale pour contrer les méthodes de criminalistique JPEG et (ii) l'amélioration apportée par l'adjonction d'un lissage perceptuel de l'histogramme DCT, (iii) l'utilisation d'un modèle d'image sophistiqué et d'un lissage non paramétrique de l'histogramme DCT visant l'amélioration de la qualité de l'image falsifiée; et, en criminalistique du filtrage médian, (iv) l'introduction d'une méthode fondée sur la déconvolution variationnelle. Les résultats expérimentaux démontrent l'efficacité des méthodes anti-criminalistiques proposées, avec notamment une meilleure indétectabilité face aux détecteurs criminalistiques actuels ainsi qu'une meilleure qualité visuelle de l'image falsifiée par rapport aux méthodes anti-criminalistiques de l'état de l'art

Novel planar photonic antennas to address the dynamic nanoarchitecture of biological membranes

The cell membrane is the encompassing protective shield of every cell and it is composed of a multitude of proteins, lipids and other molecules. The organization of the cell membrane is inextricably intertwined with its function, and sensitive to perturbations from the underlying actin cytoskeleton and the extracellular environment at the nano- and the mesoscale. Elucidating the dynamic interplay between lipids and proteins diffusing on the cell membrane, forming transient domains and (re)organizing them according to signals from the juxtaposed inner and outer meshwork, is of paramount interest in fundamental cell biology. The overarching goal of this thesis is to gain deeper insight into how lipids and proteins dynamically organize in biological membranes at the nanoscale. Photonic nano-antennas are metallic nanostructures that localize and enhance the incident optical radiation into highly confined nanometric regions (< 20 nm), leading to greatly enhanced light-matter interactions. In this thesis, we exploit an innovative design of planar gold nano-antenna arrays of different gap sizes (10-45 nm) and embedded in nanometric-size boxes. To elucidate nanoscale diffusion dynamics in biological membranes with high spatiotemporal resolution and single-molecule detection sensitivity, we further combine our nanogap antenna arrays with fluorescence correlation spectroscopy (FCS) in a serial and multiplexed manner. In this dissertation, we first describe the fabrication process of these planar gold nanogap antennas and characterize their performance by means of electron microscopy and FCS of individual molecules in solution. We demonstrate giant fluorescence enhancement factors of up to 104-105 times provided by our planar nanogap antennas in ultra-confined detection volumes and with single molecule detection sensitivity in the micromolar range. Second, we apply these planar plasmonic nano-antennas in combination with FCS for assessing the dynamic organization of mimetic lipid membranes at the nanoscale. For a ternary composition of the model membranes that include unsaturated and saturated lipids together with cholesterol, we resolve transient nanoscopic heterogeneities as small as 10 nm in size, coexisting in both macroscopically phase-separated lipid phases. Third, we add a Hyaluronic Acid (HA) layer on top of the model lipid membranes to emulate the effect of the extracellular environment surrounding native biological membranes. We extend our nano-antenna-FCS approach with atomic force microscopy and spectroscopy. We reveal a distinct influence of HA on the nanoscale lipid organization of mimetic membranes composed of lipids constituting the more ordered lipid phase. Our results indicate a synergistic effect of cholesterol and HA re-organizing biological membranes at the nanoscale. Fourth, we apply our planar nano-antenna platform combined with FCS to elucidate the nanoscale dynamics of different lipids in living cells. With our nanogap antennas we were able to breach into the sub-30 nm spatial scale on living cell membranes for the first time. We provide compelling evidence of short-lived cholesterol-induced ~10 nm nanodomain partitioning in living plasma membranes. Fifth, we demonstrate the multiplexing capabilities of our planar gold nanogap antenna platform combined with FCS in a widefield illumination scheme combined with sCMOS camera detection. Our approach allows recording of fluorescence signal from more than 200 antennas simultaneously. Moreover, we demonstrate multiplexed FCS recording on 50 nano-antennas simultaneously, both in solution as well as in living cells, with a temporal resolution in the millisecond range. The dissertation finishes with a brief discussion of the main results achieved in this research and proposes new avenues for future research in the field.La membrana plasmática separa el entorno intracelular del extracelular y está compuesta por una multitud de diferentes proteínas y lípidos. Su organización está fuertemente interconectada a su función, y es sensible a perturbaciones tanto de la actina cortical posicionada internamente en proximidad con la membrana, así como de una red extracelular en contacto próximo con la membrana exterior. Estas perturbaciones ocurren a distintas escalas temporales y espaciales, llegando a unos pocos nanómetros. Dada la estrecha relación entre la organización de la membrana y su función biológica, es tremendamente importante entender como lípidos y proteínas se organizan dinámicamente a la escala nanométrica y como se ven afectados por su entorno. El objetivo principal de esta tesis doctoral se centra en alcanzar este entendimiento. Las antenas fotónicas son nano-estructuras metálicas que incrementan la radiación electromagnética en regiones nanométricas (< 20 nm) del espacio. En esta tesis doctoral, hemos fabricado y utilizado plataformas con matrices de antenas en oro, y con regiones de confinamiento entre 10-45 nm. Además, hemos combinado estas antenas con la técnica de ¿fluorescence correlation spectroscopy (FCS)¿ a fin de obtener información espaciotemporal a la nano-escala en membranas biológicas, junto a la sensibilidad de detectar moléculas individuales a altas concentraciones. En esta disertación, describimos primero la fabricación de antenas fotónicas y caracterizamos su rendimiento utilizando técnicas de microscopía electrónica y FCS de moléculas individuales en solución. Nuestros resultados demuestran factores de incremento de la fluorescencia entre 104-105, en regiones ultra-confinadas, y una capacidad para detectar moléculas individuales en rango de concentraciones de micro-molares. Una vez validadas nuestras herramientas, nos enfocamos en su uso para el estudio dinámico de la organización de membranas lipídicas miméticas a escala nanométrica. En el caso de composiciones ternarias de lípidos insaturados, saturados y colesterol, hemos descubierto la existencia de heterogeneidades nanoscópicas y transitorias que coexisten tanto en las regiones ordenadas como desordenadas de las membranas lipídicas. El siguiente capítulo contiene resultados enfocados a estudiar el efecto del entorno extracelular en la organización dinámica de este tipo de capas lipídicas. Para ello, y como modelo, preparamos membranas lipídicas cubiertas de ácido hialurónico (HA), un componente abundantemente expresado en la matriz extracelular. Combinando FCS con microscopia y espectroscopia de fuerzas atómicas, logramos resolver la influencia de HA a escala nanométrica en la organización de la fase ordenada de las membranas lipídicas. Nuestros resultados indican la existencia de un efecto sinérgico entre HA y colesterol en el reordenamiento de la membrana a la nano-escala. El siguiente tema de investigación en esta tesis doctoral se enfoca a la aplicación de antenas fotónicas y FCS para el estudio de dominios lipídicos enriquecidos de colesterol en la membrana plasmática de células vivas. La utilización de estas antenas nos ha permitido, por primera vez, remontar la barrera de 30 nm, y demostrar de manera inequívoca la existencia de dominios enriquecidos en colesterol en células vivas con una resolución espacial de 10 nm. Finalmente, hemos demostrado la capacidad de multiplexado de nuestras antenas fotónicas, combinando una iluminación y detección en campo amplio utilizando una camera sCMOS. Describimos la implementación de nuestro esquema, así como también medidas que demuestran la detección simultánea de fluorescencia en más de 200 antenas. De manera importante, demostramos la obtención de curvas de FCS en 50 antenas simultáneamente, tanto en solución como en células vivas. Esta disertación culmina con una breve discusión de los resultados más importantes de esta investigación en el futur