Image and Video Forensics

Nowadays, images and videos have become the main modalities of information being exchanged in everyday life, and their pervasiveness has led the image forensics community to question their reliability, integrity, confidentiality, and security. Multimedia contents are generated in many different ways through the use of consumer electronics and high-quality digital imaging devices, such as smartphones, digital cameras, tablets, and wearable and IoT devices. The ever-increasing convenience of image acquisition has facilitated instant distribution and sharing of digital images on digital social platforms, determining a great amount of exchange data. Moreover, the pervasiveness of powerful image editing tools has allowed the manipulation of digital images for malicious or criminal ends, up to the creation of synthesized images and videos with the use of deep learning techniques. In response to these threats, the multimedia forensics community has produced major research efforts regarding the identification of the source and the detection of manipulation. In all cases (e.g., forensic investigations, fake news debunking, information warfare, and cyberattacks) where images and videos serve as critical evidence, forensic technologies that help to determine the origin, authenticity, and integrity of multimedia content can become essential tools. This book aims to collect a diverse and complementary set of articles that demonstrate new developments and applications in image and video forensics to tackle new and serious challenges to ensure media authenticity

Dense Wide-Baseline Stereo with Varying Illumination and its Application to Face Recognition

We study the problem of dense wide baseline stereo with varying illumination. We are motivated by the problem of face recognition across pose. Stereo matching allows us to compare face images based on physically valid, dense correspondences. We show that the stereo matching cost provides a very robust measure of the similarity of faces that is insensitive to pose variations. We build on the observation that most illumination insensitive local comparisons require the use of relatively large windows. The size of these windows is affected by foreshortening. If we do not account for this effect, we incur misalignments that are systematic and significant and are exacerbated by wide baseline conditions. We present a general formulation of dense wide baseline stereo with varying illumination and provide two methods to solve them. The first method is based on dynamic programming (DP) and fully accounts for the effect of slant. The second method is based on graph cuts (GC) and fully accounts for the effect of both slant and tilt. The GC method finds a global solution using the unary function from the general formulation and a novel smoothness term that encodes surface orientation. Our experiments show that DP dense wide baseline stereo achieves superior performance compared to existing methods in face recognition across pose. The experiments with the GC method show that accounting for both slant and tilt can improve performance in situations with wide baselines and lighting variation. Our formulation can be applied to other more sophisticated window based image comparison methods for stereo

Next generation analytics for open pervasive display networks

Public displays and digital signs are becoming increasingly widely deployed as many spaces move towards becoming highly interactive and augmented environments. Market trends suggest further significant increases in the number of digital signs and both researchers and commercial entities are working on designing and developing novel uses for this technology. Given the level of investment, it is increasingly important to be able to understand the effectiveness of public displays. Current state-of-the-art analytics technology is limited in the extent to which it addresses the challenges that arise from display deployments becoming open (increasing numbers of stakeholders), networked (viewer engagement across devices and locations) and pervasive (high density of displays and sensing technology leading to potential privacy threats for viewers). In this thesis, we provide the first exploration into achieving next generation display analytics in the context of open pervasive display networks. In particular, we investigated three areas of challenge: analytics data capture, reporting and automated use of analytics data. Drawing on the increasing number of stakeholders, we conducted an extensive review of related work to identify data that can be captured by individual stakeholders of a display network, and highlighted the opportunities for gaining insights by combining datasets owned by different stakeholders. Additionally, we identified the importance of viewer-centric analytics that use traditional display-oriented analytics data combined with viewer mobility patterns to produce entirely new sets of analytics reports. We explored a range of approaches to generating viewer-centric analytics including the use of mobility models as a way to create 'synthetic analytics' - an approach that provides highly detailed analytics whilst preserving viewer privacy. We created a collection of novel viewer-centric analytics reports providing insights into how viewers experience a large network of pervasive displays including reports regarding the effectiveness of displays, the visibility of content across the display network, and the visibility of content to viewers. We further identified additional reports specific to those display networks that support the delivery of personalised content to viewers. Additionally, we highlighted the similarities between digital signage and Web analytics and introduced novel forms of digital signage analytics reports created by leveraging existing Web analytics engines. Whilst the majority of analytics systems focus solely on the capture and reporting of analytics insights, we additionally explored the automated use of analytics data. One of the challenges in open pervasive display networks is accommodating potentially competing content scheduling constraints and requirements that originate from the large number of stakeholders - in addition to contextual changes that may originate from analytics insights. To address these challenges, we designed and developed the first lottery scheduling approach for digital signage providing a means to accommodate potentially conflicting scheduling constraints, and supporting context- and event-based scheduling based on analytics data fed back into the digital sign. In order to evaluate the set of systems and approaches presented in this thesis, we conducted large-scale, long-term trials allowing us to show both the technical feasibility of the systems developed and provide insights into the accuracy and performance of different analytics capture technologies. Our work provides a set of tools and techniques for next generation digital signage analytics and lays the foundation for more general people-centric analytics that go beyond the domain of digital signs and enable unique analytical insights and understanding into how users interact across the physical and digital world