Multimedia Forensics

This book is open access. Media forensics has never been more relevant to societal life. Not only media content represents an ever-increasing share of the data traveling on the net and the preferred communications means for most users, it has also become integral part of most innovative applications in the digital information ecosystem that serves various sectors of society, from the entertainment, to journalism, to politics. Undoubtedly, the advances in deep learning and computational imaging contributed significantly to this outcome. The underlying technologies that drive this trend, however, also pose a profound challenge in establishing trust in what we see, hear, and read, and make media content the preferred target of malicious attacks. In this new threat landscape powered by innovative imaging technologies and sophisticated tools, based on autoencoders and generative adversarial networks, this book fills an important gap. It presents a comprehensive review of state-of-the-art forensics capabilities that relate to media attribution, integrity and authenticity verification, and counter forensics. Its content is developed to provide practitioners, researchers, photo and video enthusiasts, and students a holistic view of the field

Indoor positioning with deep learning for mobile IoT systems

2022 Summer.Includes bibliographical references.The development of human-centric services with mobile devices in the era of the Internet of Things (IoT) has opened the possibility of merging indoor positioning technologies with various mobile applications to deliver stable and responsive indoor navigation and localization functionalities that can enhance user experience within increasingly complex indoor environments. But as GPS signals cannot easily penetrate modern building structures, it is challenging to build reliable indoor positioning systems (IPS). Currently, Wi-Fi sensing based indoor localization techniques are gaining in popularity as a means to build accurate IPS, benefiting from the prevalence of 802.11 family. Wi-Fi fingerprinting based indoor localization has shown remarkable performance over geometric mapping in complex indoor environments by taking advantage of pattern matching techniques. Today, the two main information extracted from Wi-Fi signals to form fingerprints are Received Signal Strength Index (RSSI) and Channel State Information (CSI) with Orthogonal Frequency-Division Multiplexing (OFDM) modulation, where the former can provide the average localization error around or under 10 meters but has low hardware and software requirements, while the latter has a higher chance to estimate locations with ultra-low distance errors but demands more resources from chipsets, firmware/software environments, etc. This thesis makes two novel contributions towards realizing viable IPS on mobile devices using RSSI and CSI information, and deep machine learning based fingerprinting. Due to the larger quantity of data and more sophisticated signal patterns to create fingerprints in complex indoor environments, conventional machine learning algorithms that need carefully engineered features suffer from the challenges of identifying features from very high dimensional data. Hence, the abilities of approximation functions generated from conventional machine learning models to estimate locations are limited. Deep machine learning based approaches can overcome these challenges to realize scalable feature pattern matching approaches such as fingerprinting. However, deep machine learning models generally require considerable memory footprint, and this creates a significant issue on resource-constrained devices such as mobile IoT devices, wearables, smartphones, etc. Developing efficient deep learning models is a critical factor to lower energy consumption for resource intensive mobile IoT devices and accelerate inference time. To address this issue, our first contribution proposes the CHISEL framework, which is a Wi-Fi RSSI- based IPS that incorporates data augmentation and compression-aware two-dimensional convolutional neural networks (2D CAECNNs) with different pruning and quantization options. The proposed model compression techniques help reduce model deployment overheads in the IPS. Unlike RSSI, CSI takes advantages of multipath signals to potentially help indoor localization algorithms achieve a higher level of localization accuracy. The compensations for magnitude attenuation and phase shifting during wireless propagation generate different patterns that can be utilized to define the uniqueness of different locations of signal reception. However, all prior work in this domain constrains the experimental space to relatively small-sized and rectangular rooms where the complexity of building interiors and dynamic noise from human activities, etc., are seldom considered. As part of our second contribution, we propose an end-to-end deep learning based framework called CSILoc for Wi-Fi CSI-based IPS on mobile IoT devices. The framework includes CSI data collection, clustering, denoising, calibration and classification, and is the first study to verify the feasibility to use CSI for floor level indoor localization with minimal knowledge of Wi-Fi access points (APs), thus avoiding security concerns during the offline data collection process

Image and Video Forensics

Nowadays, images and videos have become the main modalities of information being exchanged in everyday life, and their pervasiveness has led the image forensics community to question their reliability, integrity, confidentiality, and security. Multimedia contents are generated in many different ways through the use of consumer electronics and high-quality digital imaging devices, such as smartphones, digital cameras, tablets, and wearable and IoT devices. The ever-increasing convenience of image acquisition has facilitated instant distribution and sharing of digital images on digital social platforms, determining a great amount of exchange data. Moreover, the pervasiveness of powerful image editing tools has allowed the manipulation of digital images for malicious or criminal ends, up to the creation of synthesized images and videos with the use of deep learning techniques. In response to these threats, the multimedia forensics community has produced major research efforts regarding the identification of the source and the detection of manipulation. In all cases (e.g., forensic investigations, fake news debunking, information warfare, and cyberattacks) where images and videos serve as critical evidence, forensic technologies that help to determine the origin, authenticity, and integrity of multimedia content can become essential tools. This book aims to collect a diverse and complementary set of articles that demonstrate new developments and applications in image and video forensics to tackle new and serious challenges to ensure media authenticity

Deep Learning for Face Anti-Spoofing: A Survey

Face anti-spoofing (FAS) has lately attracted increasing attention due to its vital role in securing face recognition systems from presentation attacks (PAs). As more and more realistic PAs with novel types spring up, traditional FAS methods based on handcrafted features become unreliable due to their limited representation capacity. With the emergence of large-scale academic datasets in the recent decade, deep learning based FAS achieves remarkable performance and dominates this area. However, existing reviews in this field mainly focus on the handcrafted features, which are outdated and uninspiring for the progress of FAS community. In this paper, to stimulate future research, we present the first comprehensive review of recent advances in deep learning based FAS. It covers several novel and insightful components: 1) besides supervision with binary label (e.g., '0' for bonafide vs. '1' for PAs), we also investigate recent methods with pixel-wise supervision (e.g., pseudo depth map); 2) in addition to traditional intra-dataset evaluation, we collect and analyze the latest methods specially designed for domain generalization and open-set FAS; and 3) besides commercial RGB camera, we summarize the deep learning applications under multi-modal (e.g., depth and infrared) or specialized (e.g., light field and flash) sensors. We conclude this survey by emphasizing current open issues and highlighting potential prospects.Comment: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI