It’s About Time: Projecting Temporal Metadata for Historically Significant Recordings

Twentieth century audio recordings and motion pictures are important sources, both for scholarly analysis and for public history. In some cases, important metadata has not reached the collecting institutions along with the materials, which are now in need of richer description. This paper describes a novel technique for determining the date and time on which a recording was made based on analysis of incidentally captured traces of small variations in the electric power supply at the time the recording was made

Wide-Area Measurement-Based Applications for Power System Monitoring and Dynamic Modeling

Due to the increasingly complex behavior exhibited by large-scale power systems with more uncertain renewables introduced to the grid, wide-area measurement system (WAMS) has been utilized to complement the traditional supervisory control and data acquisition (SCADA) system to improve operators’ situational awareness. By providing wide-area GPS-time-synchronized measurements of grid status at high time-resolution, it is able to reveal power system dynamics which cannot be captured before and has become an essential tool to deal with current and future power grid challenges. According to the time requirements of different power system applications, the applications can be roughly divided into online applications (e.g., data visualization, fast disturbance and oscillation detection, and system response prediction and reduction) and offline applications (e.g., measurement-driven dynamic modeling and validation, post-event analysis, and statistical analysis of historical data). In this dissertation, various wide-area measurement-based applications are presented. Firstly a pioneering WAMS deployed at the distribution level, the frequency monitoring network (FNET/GridEye) is introduced. For conventional large-scale power grid dynamic simulation, two major challenges are 1) accuracy of detailed dynamic models, and 2) computation burden for online dynamic assessment. To overcome the restrictions of the traditional approach, a measurement-based system response prediction tool using a Multivariate AutoRegressive (MAR) model is developed. It is followed by a measurement-based power system dynamic reduction tool using an autoregressive model vi to represent the external system. In addition, phasor measurement unit (PMU) data are employed to perform the generator dynamic model validation study. It utilizes both simulation data and measurement data to explore the potentials and limitations of the proposed approach. As an innovative application of using wide-area power system measurement, digital recordings could be authenticated by comparing the extracted frequency and phase angle from recordings with power system measurement database. It includes four research studies, i.e., oscillator error removal, ENF phenomenology, tampering detection, and frequency localization. Finally, several preliminary data analytics studies including inertia estimation and analysis, fault-induced delayed voltage recovery (FIDVR) detection, and statistical analysis of oscillation database, are presented

Intrinsically Embedded Signatures for Multimedia Forensics

This dissertation examines the use of signatures that are intrinsically embedded in media recordings for studies and applications in multimedia forensics. These near-invisible signatures are fingerprints that are captured unintentionally in a recording due to influences from the environment in which it was made and the recording device that was used to make it. We focus on two types of such signatures: the Electric Network Frequency (ENF) signal and the flicker signal. The ENF is the frequency of power distribution networks and has a nominal value of 50Hz or 60Hz. The ENF fluctuates around its nominal value due to load changes in the grid. It is particularly relevant to multimedia forensics because ENF variations captured intrinsically in a media recording reflect the time and location related properties of the respective area in which it was made. This has led to a number of applications in information forensics and security, such as time-of-recording authentication/estimation and ENF-based detection of tampering in a recording. The first part of this dissertation considers the extraction and detection of the ENF signal. We discuss our proposed spectrum combining approach for ENF estimation that exploits the presence of ENF traces at several harmonics within the same recording to produce more accurate and robust ENF signal estimates. We also explore possible factors that can promote or hinder the capture of ENF traces in recordings, which is important for a better understanding of the real-world applicability of ENF signals. Next, we discuss novel real-world ENF-based applications proposed through this dissertation research. We discuss using the embedded ENF signal to identify the region-of-recording of a media signal through a pattern analysis and learning framework that distinguishes between ENF signals coming from different power grids. We also discuss the use of the ENF traces embedded in a video to characterize the video camera that had originally produced the video, an application that was inspired by our work on flicker forensics. The last part of the dissertation considers the flicker signal and its use in forensics. We address problems in the entertainment industry pertaining to movie piracy related investigations, where a pirated movie is formed by camcording media content shown on an LCD screen. The flicker signature can be inherently created in such a scenario due to the interplay between the back-light of an LCD screen and the recording mechanism of the video camera. We build an analytic model of the flicker, relating it to inner parameters of the video camera and the screen producing the video. We then demonstrate that solely analyzing such a pirated video can lead to the identification of the video camera and the screen that produced the video, which can be used as corroborating evidence in piracy investigations

Метод производства судебной экспертизы фонограмм с использованием данных о частоте электрического тока сети

The article describes new possibilities for technical authentication of audio recordings and determination of circumstances in which they were made in the context of Russian forensic practice. The proposed method is based on comparing the frequency of the electric current in the grid reflected in the submitted audio evidence with reference electric network frequency (ENF) indicators, which are recorded on a regular basis using a specialized hardware and software suite. This method is known as the Electric Network Frequency (ENF) Criterion (developed by Romanian expert Catalin Grigoras in 2005) and has been used by forensic practitioners around the world since 2009. The paper discusses the substance of the method, its possibilities and limitations, and demonstrates its practical value in helping to establish whether an audio recording was tampered with, as well as the date and time it was made. The article is intended for experts as a summary of the method, and for investigators to familiarize themselves with the new possibilities of scientific methodology.В статье описаны новые для российской экспертной практики возможности технического определения аутентичности фонограммы и особенностей ее изготовления при производстве судебной экспертизы фонограмм. В основе предлагаемого метода – сравнение показателей частоты электрического тока сети, отраженных на исследуемой фонограмме, с эталонными показателями частоты электрического тока сети, учет которых ведется с помощью специального аппаратно-программного комплекса. Метод разработан румынским экспертом Каталином Григорасом в 2005 году. За рубежом при производстве фоноскопических экспертиз данный метод используется с 2009 года и получил название Electric Network Frequency (ENF) Criterion. Раскрыта сущность метода, его возможности и ограничения, практическая ценность для установления наличия либо отсутствия монтажа фонограммы, даты и времени ее изготовления. Статья предназначена для экспертов как краткое описание метода и следователей (дознавателей) для ознакомления с новыми возможностями экспертных методов

Multimedia Forensics

This book is open access. Media forensics has never been more relevant to societal life. Not only media content represents an ever-increasing share of the data traveling on the net and the preferred communications means for most users, it has also become integral part of most innovative applications in the digital information ecosystem that serves various sectors of society, from the entertainment, to journalism, to politics. Undoubtedly, the advances in deep learning and computational imaging contributed significantly to this outcome. The underlying technologies that drive this trend, however, also pose a profound challenge in establishing trust in what we see, hear, and read, and make media content the preferred target of malicious attacks. In this new threat landscape powered by innovative imaging technologies and sophisticated tools, based on autoencoders and generative adversarial networks, this book fills an important gap. It presents a comprehensive review of state-of-the-art forensics capabilities that relate to media attribution, integrity and authenticity verification, and counter forensics. Its content is developed to provide practitioners, researchers, photo and video enthusiasts, and students a holistic view of the field

Resiliency Assessment and Enhancement of Intrinsic Fingerprinting

Intrinsic fingerprinting is a class of digital forensic technology that can detect traces left in digital multimedia data in order to reveal data processing history and determine data integrity. Many existing intrinsic fingerprinting schemes have implicitly assumed favorable operating conditions whose validity may become uncertain in reality. In order to establish intrinsic fingerprinting as a credible approach to digital multimedia authentication, it is important to understand and enhance its resiliency under unfavorable scenarios. This dissertation addresses various resiliency aspects that can appear in a broad range of intrinsic fingerprints. The first aspect concerns intrinsic fingerprints that are designed to identify a particular component in the processing chain. Such fingerprints are potentially subject to changes due to input content variations and/or post-processing, and it is desirable to ensure their identifiability in such situations. Taking an image-based intrinsic fingerprinting technique for source camera model identification as a representative example, our investigations reveal that the fingerprints have a substantial dependency on image content. Such dependency limits the achievable identification accuracy, which is penalized by a mismatch between training and testing image content. To mitigate such a mismatch, we propose schemes to incorporate image content into training image selection and significantly improve the identification performance. We also consider the effect of post-processing against intrinsic fingerprinting, and study source camera identification based on imaging noise extracted from low-bit-rate compressed videos. While such compression reduces the fingerprint quality, we exploit different compression levels within the same video to achieve more efficient and accurate identification. The second aspect of resiliency addresses anti-forensics, namely, adversarial actions that intentionally manipulate intrinsic fingerprints. We investigate the cost-effectiveness of anti-forensic operations that counteract color interpolation identification. Our analysis pinpoints the inherent vulnerabilities of color interpolation identification, and motivates countermeasures and refined anti-forensic strategies. We also study the anti-forensics of an emerging space-time localization technique for digital recordings based on electrical network frequency analysis. Detection schemes against anti-forensic operations are devised under a mathematical framework. For both problems, game-theoretic approaches are employed to characterize the interplay between forensic analysts and adversaries and to derive optimal strategies. The third aspect regards the resilient and robust representation of intrinsic fingerprints for multiple forensic identification tasks. We propose to use the empirical frequency response as a generic type of intrinsic fingerprint that can facilitate the identification of various linear and shift-invariant (LSI) and non-LSI operations

TIME AND LOCATION FORENSICS FOR MULTIMEDIA

In the modern era, a vast quantities of digital information is available in the form of audio, image, video, and other sensor recordings. These recordings may contain metadata describing important information such as the time and the location of recording. As the stored information can be easily modified using readily available digital editing software, determining the authenticity of a recording has utmost importance, especially for critical applications such as law enforcement, journalism, and national and business intelligence. In this dissertation, we study novel environmental signatures induced by power networks, which are known as Electrical Network Frequency (ENF) signals and become embedded in multimedia data at the time of recording. ENF fluctuates slightly over time from its nominal value of 50 Hz/60 Hz. The major trend of fluctuations in the ENF remains consistent across the entire power grid, including when measured at physically distant geographical locations. We investigate the use of ENF signals for a variety of applications such as estimation/verification of time and location of a recording's creation, and develop a theoretical foundation to support ENF based forensic analysis. In the first part of the dissertation, the presence of ENF signals in visual recordings captured in electric powered lighting environments is demonstrated. The source of ENF signals in visual recordings is shown to be the invisible flickering of indoor lighting sources such as fluorescent and incandescent lamps. The techniques to extract ENF signals from recordings demonstrate that a high correlation is observed between the ENF fluctuations obtained from indoor lighting and that from the power mains supply recorded at the same time. Applications of the ENF signal analysis to tampering detection of surveillance video recordings, and forensic binding of the audio and visual track of a video are also discussed. In the following part, an analytical model is developed to gain an understanding of the behavior of ENF signals. It is demonstrated that ENF signals can be modeled using a time-varying autoregressive process. The performance of the proposed model is evaluated for a timestamp verification application. Based on this model, an improved algorithm for ENF matching between a reference signal and a query signal is provided. It is shown that the proposed approach provides an improved matching performance as compared to the case when matching is performed directly on ENF signals. Another application of the proposed model in learning the power grid characteristics is also explicated. These characteristics are learnt by using the modeling parameters as features to train a classifier to determine the creation location of a recording among candidate grid-regions. The last part of the dissertation demonstrates that differences exist between ENF signals recorded in the same grid-region at the same time. These differences can be extracted using a suitable filter mechanism and follow a relationship with the distance between different locations. Based on this observation, two localization protocols are developed to identify the location of a recording within the same grid-region, using ENF signals captured at anchor locations. Localization accuracy of the proposed protocols are then compared. Challenges in using the proposed technique to estimate the creation location of multimedia recordings within the same grid, along with efficient and resilient trilateration strategies in the presence of outliers and malicious anchors, are also discussed

Multimedia Forensics

This book is open access. Media forensics has never been more relevant to societal life. Not only media content represents an ever-increasing share of the data traveling on the net and the preferred communications means for most users, it has also become integral part of most innovative applications in the digital information ecosystem that serves various sectors of society, from the entertainment, to journalism, to politics. Undoubtedly, the advances in deep learning and computational imaging contributed significantly to this outcome. The underlying technologies that drive this trend, however, also pose a profound challenge in establishing trust in what we see, hear, and read, and make media content the preferred target of malicious attacks. In this new threat landscape powered by innovative imaging technologies and sophisticated tools, based on autoencoders and generative adversarial networks, this book fills an important gap. It presents a comprehensive review of state-of-the-art forensics capabilities that relate to media attribution, integrity and authenticity verification, and counter forensics. Its content is developed to provide practitioners, researchers, photo and video enthusiasts, and students a holistic view of the field