Exploring remote photoplethysmography signals for deepfake detection in facial videos

Abstract. With the advent of deep learning-based facial forgeries, also called "deepfakes", the feld of accurately detecting forged videos has become a quickly growing area of research. For this endeavor, remote photoplethysmography, the process of extracting biological signals such as the blood volume pulse and heart rate from facial videos, offers an interesting avenue for detecting fake videos that appear utterly authentic to the human eye. This thesis presents an end-to-end system for deepfake video classifcation using remote photoplethysmography. The minuscule facial pixel colour changes are used to extract the rPPG signal, from which various features are extracted and used to train an XGBoost classifer. The classifer is then tested using various colour-to-blood volume pulse methods (OMIT, POS, LGI and CHROM) and three feature extraction window lengths of two, four and eight seconds. The classifer was found effective at detecting deepfake videos with an accuracy of 85 %, with minimal performance difference found between the window lengths. The GREEN channel signal was found to be important for this classifcationEtäfotoplethysmografian hyödyntäminen syväväärennösten tunnistamiseen. Tiivistelmä. Syväväärennösten eli syväoppimiseen perustuvien kasvoväärennöksien yleistyessä väärennösten tarkasta tunnistamisesta koneellisesti on tullut nopeasti kasvava tutkimusalue. Etäfotoplethysmografa (rPPG) eli biologisten signaalien kuten veritilavuuspulssin tai sykkeen mittaaminen videokuvasta tarjoaa kiinnostavan keinon tunnistaa väärennöksiä, jotka vaikuttavat täysin aidoilta ihmissilmälle. Tässä diplomityössä esitellään etäfotoplethysmografaan perustuva syväväärennösten tunnistusmetodi. Kasvojen minimaalisia värimuutoksia hyväksikäyttämällä mitataan fotoplethysmografasignaali, josta lasketuilla ominaisuuksilla koulutetaan XGBoost-luokittelija. Luokittelijaa testataan usealla eri värisignaalista veritilavuussignaaliksi muuntavalla metodilla sekä kolmella eri ominaisuuksien ikkunapituudella. Luokittelija pystyy tunnistamaan väärennetyn videon aidosta 85 % tarkkuudella. Eri ikkunapituuksien välillä oli minimaalisia eroja, ja vihreän värin signaalin havaittiin olevan luokittelun suorituskyvyn kannalta merkittävä

A Comparative Evaluation of Heart Rate Estimation Methods using Face Videos

This paper presents a comparative evaluation of methods for remote heart rate estimation using face videos, i.e., given a video sequence of the face as input, methods to process it to obtain a robust estimation of the subjects heart rate at each moment. Four alternatives from the literature are tested, three based in hand crafted approaches and one based on deep learning. The methods are compared using RGB videos from the COHFACE database. Experiments show that the learning-based method achieves much better accuracy than the hand crafted ones. The low error rate achieved by the learning based model makes possible its application in real scenarios, e.g. in medical or sports environments.Comment: Accepted in "IEEE International Workshop on Medical Computing (MediComp) 2020

Respiratory Rate Estimation from Face Videos

Vital signs, such as heart rate (HR), heart rate variability (HRV), respiratory rate (RR), are important indicators for a person's health. Vital signs are traditionally measured with contact sensors, and may be inconvenient and cause discomfort during continuous monitoring. Commercial cameras are promising contact-free sensors, and remote photoplethysmography (rPPG) have been studied to remotely monitor heart rate from face videos. For remote RR measurement, most prior art was based on small periodical motions of chest regions caused by breathing cycles, which are vulnerable to subjects' voluntary movements. This paper explores remote RR measurement based on rPPG obtained from face videos. The paper employs motion compensation, two-phase temporal filtering, and signal pruning to capture signals with high quality. The experimental results demonstrate that the proposed framework can obtain accurate RR results and can provide HR, HRV and RR measurement synergistically in one framework

Dual-path TokenLearner for Remote Photoplethysmography-based Physiological Measurement with Facial Videos

Remote photoplethysmography (rPPG) based physiological measurement is an emerging yet crucial vision task, whose challenge lies in exploring accurate rPPG prediction from facial videos accompanied by noises of illumination variations, facial occlusions, head movements, \etc, in a non-contact manner. Existing mainstream CNN-based models make efforts to detect physiological signals by capturing subtle color changes in facial regions of interest (ROI) caused by heartbeats. However, such models are constrained by the limited local spatial or temporal receptive fields in the neural units. Unlike them, a native Transformer-based framework called Dual-path TokenLearner (Dual-TL) is proposed in this paper, which utilizes the concept of learnable tokens to integrate both spatial and temporal informative contexts from the global perspective of the video. Specifically, the proposed Dual-TL uses a Spatial TokenLearner (S-TL) to explore associations in different facial ROIs, which promises the rPPG prediction far away from noisy ROI disturbances. Complementarily, a Temporal TokenLearner (T-TL) is designed to infer the quasi-periodic pattern of heartbeats, which eliminates temporal disturbances such as head movements. The two TokenLearners, S-TL and T-TL, are executed in a dual-path mode. This enables the model to reduce noise disturbances for final rPPG signal prediction. Extensive experiments on four physiological measurement benchmark datasets are conducted. The Dual-TL achieves state-of-the-art performances in both intra- and cross-dataset testings, demonstrating its immense potential as a basic backbone for rPPG measurement. The source code is available at \href{https://github.com/VUT-HFUT/Dual-TL}{https://github.com/VUT-HFUT/Dual-TL

Robust contactless pulse transit time estimation based on signal quality metric

The pulse transit time (PTT) can provide valuable insight into cardiovascular health, specifically regarding arterial stiffness and blood pressure. Traditionally, PTT is derived by calculating the time difference between two photoplethysmography (PPG) measurements, which require a set of body-worn sensors attached to the skin. Recently, remote photoplethysmography (rPPG) has been proposed as a contactless monitoring alternative. The main problem with rPPG based PTT estimation is that motion artifacts affect the shape of waveform leading to the shift or over-detected peaks, which decreases the accuracy of PTT. To overcome this problem, this paper presents a robust pulse-by-pulse PTT estimation framework using a signal quality metric. By exploiting the local temporal information and global periodic characteristics, the metric automatically assesses pulse quality of signal on a pulse-by-pulse basis, and calculates the probabilities of the pulse peak being the actual peak. Furthermore, in order to cope with over-detected and shift pulse peaks, Kalman filter complemented by the proposed signal quality metric is used to adaptively adjust the peaks based on the estimated probability. All the refined peaks are finally used for pulse-by-pulse PTT estimation. The experiment results are promising, suggesting that the proposed framework provides a robust and more accurate PTT estimation in real applications