Exploring remote photoplethysmography signals for deepfake detection in facial videos

Abstract. With the advent of deep learning-based facial forgeries, also called "deepfakes", the feld of accurately detecting forged videos has become a quickly growing area of research. For this endeavor, remote photoplethysmography, the process of extracting biological signals such as the blood volume pulse and heart rate from facial videos, offers an interesting avenue for detecting fake videos that appear utterly authentic to the human eye. This thesis presents an end-to-end system for deepfake video classifcation using remote photoplethysmography. The minuscule facial pixel colour changes are used to extract the rPPG signal, from which various features are extracted and used to train an XGBoost classifer. The classifer is then tested using various colour-to-blood volume pulse methods (OMIT, POS, LGI and CHROM) and three feature extraction window lengths of two, four and eight seconds. The classifer was found effective at detecting deepfake videos with an accuracy of 85 %, with minimal performance difference found between the window lengths. The GREEN channel signal was found to be important for this classifcationEtäfotoplethysmografian hyödyntäminen syväväärennösten tunnistamiseen. Tiivistelmä. Syväväärennösten eli syväoppimiseen perustuvien kasvoväärennöksien yleistyessä väärennösten tarkasta tunnistamisesta koneellisesti on tullut nopeasti kasvava tutkimusalue. Etäfotoplethysmografa (rPPG) eli biologisten signaalien kuten veritilavuuspulssin tai sykkeen mittaaminen videokuvasta tarjoaa kiinnostavan keinon tunnistaa väärennöksiä, jotka vaikuttavat täysin aidoilta ihmissilmälle. Tässä diplomityössä esitellään etäfotoplethysmografaan perustuva syväväärennösten tunnistusmetodi. Kasvojen minimaalisia värimuutoksia hyväksikäyttämällä mitataan fotoplethysmografasignaali, josta lasketuilla ominaisuuksilla koulutetaan XGBoost-luokittelija. Luokittelijaa testataan usealla eri värisignaalista veritilavuussignaaliksi muuntavalla metodilla sekä kolmella eri ominaisuuksien ikkunapituudella. Luokittelija pystyy tunnistamaan väärennetyn videon aidosta 85 % tarkkuudella. Eri ikkunapituuksien välillä oli minimaalisia eroja, ja vihreän värin signaalin havaittiin olevan luokittelun suorituskyvyn kannalta merkittävä

Multi-hierarchical Convolutional Network for Efficient Remote Photoplethysmograph Signal and Heart Rate Estimation from Face Video Clips

Heart beat rhythm and heart rate (HR) are important physiological parameters of the human body. This study presents an efficient multi-hierarchical spatio-temporal convolutional network that can quickly estimate remote physiological (rPPG) signal and HR from face video clips. First, the facial color distribution characteristics are extracted using a low-level face feature Generation (LFFG) module. Then, the three-dimensional (3D) spatio-temporal stack convolution module (STSC) and multi-hierarchical feature fusion module (MHFF) are used to strengthen the spatio-temporal correlation of multi-channel features. In the MHFF, sparse optical flow is used to capture the tiny motion information of faces between frames and generate a self-adaptive region of interest (ROI) skin mask. Finally, the signal prediction module (SP) is used to extract the estimated rPPG signal. The experimental results on the three datasets show that the proposed network outperforms the state-of-the-art methods.Comment: 33 pages,9 figure

Automated Remote Pulse Oximetry System (ARPOS)

Funding: This research is funded by the School of Computer Science and by St Leonard’s Postgraduate College Doctoral Scholarship, both at the University of St Andrews for Pireh Pirzada’s PhD. Early work was funded by the Digital Health & Care Innovation Centre (DHI).Current methods of measuring heart rate (HR) and oxygen levels (SPO2) require physical contact, are individualised, and for accurate oxygen levels may also require a blood test. No-touch or non-invasive technologies are not currently commercially available for use in healthcare settings. To date, there has been no assessment of a system that measures HR and SPO2 using commercial off-the-shelf camera technology that utilises R, G, B and IR data. Moreover, no formal remote photoplethysmography studies have been done in real life scenarios with participants at home with different demographic characteristics. This novel study addresses all these objectives by developing, optimising, and evaluating a system that measures the HR and SPO2 of 40 participants. HR and SPO2 are determined by measuring the frequencies from different wavelength band regions using FFT and radiometric measurements after pre-processing face regions of interest (forehead, lips, and cheeks) from Colour, IR and Depth data. Detrending, interpolating, hamming, and normalising the signal with FastICA produced the lowest RMSE of 7.8 for HR with the r-correlation value of 0.85 and RMSE 2.3 for SPO2. This novel system could be used in several critical care settings, including in care homes and in hospitals and prompt clinical intervention as required.Publisher PDFPeer reviewe

Aplicación de captura de datasets y demostración de algoritmos de detección de ritmo cardíaco mediante análisis de secuencias de vídeo

El objetivo de este trabajo es desarrollar una aplicación que permita la generación de datasets de vídeos y probar diferentes algoritmos de detección del ritmo cardíaco mediante el análisis de secuencias de vídeo. Para la creación de los datasets la aplicación integra un dispositivo de bluetooth con el cual poder registrar y guardar el ritmo cardíaco real del usuario al que se le está grabando (groud-truth). La principal razón por la cual se ha realizado este trabajo ha sido debido a la importancia que tiene el bombeo de la sangre por parte del corazón a las diferentes partes del cuerpo, y a que gracias a estos métodos de cálculo del ritmo cardíaco mediante el análisis de secuencias de vídeo son mucho menos intrusivos. Con esta aplicación vamos a conseguir probar de manera mucho más sencilla los diferentes algoritmos que se vayan creando en este ámbito. Basándonos en las necesidades se ha desarrollado una aplicación íntegramente creada en C#, lenguaje elegido principalmente para poder hacer uso de las funcionalidades aportadas por la API de Windows para la Kinect v2, cámara con la que se van a grabar los diferentes vídeos. Con este lenguaje además hemos podido incorporar de manera sencilla una interfaz de usuario para Windows, XAML, que nos facilita la conexión entre las diferentes vistas y la parte lógica de la aplicación. Para medir el ritmo cardíaco real se va a hacer uso de un dispositivo de Bluetooth de baja energía (BLE) con la ayuda de los protocolos existentes de Bluetooth que nos permiten conectarlo de manera programática y en concreto de las últimas librerías añadidas por Microsoft en la API de UWP (Universal Windows Platform) para dispositivos Bluetooth. Como el objetivo principal de la aplicación es que permita que se puedan probar algoritmos nuevos se ha decidido integrar un algoritmo que hace uso de secuencias de vídeo para calcular el ritmo cardíaco para poder probar esta funcionalidad. El algoritmo que se ha integrado es un algoritmo sencillo que calcula el ritmo cardíaco mediante los cambios en la luz producidos por el bombeo de la sangre. Por último, se ha creado un pequeño dataset de vídeos en color y de imágenes en profundidad que nos permite probar algoritmos nuevos sin necesidad de tener que generar nuevos vídeos

Mη επεμβατική εξαγωγή σήματος PPG και ρυθμού καρδιάς με χρήση video

Στην διπλωματική εργασία αυτή παρουσιάζεται η δυνατότητα αξιόπιστης μη επεμβατικής μέτρησης φυσιολογικών παραμέτρων, όπως είναι ο ρυθμός καρδιάς. Αυτό καθίσταται εφικτό μετά από την επεξεργασία του εξ αποστάσεως λαμβανόμενου σήματος της φωτοπληθυσμογραφίας (PhotoPlethysmoGraphy-PPG), μέσω video στο οποίο απεικονίζεται συγκεκριμένη περιοχή του προσώπου. Στα πλαίσια της εργασίας αυτής πραγματοποιήθηκε βιβλιογραφική έρευνα όσον αφορά την τεχνολογία της φωτοπληθυσμογραφίας και τις σύγχρονες εξελίξεις στην προσπάθεια μη επεμβατικής εφαρμογής αυτής. Μελετήθηκαν επίσης οι αλγόριθμοι εντοπισμού και ανίχνευσης προσώπου και οι αλγόριθμοι μείωσης διάστασης οι οποίοι αποτελούν βασικά εργαλεία για την εφαρμογή μη επεμβατικής φωτοπληθυσμογραφίας. Ακολούθως, πραγματοποιήθηκε η συλλογή των δεδομένων, όπου και έλαβε μέρος ένα σύνολο 62 συμμετεχόντων. Κατόπιν, σχεδιάστηκε και υλοποιήθηκε πολυ-επίπεδη αλγοριθμική διαδικασία για την εξαγωγή της επιθυμητής παραμέτρου του ρυθμού καρδιάς, η οποία σε αντίθεση με τις έως τώρα προσπάθειες, έλαβε υπόψη την ύπαρξη μεταβλητών συνθηκών φωτισμού. Από τα αποτελέσματα που παρουσιάζονται, πιστοποιήθηκε η δυνατότητα μέτρησης ζωτικών παραμέτρων με υψηλή ακρίβεια, εξ αποστάσεως και χωρίς την απαίτηση κάποιας ιδιαίτερης συμμετοχής από την πλευρά του ασθενή, σε περιβάλλοντα με μεταβλητές συνθήκες φωτισμού και σε αποστάσεις έως ενός μέτρου του συμμετέχοντα από την συμβατική web-κάμερα.This thesis investigates the potential non-invasive measurement of physiological parameters such as the heart rate. These parameters derive from the remotely acquired Photoplethysmography (PPG) signal through video depicting a specific region of interest of the patient’s face. In the framework of this thesis, a thorough literature review of Photoplethysmography as well as a study of the recent advances and its unobtrusive application was carried out. Main principles of face detection and dimensionality reduction algorithms which constitute basic tools of these non-invasive methods are presented. A dataset collection procedure with the participation of 42 patients took place and is described in detail. A novel multi-level algorithm for the estimation of the heart rate is proposed, differing from the already existed methods in taking into account the variability of the light conditions. Our results show that unobtrusive remote measurement of vital signs of high accuracy is feasible even in variable environments in terms of the light conditions and for distances between the patient’s face and the webcam of up to 1 meter

Methods for acquisition and integration of personal wellness parameters

Wellness indicates the state or condition of being in good physical and mental health. Stress is a common state of emotional strain that plays a crucial role in the everyday quality of life. Nowadays, there is a growing individual awareness of the importance of a proper lifestyle and a generalized trend to become an active part in monitoring, preserving, and improving personal wellness for both physical and emotional aspects. The majority studies in this field relies on the evaluation of the changes of sensed parameters passing from rest to “maximal” stress. However, the vast majority of people usually experiences stressing circumstances in everyday life. This led us to investigate the impact of mild cognitive activation which can be somehow comparable to usual situations that everyone can face in daily life. Several signals and data can be useful to characterize the state of a person, but not all of them are equally important. So it is crucial to analyse the mutual relevance of the different pieces of information. In this work we focus on a subset of well-established psychophysical descriptors and we identified a set of devices enabling the measurement of these parameters . The design of the experimental setup and the selection of sensing devices were driven by qualitative criteria such as intrusiveness, reliability, and ease of use. These are deemed crucial for implementing effective (self-)monitoring strategies. A reference dataset, named “Mild Cognitive Activation” (MCA), was collected. The last aim of the project was the definition of a quantitative model for data integration providing a concise description of the wellness status of a person. This process was based on unsupervised learning paradigms. Data from MCA were integrated with data from the “Stress Recognition in Automobile Drivers” dataset . This allowed a cross validation of the integration methodology

Remote Assessment of the Cardiovascular Function Using Camera-Based Photoplethysmography

Camera-based photoplethysmography (cbPPG) is a novel measurement technique that allows the continuous monitoring of vital signs by using common video cameras. In the last decade, the technology has attracted a lot of attention as it is easy to set up, operates remotely, and offers new diagnostic opportunities. Despite the growing interest, cbPPG is not completely established yet and is still primarily the object of research. There are a variety of reasons for this lack of development including that reliable and autonomous hardware setups are missing, that robust processing algorithms are needed, that application fields are still limited, and that it is not completely understood which physiological factors impact the captured signal. In this thesis, these issues will be addressed. A new and innovative measuring system for cbPPG was developed. In the course of three large studies conducted in clinical and non-clinical environments, the system’s great flexibility, autonomy, user-friendliness, and integrability could be successfully proven. Furthermore, it was investigated what value optical polarization filtration adds to cbPPG. The results show that a perpendicular filter setting can significantly enhance the signal quality. In addition, the performed analyses were used to draw conclusions about the origin of cbPPG signals: Blood volume changes are most likely the defining element for the signal's modulation. Besides the hardware-related topics, the software topic was addressed. A new method for the selection of regions of interest (ROIs) in cbPPG videos was developed. Choosing valid ROIs is one of the most important steps in the processing chain of cbPPG software. The new method has the advantage of being fully automated, more independent, and universally applicable. Moreover, it suppresses ballistocardiographic artifacts by utilizing a level-set-based approach. The suitability of the ROI selection method was demonstrated on a large and challenging data set. In the last part of the work, a potentially new application field for cbPPG was explored. It was investigated how cbPPG can be used to assess autonomic reactions of the nervous system at the cutaneous vasculature. The results show that changes in the vasomotor tone, i.e. vasodilation and vasoconstriction, reflect in the pulsation strength of cbPPG signals. These characteristics also shed more light on the origin problem. Similar to the polarization analyses, they support the classic blood volume theory. In conclusion, this thesis tackles relevant issues regarding the application of cbPPG. The proposed solutions pave the way for cbPPG to become an established and widely accepted technology