Cardiovascular assessment by imaging photoplethysmography – a review

AbstractOver the last few years, the contactless acquisition of cardiovascular parameters using cameras has gained immense attention. The technique provides an optical means to acquire cardiovascular information in a very convenient way. This review provides an overview on the technique’s background and current realizations. Besides giving detailed information on the most widespread application of the technique, namely the contactless acquisition of heart rate, we outline further concepts and we critically discuss the current state.</jats:p

Design and development of optical reflectance spectroscopy and optical coherence tomography catheters for myocardial tissue characterization

Catheter ablation therapy attempts to restore sinus rhythm in arrhythmia patients by producing site-specific tissue modification along regions which cause abnormal electrical activity. This treatment, though widely used, often requires repeat procedures to observe long-term therapeutic benefits. This limitation is driven in part by challenges faced by conventional schemes in validating lesion adequacy at the time of the procedure. Optical techniques are well-suited for the interrogation and characterization of biological tissues. In particular, optical coherence tomography (OCT) relies on coherence gating of singly-scattered light to enable high-resolution structural imaging for tissue diagnostics and procedural guidance. Alternatively, optical reflectance spectroscopy (ORS) is a point measurement technique which makes use of incoherent, multiply-scattered light to probe tissue volumes and derive important data from its optical signature. ORS relies on the fact that light-tissue interactions are regulated by absorption and scattering, which directly relate to the intrinsic tissue biochemistry and cellular organization. In this thesis, we explore the integration of these modalities into ablation catheters for obtaining procedural metrics which could be utilized to guide catheter ablation therapy. We first present the development of an accelerated computational light transport model and its application for guiding ORS catheter design. A custom ORS-integrated ablation catheter is then implemented and tested within porcine specimens in vitro. A model is proposed for real-time estimation of lesion size based on changes in spectral morphology acquired during ablation. We then fabricated custom integrated OCT M-mode RF catheters and present a model for detecting contact status based on deep convolutional neural networks trained on endomyocardial images. Additionally, we demonstrate for the first time, tracking of RF-induced lesion formation employing OCT Doppler micro-velocimetry; this response is shown to be commensurate with the degree of treatment. We further demonstrate for the first time spectroscopic tracking of kinetics related to the heme oxidation cascade during thermal treatment, which are linked to tissue denaturation. The pairing of these modalities into a single RF catheter was also validated for guiding lesion delivery in vitro and within live pigs. Finally, we conclude with a proof-of-concept demonstration of ORS as a mapping tool to guide epicardial ablation in human donor hearts. These results showcase the vast potential of ORS and OCT empowered RF catheters for aiding intraprocedural guidance of catheter ablation procedures which could be utilized alongside current practices

Remote Assessment of the Cardiovascular Function Using Camera-Based Photoplethysmography

Camera-based photoplethysmography (cbPPG) is a novel measurement technique that allows the continuous monitoring of vital signs by using common video cameras. In the last decade, the technology has attracted a lot of attention as it is easy to set up, operates remotely, and offers new diagnostic opportunities. Despite the growing interest, cbPPG is not completely established yet and is still primarily the object of research. There are a variety of reasons for this lack of development including that reliable and autonomous hardware setups are missing, that robust processing algorithms are needed, that application fields are still limited, and that it is not completely understood which physiological factors impact the captured signal. In this thesis, these issues will be addressed. A new and innovative measuring system for cbPPG was developed. In the course of three large studies conducted in clinical and non-clinical environments, the system’s great flexibility, autonomy, user-friendliness, and integrability could be successfully proven. Furthermore, it was investigated what value optical polarization filtration adds to cbPPG. The results show that a perpendicular filter setting can significantly enhance the signal quality. In addition, the performed analyses were used to draw conclusions about the origin of cbPPG signals: Blood volume changes are most likely the defining element for the signal's modulation. Besides the hardware-related topics, the software topic was addressed. A new method for the selection of regions of interest (ROIs) in cbPPG videos was developed. Choosing valid ROIs is one of the most important steps in the processing chain of cbPPG software. The new method has the advantage of being fully automated, more independent, and universally applicable. Moreover, it suppresses ballistocardiographic artifacts by utilizing a level-set-based approach. The suitability of the ROI selection method was demonstrated on a large and challenging data set. In the last part of the work, a potentially new application field for cbPPG was explored. It was investigated how cbPPG can be used to assess autonomic reactions of the nervous system at the cutaneous vasculature. The results show that changes in the vasomotor tone, i.e. vasodilation and vasoconstriction, reflect in the pulsation strength of cbPPG signals. These characteristics also shed more light on the origin problem. Similar to the polarization analyses, they support the classic blood volume theory. In conclusion, this thesis tackles relevant issues regarding the application of cbPPG. The proposed solutions pave the way for cbPPG to become an established and widely accepted technology

Widefield Computational Biophotonic Imaging for Spatiotemporal Cardiovascular Hemodynamic Monitoring

Cardiovascular disease is the leading cause of mortality, resulting in 17.3 million deaths per year globally. Although cardiovascular disease accounts for approximately 30% of deaths in the United States, many deleterious events can be mitigated or prevented if detected and treated early. Indeed, early intervention and healthier behaviour adoption can reduce the relative risk of first heart attacks by up to 80% compared to those who do not adopt new healthy behaviours. Cardiovascular monitoring is a vital component of disease detection, mitigation, and treatment. The cardiovascular system is an incredibly dynamic system that constantly adapts to internal and external stimuli. Monitoring cardiovascular function and response is vital for disease detection and monitoring. Biophotonic technologies provide unique solutions for cardiovascular assessment and monitoring in naturalistic and clinical settings. These technologies leverage the properties of light as it enters and interacts with the tissue, providing safe and rapid sensing that can be performed in many different environments. Light entering into human tissue undergoes a complex series of absorption and scattering events according to both the illumination and tissue properties. The field of quantitative biomedical optics seeks to quantify physiological processes by analysing the remitted light characteristics relative to the controlled illumination source. Drawing inspiration from contact-based biophotonic sensing technologies such as pulse oximetry and near infrared spectroscopy, we explored the feasibility of widefield hemodynamic assessment using computational biophotonic imaging. Specifically, we investigated the hypothesis that computational biophotonic imaging can assess spatial and temporal properties of pulsatile blood flow across large tissue regions. This thesis presents the design, development, and evaluation of a novel photoplethysmographic imaging system for assessing spatial and temporal hemodynamics in major pulsatile vasculature through the sensing and processing of subtle light intensity fluctuations arising from local changes in blood volume. This system co-integrates methods from biomedical optics, electronic control, and biomedical image and signal processing to enable non-contact widefield hemodynamic assessment over large tissue regions. A biophotonic optical model was developed to quantitatively assess transient blood volume changes in a manner that does not require a priori information about the tissue's absorption and scattering characteristics. A novel automatic blood pulse waveform extraction method was developed to encourage passive monitoring. This spectral-spatial pixel fusion method uses physiological hemodynamic priors to guide a probabilistic framework for learning pixel weights across the scene. Pixels are combined according to their signal weight, resulting in a single waveform. Widefield hemodynamic imaging was assessed in three biomedical applications using the aforementioned developed system. First, spatial vascular distribution was investigated across a sample with highly varying demographics for assessing common pulsatile vascular pathways. Second, non-contact biophotonic assessment of the jugular venous pulse waveform was assessed, demonstrating clinically important information about cardiac contractility function in a manner which is currently assessed through invasive catheterization. Lastly, non-contact biophotonic assessment of cardiac arrhythmia was demonstrated, leveraging the system's ability to extract strong hemodynamic signals for assessing subtle fluctuations in the waveform. This research demonstrates that this novel approach for computational biophotonic hemodynamic imaging offers new cardiovascular monitoring and assessment techniques, which can enable new scientific discoveries and clinical detection related to cardiovascular function

Camera-Based Heart Rate Extraction in Noisy Environments

Remote photoplethysmography (rPPG) is a non-invasive technique that benefits from video to measure vital signs such as the heart rate (HR). In rPPG estimation, noise can introduce artifacts that distort rPPG signal and jeopardize accurate HR measurement. Considering that most rPPG studies occurred in lab-controlled environments, the issue of noise in realistic conditions remains open. This thesis aims to examine the challenges of noise in rPPG estimation in realistic scenarios, specifically investigating the effect of noise arising from illumination variation and motion artifacts on the predicted rPPG HR. To mitigate the impact of noise, a modular rPPG measurement framework, comprising data preprocessing, region of interest, signal extraction, preparation, processing, and HR extraction is developed. The proposed pipeline is tested on the LGI-PPGI-Face-Video-Database public dataset, hosting four different candidates and real-life scenarios. In the RoI module, raw rPPG signals were extracted from the dataset using three machine learning-based face detectors, namely Haarcascade, Dlib, and MediaPipe, in parallel. Subsequently, the collected signals underwent preprocessing, independent component analysis, denoising, and frequency domain conversion for peak detection. Overall, the Dlib face detector leads to the most successful HR for the majority of scenarios. In 50% of all scenarios and candidates, the average predicted HR for Dlib is either in line or very close to the average reference HR. The extracted HRs from the Haarcascade and MediaPipe architectures make up 31.25% and 18.75% of plausible results, respectively. The analysis highlighted the importance of fixated facial landmarks in collecting quality raw data and reducing noise

Imaging photoplethysmography: towards effective physiological measurements

Since its conception decades ago, Photoplethysmography (PPG) the non-invasive opto-electronic technique that measures arterial pulsations in-vivo has proven its worth by achieving and maintaining its rank as a compulsory standard of patient monitoring. However successful, conventional contact monitoring mode is not suitable in certain clinical and biomedical situations, e.g., in the case of skin damage, or when unconstrained movement is required. With the advance of computer and photonics technologies, there has been a resurgence of interest in PPG and one potential route to overcome the abovementioned issues has been increasingly explored, i.e., imaging photoplethysmography (iPPG). The emerging field of iPPG offers some nascent opportunities in effective and comprehensive interpretation of the physiological phenomena, indicating a promising alternative to conventional PPG. Heart and respiration rate, perfusion mapping, and pulse rate variability have been accessed using iPPG. To effectively and remotely access physiological information through this emerging technique, a number of key issues are still to be addressed. The engineering issues of iPPG, particularly the influence of motion artefacts on signal quality, are addressed in this thesis, where an engineering model based on the revised Beer-Lambert law was developed and used to describe opto-physiological phenomena relevant to iPPG. An iPPG setup consisting of both hardware and software elements was developed to investigate its reliability and reproducibility in the context of effective remote physiological assessment. Specifically, a first study was conducted for the acquisition of vital physiological signs under various exercise conditions, i.e. resting, light and heavy cardiovascular exercise, in ten healthy subjects. The physiological parameters derived from the images captured by the iPPG system exhibited functional characteristics comparable to conventional contact PPG, i.e., maximum heart rate difference was <3 bpm and a significant (p < 0.05) correlation between both measurements were also revealed. Using a method for attenuation of motion artefacts, the heart rate and respiration rate information was successfully assessed from different anatomical locations even in high-intensity physical exercise situations. This study thereby leads to a new avenue for noncontact sensing of vital signs and remote physiological assessment, showing clear and promising applications in clinical triage and sports training. A second study was conducted to remotely assess pulse rate variability (PRV), which has been considered a valuable indicator of autonomic nervous system (ANS) status. The PRV information was obtained using the iPPG setup to appraise the ANS in ten normal subjects. The performance of the iPPG system in accessing PRV was evaluated via comparison with the readings from a contact PPG sensor. Strong correlation and good agreement between these two techniques verify the effectiveness of iPPG in the remote monitoring of PRV, thereby promoting iPPG as a potential alternative to the interpretation of physiological dynamics related to the ANS. The outcomes revealed in the thesis could present the trend of a robust non-contact technique for cardiovascular monitoring and evaluation

Tomographic measurement of all orthogonal components of three-dimensional displacement fields within scattering materials using wavelength scanning interferometry

Experimental mechanics is currently contemplating tremendous opportunities of further advancements thanks to a combination of powerful computational techniques and also fullfield non-contact methods to measure displacement and strain fields in a wide variety of materials. Identification techniques, aimed to evaluate material mechanical properties given known loads and measured displacement or strain fields, are bound to benefit from increased data availability (both in density and dimensionality) and efficient inversion methods such as finite element updating (FEU) and the virtual fields method (VFM). They work at their best when provided with dense and multicomponent experimental displacement (or strain) data, i.e. when all orthogonal components of displacements (or all components of the strain tensor) are known at points closely spaced within the volume of the material under study. Although a very challenging requirement, an increasing number of techniques are emerging to provide such data. In this Thesis, a novel wavelength scanning interferometry (WSI) system that provides three dimensional (3-D) displacement fields inside the volume of semi-transparent scattering materials is proposed. Sequences of two-dimensional interferograms are recorded whilst tuning the frequency of a laser at a constant rate. A new approach based on frequency multiplexing is used to encode the interference signal corresponding to multiple illumination directions at different spectral bands. Different optical paths along each illumination direction ensure that the signals corresponding to each sensitivity vector do not overlap in the frequency domain. All the information required to reconstruct the location and the 3-D displacement vector of scattering points within the material is thus recorded simultaneously in a single wavelength scan. By comparing phase data volumes obtained for two successive scans, all orthogonal components of the three dimensional displacement field introduced between scans (e.g. by means of loading or moving the sample under study) are readily obtained with high displacement sensitivity. The fundamental principle that describes the technique is presented in detail, including the correspondence between interference signal frequency and its associated depth within the sample, depth range, depth resolution, transverse resolution and displacement sensitivity. Data processing of the interference signal includes Fourier transformation, noise reduction, re-registration of data volumes, measurement of the illumination and sensitivity vectors from experimental data using a datum surface, phase difference evaluation, 3-D phase unwrapping and 3-D displacement field evaluation. Experiments consisting of controlled rigid body rotations and translations of a phantom were performed to validate the results. Both in-plane and the out-of-plane displacement components were measured for each voxel in the resulting data volume, showing an excellent agreement with the expected 3-D displacement

Modular Instrumentation for Controlling and Monitoring In-Vitro Cultivation Environment and Image-based Functionality Measurements of Human Stem Cells

Artificial animal cell culture was successfully developed by Ross Harrison in 1907. But it was not until the 1940’s and 1950’s that several developments occurred, which expedited the cell culturing in-vitro (C-Vitro) to be a consistent and reproducible technique to study isolated living-cells in a controlled environment. Currently, CVitro is one of the major tools in cellular and molecular biology both in the academia and industry. They are extensively utilised to study the cellular physiology/biochemistry, to screen drugs/therapeutic compounds, to understand the effects of drugs/toxic compounds and also to identify the pathways of carcinogenesis/mutagenesis. It is also used in large scale manufacturing of vaccines and therapeutic proteins. In any experimental setup, it is important that the C-Vitro model should represent the physiological phenomena of interest with reasonable accuracy so that all experimental results are statistically consistent and reproducible. In this direction, sensors and measurement systems play important roles in in-situ detection and/or control/manipulation of cells/tissues/environment. This thesis aimed to develop new technology for tailored cell culturing and integrated measurements. Firstly, design and assembly of a portable Invert-upright microscope interchangeable modular cell culturing platform (iuCMP) was envisioned. In contrast to conventional methods, micro-scaled systems mimic the cells' natural microenvironment more precisely, facilitating accurate and tractable models. The iuCMP integrates modular measurement schemes with a mini culture chamber using biocompatible cell-friendly materials, automated environment-control (temperature and gas concentrations), oxygen sensing and simultaneous functional measurements (electrophysiological and image-based). Time lapse microscopy is very useful in cell biology, but integration of advanced >i>in-vitro/device based biological systems (e.g. lab/organ/body-on-chips, or mini-bioreactors/microfluidic systems) into conventional microscopes can be challenging in several circumstances due to multiple reasons. But in iuCMP the main advantage is, the microscope can be switched either as an inverted or as an upright system and therefore can accommodate virtually any in-vitro device. It can capture images from regions that are otherwise inaccessible by conventional microscopes, for example, cells cultured on physical or biochemical sensor systems. The modular design also allows accommodating more sensor or measurement systems quite freely. We have demonstrated the system for video-based beating analysis of cardiomyocytes, cell orientation analysis on nanocellulose, and simultaneous long-term in-situ microscopy with oxygen and temperature sensing in hypoxia. In an example application, the system was utilised for long-term temperature stressing and simultaneous mechanobiological analysis of human induced pluripotent stem cell-derived cardiomyocytes (hiPSC-CMs). For this the iuCMP together with a temperature sensor plate (TSP) and a novel non-invasive beating analysis software (CMaN—cardiomyocyte function analysis tool, scripted as a subpart of this thesis), was applied for automated temperature response studies in hiPSC-CM cultures. In-situ temperature sensing is usually challenging with bulky external sensors, but TSPs solved this issue. In the temperature response study, we showed that the relationship between hiPSC-CM beating frequency and temperature is non-linear and measured the Q10 temperature coefficients. Moreover, we observed the hiPSC-CM contractile networking, including propagation of the action potential signal between dissociated clusters and their non-invasive measurements. It was the first case where these events were reported in hiPSC-CM clusters and their noninvasive measurements by image processing. The software CMaN comes with a user-friendly interface and, is equipped with features for batch processing, movement centre detection and cluster finding. It can extract six different signals of the contractile motion of cardiomyocytes (clusters or single cells) per processing. This ensures a minimum of one useful beating signal even in the cases of complex beating videos. On the processing end, compared to similar tools, CMaN is faster, more sensitive, and computationally less expensive and allows ROI based processing. In the case of healthy cells, the waveform of the signal from the CMaN resembles an ECG signal with positive and negative segments, allowing the computation of contraction and relaxation features separately. In addition to iuCMP, a Modular optical pH measurement system (MO-pH) for 24/7 non-contact cell culture measurements was also developed. The MO-pH incorporates modular sterilisable optical parts and is used in phenol-red medium cell cultures. The modular assembly of MO-pH cassettes is unique and reusable. Measurements are carried out in a closed flow system without wasting any culture medium and requires no special manual attention or recalibrations during culture. Furthermore, a new absorption correction model was put forward that minimised errors caused e.g. by biolayers in spectrometric pH measurement, which improved the pH measurement accuracy. MO-pH has been applied in long-term human adipose stem cells (hASC) expansion cultures in CO2 dependent and independent media. Additionally, the MO-pH was also utilised to comprehend the behaviour of pH, temperature and humidity in water jacked incubators as well as to record the pH response as a function of temperature in the presence and absence of CO2 in the context of stem cell cultures. The resulting plots clearly showed the interplay between measured parameters indicating a few stress sources present all through the culture. Additionally, it provided an overall picture of behaviour of critical control parameters in an incubator and pointed out the need for bioprocess systems with automatic process monitoring and smart control for maximum yield, optimal growth and maintenance of the cells. Besides, we also integrated MO-pH into flasks with reclosable lids (RL-F) and tested its applicability in stem cell cultures. A standalone system around an RL-F flask was built by combining the cell culture, medium perfusion and optical measurements. The developed RL-F system has been successfully tested in ASC-differentiation cultures. Finally, a few trial experiments for image-based pH estimation aimed for iuCMP have also been carried out. This includes tests with LCD illumination, optical projection tomography, and webcam systems. In reality, the pH is not distributed uniformly in tissues, and has shown a gradient of up to 1.0 pH unit within 1 cm distance. Therefore, producing reliable pH maps also in in-vitro can be important in understanding various common pathologies and location of lesions. A reliable and adequately developed long-term pH mapping method will be an important addition into the iuCMP

Extraction of Heart Rate from Multimodal Video Streams of Neonates using Methods of Machine Learning

The World Health Organization estimates that more than one-tenth of births are premature. Premature births are linked to an increase of the mortality risk, when compared with full-term infants. In fact, preterm birth complications are the leading cause of perinatal mortality. These complications range from respiratory distress to cardiovascular disorders. Vital signs changes are often prior to these major complications, therefore it is crucial to perform continuous monitoring of this signals. Heart rate monitoring is particularly important. Nowadays, the standard method to monitor this vital sign requires adhesive electrodes or sensors that are attached to the infant. This contact-based methods can damage the skin of the infant, possibly leading to infections. Within this context, there is a need to evolve to remote heart rate monitoring methods. This thesis introduces a new method for region of interest selection to improve remote heart rate monitoring in neonatology through Photoplethysmography Imaging. The heart rate assessment is based on the standard photoplethysmography principle, which makes use of the subtle fluctuations of visible or infrared light that is reflected from the skin surface within the cardiac cycle. A camera is used, instead of the contact-based sensors. Specifically, this thesis presents an alternative method to manual region of interest selection using methods of Machine Learning, aiming to improve the robustness of Photoplethysmography Imaging. This method comprises a highly efficient Fully Convolutional Neural Network to select six different body regions, within each video frame. The developed neural network was built upon a ResNet network and a custom upsampling network. Additionally, a new post-processing method was developed to refine the body segmentation results, using a sequence of morphological operations and centre of mass analysis. The developed region of interest selection method was validated with clinical data, demonstrating a good agreement (78%) between the estimated heart rate and the reference

Proceedings of the 2011 Joint Workshop of Fraunhofer IOSB and Institute for Anthropomatics, Vision and Fusion Laboratory

This book is a collection of 15 reviewed technical reports summarizing the presentations at the 2011 Joint Workshop of Fraunhofer IOSB and Institute for Anthropomatics, Vision and Fusion Laboratory. The covered topics include image processing, optical signal processing, visual inspection, pattern recognition and classification, human-machine interaction, world and situation modeling, autonomous system localization and mapping, information fusion, and trust propagation in sensor networks