Illumination variation-resistant network for heart rate measurement by exploring RGB and MSR spaces

Abstract Remote photoplethysmography (rPPG) is an essential way of monitoring the physiological indicator heart rate (HR), which has important guiding significance for preventing and controlling cardiovascular diseases. However, most existing HR measurement approaches require ideal illumination conditions, and the illumination variation in a realistic situation is complicated. In view of this issue, this article proposes a robust HR measurement method to reduce performance degradation due to unstable illumination in facial videos. Specifically, two complementary color spaces [RGB and multiscale retinex (MSR)] are abundantly utilized by exploring the potential of space-shared information and space-specific characteristics. Subsequently, the time-space Transformer with sequential feature aggregation (TST-SFA) is exploited to extract physiological signal features. In addition, a novel optimization strategy for model learning, including affinity variation, discrepancy, and task losses, is proposed to train the whole algorithm in an end-to-end manner jointly. Experimental results on three public datasets show that our proposed method outperforms other approaches and can achieve more accurate HR measurement under different illuminations. The code will be released at https://github.com/Llili314/IRHrNet.Abstract Remote photoplethysmography (rPPG) is an essential way of monitoring the physiological indicator heart rate (HR), which has important guiding significance for preventing and controlling cardiovascular diseases. However, most existing HR measurement approaches require ideal illumination conditions, and the illumination variation in a realistic situation is complicated. In view of this issue, this article proposes a robust HR measurement method to reduce performance degradation due to unstable illumination in facial videos. Specifically, two complementary color spaces [RGB and multiscale retinex (MSR)] are abundantly utilized by exploring the potential of space-shared information and space-specific characteristics. Subsequently, the time-space Transformer with sequential feature aggregation (TST-SFA) is exploited to extract physiological signal features. In addition, a novel optimization strategy for model learning, including affinity variation, discrepancy, and task losses, is proposed to train the whole algorithm in an end-to-end manner jointly. Experimental results on three public datasets show that our proposed method outperforms other approaches and can achieve more accurate HR measurement under different illuminations. The code will be released at https://github.com/Llili314/IRHrNet

Information-enhanced Network for Noncontact Heart Rate Estimation from Facial Videos

Abstract Remote photoplethysmography (rPPG) is a vital way of measuring heart rate (HR) to reflect human physical and mental health, which is useful for diagnosing cardiovascular and neurological diseases. Many non-contact HR estimation methods have been proposed gradually in recent years, but the majority of approaches are based on a single-modal HR information source, resulting in ineffective and unsatisfactory estimation results due to noise and insufficient information. This paper proposes a novel information-enhanced network for HR estimation based on multimodal (e.g., RGB and NIR) sources to address these problems. In the network, context and modal difference information are sequentially enhanced from spatiotemporal and modal views for accurately describing HR-aware features, while maximum frequency information is enhanced for inhibiting heartbeat noise. Specifically, a context-enhanced video Swin-Transformer (CET) module is exploited to extract useful rPPG signal features from facial visible-light and near-infrared videos. Then, a novel modal difference enhanced fusion (MDEF) module is designed to acquire a fused rPPG signal, which is taken as the input of the frequency-enhanced estimation (FEE) module to obtain the corresponding HR value. These three modules are integrated and jointly learned in an end-to-end way, and the multimodal combinations can provide highly complementary information for estimating HR value. Experimental and evaluation results on three multimodal datasets show that the proposed model achieves a superior effect compared to the state-of-the-art methods.Abstract Remote photoplethysmography (rPPG) is a vital way of measuring heart rate (HR) to reflect human physical and mental health, which is useful for diagnosing cardiovascular and neurological diseases. Many non-contact HR estimation methods have been proposed gradually in recent years, but the majority of approaches are based on a single-modal HR information source, resulting in ineffective and unsatisfactory estimation results due to noise and insufficient information. This paper proposes a novel information-enhanced network for HR estimation based on multimodal (e.g., RGB and NIR) sources to address these problems. In the network, context and modal difference information are sequentially enhanced from spatiotemporal and modal views for accurately describing HR-aware features, while maximum frequency information is enhanced for inhibiting heartbeat noise. Specifically, a context-enhanced video Swin-Transformer (CET) module is exploited to extract useful rPPG signal features from facial visible-light and near-infrared videos. Then, a novel modal difference enhanced fusion (MDEF) module is designed to acquire a fused rPPG signal, which is taken as the input of the frequency-enhanced estimation (FEE) module to obtain the corresponding HR value. These three modules are integrated and jointly learned in an end-to-end way, and the multimodal combinations can provide highly complementary information for estimating HR value. Experimental and evaluation results on three multimodal datasets show that the proposed model achieves a superior effect compared to the state-of-the-art methods

Heart rate estimation by leveraging static and dynamic region weights

Abstract As the demand for long-term health evaluation grows, researchers show increased interest in remote photoplethysmography studies. However, conventional methods are vulnerable to noise interference caused by non-rigid facial movements (facial expression, talking, etc.). Consequently, avoiding these interferences and improving the remote photoplethysmography (rPPG) signal quality become important tasks during heart rate (HR) estimation. We propose an approach that extracts high-quality rPPG signals from various subregions of the face by fusing static and dynamic weights and then employs the convolutional neural network to estimate HR value by converting the 1D rPPG signal into 2D time-frequency analysis maps. Specifically, chrominance features from various regions of interest are used to generate the raw subregion rPPG signal set that is further utilized to estimate the static weights of different regions through a clustering method. Additionally, a measurement method called enclosed area distance is proposed to perform static weights estimation. The dynamic weights of different regions are calculated using the 3D-gradient descriptor to eliminate motion interference, which evaluates the inactivation degree under regional movement situations. The final rPPG signal is reconstructed by combining the rPPG signals from the different subregions using the static and dynamic weights. The experiments are conducted on two widely used public datasets, i.e., MAHNOB-HCI and PURE. The results demonstrate that the proposed method achieves 3.12 MAE and 3.78 SD on MAHNOB-HCI and the best r on the PURE, which significantly outperforms state-of-the-art methods.Abstract As the demand for long-term health evaluation grows, researchers show increased interest in remote photoplethysmography studies. However, conventional methods are vulnerable to noise interference caused by non-rigid facial movements (facial expression, talking, etc.). Consequently, avoiding these interferences and improving the remote photoplethysmography (rPPG) signal quality become important tasks during heart rate (HR) estimation. We propose an approach that extracts high-quality rPPG signals from various subregions of the face by fusing static and dynamic weights and then employs the convolutional neural network to estimate HR value by converting the 1D rPPG signal into 2D time-frequency analysis maps. Specifically, chrominance features from various regions of interest are used to generate the raw subregion rPPG signal set that is further utilized to estimate the static weights of different regions through a clustering method. Additionally, a measurement method called enclosed area distance is proposed to perform static weights estimation. The dynamic weights of different regions are calculated using the 3D-gradient descriptor to eliminate motion interference, which evaluates the inactivation degree under regional movement situations. The final rPPG signal is reconstructed by combining the rPPG signals from the different subregions using the static and dynamic weights. The experiments are conducted on two widely used public datasets, i.e., MAHNOB-HCI and PURE. The results demonstrate that the proposed method achieves 3.12 MAE and 3.78 SD on MAHNOB-HCI and the best r on the PURE, which significantly outperforms state-of-the-art methods

A New Self-Powered Sensor Using the Radial Field Piezoelectric Diaphragm in d ₃₃ Mode for Detecting Underwater Disturbances

This paper presents a new sensor based on a radial field bulk piezoelectric diaphragm to provide energy-efficient and high-performance situational sensing for autonomous underwater vehicles (AUVs). This sensor is self-powered, does not need an external power supply, and works efficiently in d33 mode by using inter-circulating electrodes to release the radial in-plane poling. Finite element analysis was conducted to estimate the sensor behavior. Sensor prototypes were fabricated by microfabrication technology. The dynamic behaviors of the piezoelectric diaphragm were examined by the impedance spectrum. By imitating the underwater disturbance and generating the oscillatory flow velocities with a vibrating sphere, the performance of the sensor in detecting the oscillatory flow was tested. Experimental results show that the sensitivity of the sensor is up to 1.16 mV/(mm/s), and the detectable oscillatory flow velocity is as low as 4 mm/s. Further, this sensor can work well under a disturbance with low frequency. The present work provides a good application prospect for the underwater sensing of AUVs

A Two-Modal Weather Classification Method and Its Application in Photovoltaic Power Probability Prediction

Weather classification is an indispensable preprocessing step in photovoltaic (PV) power prediction. A new two-modal weather classification methods based on PV power clustering was proposed to finely depict the uncertainty of PV power output. Both PV power data and meteorological data were considered for weather classification, providing a novel and effective path for PV power prediction. In addition, data fusion technology was used to extract relevant information from both numeric weather prediction (NWP) data and measured meteorological data to help for weather classification. This approach reduces the model’s reliance on the accuracy of forecasted meteorological indicators and improve the robustness of the model. Experiments based on data from a PV power station in Jilin demonstrated the rationality of the proposed weather classification method. Combining the PV power probability prediction with the proposed weather classifier resulted in prediction interval coverage probabilities closer to the preassigned confidence level and narrower mean prediction interval width