Remote Human Vital Sign Monitoring Using Multiple-Input Multiple-Output Radar at Millimeter-Wave Frequencies

Non-contact respiration rate (RR) and heart rate (HR) monitoring using millimeter-wave (mmWave) radars has gained lots of attention for medical, civilian, and military applications. These mmWave radars are small, light, and portable which can be deployed to various places. To increase the accuracy of RR and HR detection, distributed multi-input multi-output (MIMO) radar can be used to acquire non-redundant information of vital sign signals from different perspectives because each MIMO channel has different fields of view with respect to the subject under test (SUT). This dissertation investigates the use of a Frequency Modulated Continuous Wave (FMCW) radar operating at 77-81 GHz for this application. Vital sign signal is first reconstructed with Arctangent Demodulation (AD) method using phase change’s information collected by the radar due to chest wall displacement from respiration and heartbeat activities. Since the heartbeat signals can be corrupted and concealed by the third/fourth harmonics of the respiratory signals as well as random body motion (RBM) from the SUT, we have developed an automatic Heartbeat Template (HBT) extraction method based on Constellation Diagrams of the received signals. The extraction method will automatically spot and extract signals’ portions that carry good amount of heartbeat signals which are not corrupted by the RBM. The extracted HBT is then used as an adapted wavelet for Continuous Wavelet Transform (CWT) to reduce interferences from respiratory harmonics and RBM, as well as magnify the heartbeat signals. As the nature of RBM is unpredictable, the extracted HBT may not completely cancel the interferences from RBM. Therefore, to provide better HR detection’s accuracy, we have also developed a spectral-based HR selection method to gather frequency spectra of heartbeat signals from different MIMO channels. Based on this gathered spectral information, we can determine an accurate HR even if the heartbeat signals are significantly concealed by the RBM. To further improve the detection’s accuracy of RR and HR, two deep learning (DL) frameworks are also investigated. First, a Convolutional Neural Network (CNN) has been proposed to optimally select clean MIMO channels and eliminate MIMO channels with low SNR of heartbeat signals. After that, a Multi-layer Perceptron (MLP) neural network (NN) is utilized to reconstruct the heartbeat signals that will be used to assess and select the final HR with high confidence

Guest Editorial Cardiovascular Health Informatics: Risk Screening and Intervention

Despite enormous efforts to prevent cardiovascular disease (CVD) in the past, it remains the leading cause of death in most countries worldwide. Around two-thirds of these deaths are due to acute events, which frequently occur suddenly and are often fatal beforemedical care can be given. New strategies for screening and early intervening CVD, in addition to the conventional methods, are therefore needed in order to provide personalized and pervasive healthcare. In this special issue, selected emerging technologies in health informatics for screening and intervening CVDs are reported. These papers include reviews or original contributions on 1) new potential genetic biomarkers for screening CVD outcomes and high-throughput techniques for mining genomic data; 2) new imaging techniques for obtaining faster and higher resolution images of cardiovascular imaging biomarkers such as the cardiac chambers and atherosclerotic plaques in coronary arteries, as well as possible automatic segmentation, identification, or fusion algorithms; 3) new physiological biomarkers and novel wearable and home healthcare technologies for monitoring them in daily lives; 4) new personalized prediction models of plaque formation and progression or CVD outcomes; and 5) quantifiable indices and wearable systems to measure them for early intervention of CVD through lifestyle changes. It is hoped that the proposed technologies and systems covered in this special issue can result in improved CVD management and treatment at the point of need, offering a better quality of life to the patient

Contactless Electrocardiogram Monitoring with Millimeter Wave Radar

The electrocardiogram (ECG) has always been an important biomedical test to diagnose cardiovascular diseases. Current approaches for ECG monitoring are based on body attached electrodes leading to uncomfortable user experience. Therefore, contactless ECG monitoring has drawn tremendous attention, which however remains unsolved. In fact, cardiac electrical-mechanical activities are coupling in a well-coordinated pattern. In this paper, we achieve contactless ECG monitoring by breaking the boundary between the cardiac mechanical and electrical activity. Specifically, we develop a millimeter-wave radar system to contactlessly measure cardiac mechanical activity and reconstruct ECG without any contact in. To measure the cardiac mechanical activity comprehensively, we propose a series of signal processing algorithms to extract 4D cardiac motions from radio frequency (RF) signals. Furthermore, we design a deep neural network to solve the cardiac related domain transformation problem and achieve end-to-end reconstruction mapping from RF input to the ECG output. The experimental results show that our contactless ECG measurements achieve timing accuracy of cardiac electrical events with median error below 14ms and morphology accuracy with median Pearson-Correlation of 90% and median Root-Mean-Square-Error of 0.081mv compared to the groudtruth ECG. These results indicate that the system enables the potential of contactless, continuous and accurate ECG monitoring

NON-CONTACT TECHNIQUES FOR HUMAN VITAL SIGN DETECTION AND GAIT ANALYSIS

Human vital signs including respiratory rate, heart rate, oxygen saturation, blood pressure, and body temperature are important physiological parameters that are used to track and monitor human health condition. Another important biological parameter of human health is human gait. Human vital sign detection and gait investigations have been attracted many scientists and practitioners in various fields such as sport medicine, geriatric medicine, bio-mechanic and bio-medical engineering and has many biological and medical applications such as diagnosis of health issues and abnormalities, elderly care and health monitoring, athlete performance analysis, and treatment of joint problems. Thoroughly tracking and understanding the normal motion of human limb joints can help to accurately monitor human subjects or patients over time to provide early flags of possible complications in order to aid in a proper diagnosis and development of future comprehensive treatment plans. With the spread of COVID-19 around the world, it has been getting more important than ever to employ technology that enables us to detect human vital signs in a non-contact way and helps protect both patients and healthcare providers from potentially life-threatening viruses, and have the potential to also provide a convenient way to monitor people health condition, remotely. A popular technique to extract biological parameters from a distance is to use cameras. Radar systems are another attractive solution for non-contact human vital signs monitoring and gait investigation that track and monitor these biological parameters without invading people privacy. The goal of this research is to develop non-contact methods that is capable of extracting human vital sign parameters and gait features accurately. To do that, in this work, optical systems including cameras and proper filters have been developed to extract human respiratory rate, heart rate, and oxygen saturation. Feasibility of blood pressure extraction using the developed optical technique has been investigated, too. Moreover, a wideband and low-cost radar system has been implemented to detect single or multiple human subject’s respiration and heart rate in dark or from behind the wall. The performance of the implemented radar system has been enhanced and it has been utilized for non-contact human gait analysis. Along with the hardware, advanced signal processing schemes have been enhanced and applied to the data collected using the aforementioned radar system. The data processing algorithms have been extended for multi-subject scenarios with high accuracy for both human vital sign detection and gait analysis. In addition, different configurations of this and high-performance radar system including mono-static and MIMO have been designed and implemented with great success. Many sets of exhaustive experiments have been conducted using different human subjects and various situations and accurate reference sensors have been used to validate the performance of the developed systems and algorithms

Bio-Radar Applications for Remote Vital Signs Monitoring

Nowadays, most vital signs monitoring techniques used in a medical context and/or daily life routines require direct contact with skin, which can become uncomfortable or even impractical to be used regularly. Radar technology has been appointed as one of the most promising contactless tools to overcome these hurdles. However, there is a lack of studies that cover a comprehensive assessment of this technology when applied in real-world environments. This dissertation aims to study radar technology for remote vital signs monitoring, more specifically, in respiratory and heartbeat sensing. Two off-the-shelf radars, based on impulse radio ultra-wideband and frequency modu lated continuous wave technology, were customized to be used in a small proof of concept experiment with 10 healthy participants. Each subject was monitored with both radars at three different distances for two distinct conditions: breathing and voluntary apnea. Signals processing algorithms were developed to detect and estimate respiratory and heartbeat parameters, assessed using qualitative and quantitative methods. Concerning respiration, a minimum error of 1.6% was found when radar respiratory peaks signals were directly compared with their reference, whereas a minimum mean absolute error of 0.3 RPM was obtained for the respiration rate. Concerning heartbeats, their expression in radar signals was not as clear as the respiration ones, however a minimum mean absolute error of 1.8 BPM for heartbeat was achieved after applying a novel selective algorithm developed to validate if heart rate value was estimated with reliability. The results proved the potential for radars to be used in respiratory and heartbeat contactless sensing, showing that the employed methods can be already used in some mo tionless situations. Notwithstanding, further work is required to improve the developed algorithms in order to obtain more robust and accurate systems.Atualmente, a maioria das técnicas usadas para a monitorização de sinais vitais em contexto médicos e/ou diário requer contacto direto com a pele, o que poderá tornar-se incómodo ou até mesmo inviável em certas situações. A tecnologia radar tem vindo a ser apontada como uma das mais promissoras ferramentas para medição de sinais vitais à distância e sem contacto. Todavia, são necessários mais estudos que permitam avaliar esta tecnologia quando aplicada a situações mais reais. Esta dissertação tem como objetivo o estudo da tecnologia radar aplicada no contexto de medição remota de sinais vitais, mais concretamente, na medição de atividade respiratória e cardíaca. Dois aparelhos radar, baseados em tecnologia banda ultra larga por rádio de impulso e em tecnologia de onda continua modulada por frequência, foram configurados e usados numa prova de conceito com 10 participantes. Cada sujeito foi monitorizado com cada um dos radar em duas situações distintas: respirando e em apneia voluntária. Algorit mos de processamento de sinal foram desenvolvidos para detetar e estimar parâmetros respiratórios e cardíacos, avaliados através de métodos qualitativos e quantitativos. Em relação à respiração, o menor erro obtido foi de 1,6% quando os sinais de radar respiratórios foram comparados diretamente com os sinais de referência, enquanto que, um erro médio absoluto mínimo de 0,3 RPM foi obtido para a estimação da frequência respiratória via radar. A expressão cardíaca nos sinais radar não se revelou tão evidente como a respiratória, no entanto, um erro médio absoluto mínimo de 1,8 BPM foi obtido para a estimação da frequência cardíaca após a aplicação de um novo algoritmo seletivo, desenvolvido para validar a confiança dos valores obtidos. Os resultados obtidos provaram o potencial do uso de radares na medição de atividade respiratória e cardíaca sem contacto, sendo esta tecnologia viável de ser implementada em situações onde não existe muito movimento. Não obstante, os algoritmos desenvolvidos devem ser aperfeiçoados no futuro de forma a obter sistemas mais robustos e precisos

Remote Assessment of the Cardiovascular Function Using Camera-Based Photoplethysmography

Camera-based photoplethysmography (cbPPG) is a novel measurement technique that allows the continuous monitoring of vital signs by using common video cameras. In the last decade, the technology has attracted a lot of attention as it is easy to set up, operates remotely, and offers new diagnostic opportunities. Despite the growing interest, cbPPG is not completely established yet and is still primarily the object of research. There are a variety of reasons for this lack of development including that reliable and autonomous hardware setups are missing, that robust processing algorithms are needed, that application fields are still limited, and that it is not completely understood which physiological factors impact the captured signal. In this thesis, these issues will be addressed. A new and innovative measuring system for cbPPG was developed. In the course of three large studies conducted in clinical and non-clinical environments, the system’s great flexibility, autonomy, user-friendliness, and integrability could be successfully proven. Furthermore, it was investigated what value optical polarization filtration adds to cbPPG. The results show that a perpendicular filter setting can significantly enhance the signal quality. In addition, the performed analyses were used to draw conclusions about the origin of cbPPG signals: Blood volume changes are most likely the defining element for the signal's modulation. Besides the hardware-related topics, the software topic was addressed. A new method for the selection of regions of interest (ROIs) in cbPPG videos was developed. Choosing valid ROIs is one of the most important steps in the processing chain of cbPPG software. The new method has the advantage of being fully automated, more independent, and universally applicable. Moreover, it suppresses ballistocardiographic artifacts by utilizing a level-set-based approach. The suitability of the ROI selection method was demonstrated on a large and challenging data set. In the last part of the work, a potentially new application field for cbPPG was explored. It was investigated how cbPPG can be used to assess autonomic reactions of the nervous system at the cutaneous vasculature. The results show that changes in the vasomotor tone, i.e. vasodilation and vasoconstriction, reflect in the pulsation strength of cbPPG signals. These characteristics also shed more light on the origin problem. Similar to the polarization analyses, they support the classic blood volume theory. In conclusion, this thesis tackles relevant issues regarding the application of cbPPG. The proposed solutions pave the way for cbPPG to become an established and widely accepted technology

Methods for Doppler Radar Monitoring of Physiological Signals

Unobtrusive health monitoring includes advantages such as long-term monitoring of rarely occurring conditions or of slow changes in health, at reasonable costs. In addition, the preparation of electrodes or other sensors is not needed. Currently, the main limitation of remote patient monitoring is not in the existing communication infrastructure but the lack of reliable, easy-to-use, and well-studied sensors.The aim of this thesis was to develop methods for monitoring cardiac and respiratory activity with microwave continuous wave (CW) Doppler radar. When considering cardiac and respiration monitoring, the heart and respiration rates are often the first monitored parameters. The motivation of this thesis, however, is to measure not only rate-related parameters but also the cardiac and respiratory waveforms, including the chest wall displacement information.This dissertation thoroughly explores the signal processing methods for accurate chest wall displacement measurement with a radar sensor. The sensor prototype and measurement setup choices are reported. The contributions of this dissertation encompass an I/Q imbalance estimation method and a nonlinear demodulation method for a quadrature radar sensor. Unlike the previous imbalance estimation methods, the proposed method does not require the use of laboratory equipment. The proposed nonlinear demodulation method, on the other hand, is shown to be more accurate than other methods in low-noise cases. In addition, the separation of the cardiac and respiratory components with independent component analysis (ICA) is discussed. The developed methods were validated with simulations and with simplified measurement setups in an office environment. The performance of the nonlinear demodulation method was also studied with three patients for sleep-time respiration monitoring. This is the first time that whole-night measurements have been analyzed with the method in an uncontrolled environment. Data synchronization between the radar sensor and a commercial polysomnographic (PSG) device was assured with a developed infrared (IR) link, which is reported as a side result.The developed methods enable the extraction of more useful information from a radar sensor and extend its application. This brings Doppler radar sensors one step closer to large-scale commercial use for a wide range of applications, including home health monitoring, sleep-time respiration monitoring, and measuring gating signals for medical imaging

Laser doppler vibrometry for cardiovascular monitoring and photoacoustic imaging

Nowadays, techniques for health monitoring mainly require physical contact with patients, which is not always ideal. Non-contact health monitoring has become an important research topic in the last decades. The non-contact detection of a patient's health condition represents a beneficial tool in different biomedical fields. Examples can be found in intensive care, home health care, the nursing of the elderly, the monitoring of physical efforts, and in human-machine interactions. Cardiovascular diseases (CV) are one of the most spread causes of death in developed countries. Their monitoring techniques involve physical contact with patients. A non-contact technique for cardiovascular monitoring could overcome problems related to the contact with the patient such as skin lesions. It could also expand the availability of monitoring to those cases where contact is not possible or should be avoided to reduce the exposure of medical personnel to biochemical hazard conditions.Several research groups have investigated different techniques for non-contact monitoring of health; among them, the laser Doppler Vibrometry (LDVy) has one of the highest accuracies and signal to noise ratios for cardiorespiratory signals detection. Moreover, the simplicity of data processing, the long-distance measurement range, and the high bandwidth make the laser Doppler vibrometer (LDV) suitable for daily measurements. LDVy is an interferometric technique employed for the measurements of displacement or velocity signals in various fields. In particular, it is deployed in the biomedical field for the extraction of several cardiovascular parameters, such as the PR-time. Generally, the extraction of these parameters requires ideal measuring conditions (measuring spot and laser direction), which are not realistic for daily monitoring in non-laboratory conditions, and especially in tracking applications. The first scientific hypothesis of this work is that the PR-time detected with LDV has an acceptable uncertainty for a realistic variety of measurement spot positions and angles of the incident laser beam. Therefore, I investigated the uncertainty contribution to the detection of the PR-time from LDV signals resulting from the laser beam direction and from the measurement point position; these investigations were carried out with a multipoint laser Doppler vibrometer. The uncertainties were evaluated according to the Guide to the Expression of Uncertainty in Measurement. Successively, the ranges of PR-time values where it is possible to state with 95% certainty that a diagnosis is correct are identified. Normal values of PR-time are included in the range 120 ms -200 ms. For single value measurements with precise alignment the reliable range for the detection of the healthy condition is 146.4 ms -173.6 ms. The detection of CV diseases is reliable for measured values lower than 93.6 ms and greater than 226.4 ms. For mean value measurements with precise alignment the reliable range for the detection of the healthy condition is 126.6 ms -193.4 ms. The detection of CV diseases is reliable for measured values lower than 113.4 ms and greater than 206.6 ms. Therefore, for measured values included in the mentioned ranges, the detection of the PR-time and relative diagnosis with the LDVy in non-laboratory conditions is reliable. The method for the estimation of the uncertainty contribution proposed in this work can be applied to other cardiovascular parameters extracted with the LDVy. Recently, the LDVy was employed for the detection of tumors in tissue-mimic phantoms as a noncontact alternative to the ultrasound sensors employed in photoacoustic imaging (PAI). A non-contact method has considerable advantages for photoacustic imaging, too. Several works present the possibility to perform PAI measurements with LDVy. However, a successful detection of the signals generated by a tumor depends on the metrological characteristics of the LDV, on the properties of the tumor and of the tissue. The conditions under which a tumor is detectable with the laser Doppler vibrometer has not been investigated yet. The second scientific hypothesis of this work is that, under certain conditions, photoacoustic imaging measurements with LDVy are feasible. Therefore, I identified those conditions to determine the detection limits of LDVy for PAI measurements. These limits were deduced by considering the metrological characteristics of a commercial LDV, the dimensions and the position of the tumor in the tissue. I derived a model for the generation and propagation of PA signals and its detection with an LDV. The model was validated by performing experiments on silicone tissue-micking phantoms. The validated model with breast-tissue parameters reveals the limits of tumor detection with LDVy-based PAI. The results show that commercial LDVs can detect tumors with a minimal radius of ≈350 μm reliably if they are located at a maximal depth in tissue of ≈2 cm. Depending on the position of the detection point, the maximal depth can diminish and depending on the absorption characteristics of the tumor, the detection range increases.Heutzutage erfordern Techniken zur Gesundheitsüberwachung hauptsächlich den physischen Kontakt mit dem Patienten, was nicht immer ideal ist. Die berührungslose Gesundheitsüberwachung hat sich in den letzten Jahrzehnten zu einem wichtigen Forschungsthema entwickelt. Die berührungslose Erkennung des Gesundheitszustands eines Patienten stellt ein nützliches Instrument in verschiedenen biomedizinischen Bereichen dar. Beispiele finden sich in der Intensivpflege, der häuslichen Krankenpflege, der Altenpflege, der Überwachung körperlicher Anstrengungen und in der MenschMaschine-Interaktion. Herz-Kreislauf-Erkrankungen sind eine der am weitesten verbreiteten Todesursachen in den Industrieländern. Ihre Überwachungstechniken erfordern einen physischen Kontakt mit den Patienten. Eine berührungslose Technik für die Überwachung von Herz-KreislaufErkrankungen könnte Probleme im Zusammenhang mit dem Kontakt mit dem Patienten, wie z. B. Hautverletzungen, überwinden. Verschiedene Messgeräte wurden für die berührungslose Überwachung der Gesundheit untersucht; unter ihnen hat das Laser-Doppler-Vibrometrer (LDV) eine der höchsten Genauigkeiten und Signal-Rausch-Verhältnisse für die Erkennung kardiorespiratorischer Signale. Darüber hinaus ist das Laser-Doppler-Vibrometer (LDV) aufgrund der einfachen Datenverarbeitung, des großen Messbereichs und der hohen Bandbreite für tägliche Messungen geeignet. LDV ist ein interferometrisches Verfahren, das zur Messung von Weg- oder Geschwindigkeitssignalen in verschiedenen Bereichen eingesetzt wird. Insbesondere wird es im biomedizinischen Bereich für die Extraktion verschiedener kardiovaskulärer Parameter, wie z. B. der PR-Zeit, eingesetzt. Im Allgemeinen erfordert die Extraktion dieser Parameter ideale Messbedingungen (Messfleck und Laserrichtung), die für die tägliche Überwachung unter Nicht-Laborbedingungen und insbesondere für TrackingAnwendungen nicht realistisch sind. Die erste wissenschaftliche Hypothese dieser Arbeit ist, dass die mit dem LDV ermittelte PR-Zeit eine akzeptable Unsicherheit für eine realistische Vielzahl von Messpunktpositionen und Winkeln des einfallenden Laserstrahls aufweist. Daher wurde der Unsicherheitsbeitrag zur Ermittlung der PR-Zeit aus LDV-Signalen untersucht, der sich aus der Laserstrahlrichtung und der Messpunktposition ergibt; diese Untersuchungen wurden mit einem Mehrpunkt-Laser-Doppler-Vibrometer durchgeführt. Die Unsicherheiten wurden gemäß der Technische Regel ISO/IEC Guide 98-3:2008-09 Messunsicherheit – Teil 3: Leitfaden zur Angabe der Unsicherheit beim Messen bewertet. Nacheinander werden die Bereiche der PR-Zeit-Werte ermittelt, in denen mit 95%iger Sicherheit eine korrekte Diagnose gestellt werden kann. Die in dieser Arbeit vorgeschlagene Methode zur Schätzung des Unsicherheitsbeitrags kann auch auf andere kardiovaskuläre Parameter angewendet werden, die mit dem LDV extrahiert werden. Kürzlich wurde das LDV zur Erkennung von Tumoren in gewebeähnlichen Phantomen als berührungslose Alternative zu den Ultraschallsensoren eingesetzt, die bei der photoakustischen Bildgebung (PAI) verwendet werden. Eine berührungslose Methode hat auch für die photoakustische Bildgebung erhebliche Vorteile. In mehreren Arbeiten wird die Möglichkeit vorgestellt, PAIMessungen mit LDV durchzuführen. Die erfolgreiche Erkennung der von einem Tumor erzeugten Signale hängt jedoch von den messtechnischen Eigenschaften des LDV sowie von den Eigenschaften des Tumors und des Gewebes ab. Die Bedingungen, unter denen ein Tumor mit dem LDV detektierbar ist, wurden bisher nicht untersucht. Die zweite wissenschaftliche Hypothese dieser Arbeit ist, dass unter bestimmten Bedingungen photoakustische Bildgebungsmessungen mit dem LDV möglich sind. Daher wurden diese Bedingungen ermittelt, um die Nachweisgrenzen von LDV für PAI-Messungen zu bestimmen. Diese Grenzen wurden unter Berücksichtigung der messtechnischen Eigenschaften eines handelsüblichen LDV, der Abmessungen und der Position des Tumors im Gewebe abgeleitet. In dieser Arbeit wurde ein Modell für die Erzeugung und Ausbreitung von PA-Signalen und deren Nachweis mit einem LDV abgeleitet. Das Modell wurde durch Experimente an Silikongewebe-Phantomen validiert. Das validierte Modell mit Parametern des Brustgewebes zeigt die Grenzen der Tumorerkennung mit LDV-basierter PAI auf. Die Ergebnisse zeigen, dass kommerzielle LDV Tumore mit einem minimalen Radius von ≈350 μm zuverlässig erkennen können

State of the art of audio- and video based solutions for AAL

Working Group 3. Audio- and Video-based AAL ApplicationsIt is a matter of fact that Europe is facing more and more crucial challenges regarding health and social care due to the demographic change and the current economic context. The recent COVID-19 pandemic has stressed this situation even further, thus highlighting the need for taking action. Active and Assisted Living (AAL) technologies come as a viable approach to help facing these challenges, thanks to the high potential they have in enabling remote care and support. Broadly speaking, AAL can be referred to as the use of innovative and advanced Information and Communication Technologies to create supportive, inclusive and empowering applications and environments that enable older, impaired or frail people to live independently and stay active longer in society. AAL capitalizes on the growing pervasiveness and effectiveness of sensing and computing facilities to supply the persons in need with smart assistance, by responding to their necessities of autonomy, independence, comfort, security and safety. The application scenarios addressed by AAL are complex, due to the inherent heterogeneity of the end-user population, their living arrangements, and their physical conditions or impairment. Despite aiming at diverse goals, AAL systems should share some common characteristics. They are designed to provide support in daily life in an invisible, unobtrusive and user-friendly manner. Moreover, they are conceived to be intelligent, to be able to learn and adapt to the requirements and requests of the assisted people, and to synchronise with their specific needs. Nevertheless, to ensure the uptake of AAL in society, potential users must be willing to use AAL applications and to integrate them in their daily environments and lives. In this respect, video- and audio-based AAL applications have several advantages, in terms of unobtrusiveness and information richness. Indeed, cameras and microphones are far less obtrusive with respect to the hindrance other wearable sensors may cause to one’s activities. In addition, a single camera placed in a room can record most of the activities performed in the room, thus replacing many other non-visual sensors. Currently, video-based applications are effective in recognising and monitoring the activities, the movements, and the overall conditions of the assisted individuals as well as to assess their vital parameters (e.g., heart rate, respiratory rate). Similarly, audio sensors have the potential to become one of the most important modalities for interaction with AAL systems, as they can have a large range of sensing, do not require physical presence at a particular location and are physically intangible. Moreover, relevant information about individuals’ activities and health status can derive from processing audio signals (e.g., speech recordings). Nevertheless, as the other side of the coin, cameras and microphones are often perceived as the most intrusive technologies from the viewpoint of the privacy of the monitored individuals. This is due to the richness of the information these technologies convey and the intimate setting where they may be deployed. Solutions able to ensure privacy preservation by context and by design, as well as to ensure high legal and ethical standards are in high demand. After the review of the current state of play and the discussion in GoodBrother, we may claim that the first solutions in this direction are starting to appear in the literature. A multidisciplinary 4 debate among experts and stakeholders is paving the way towards AAL ensuring ergonomics, usability, acceptance and privacy preservation. The DIANA, PAAL, and VisuAAL projects are examples of this fresh approach. This report provides the reader with a review of the most recent advances in audio- and video-based monitoring technologies for AAL. It has been drafted as a collective effort of WG3 to supply an introduction to AAL, its evolution over time and its main functional and technological underpinnings. In this respect, the report contributes to the field with the outline of a new generation of ethical-aware AAL technologies and a proposal for a novel comprehensive taxonomy of AAL systems and applications. Moreover, the report allows non-technical readers to gather an overview of the main components of an AAL system and how these function and interact with the end-users. The report illustrates the state of the art of the most successful AAL applications and functions based on audio and video data, namely (i) lifelogging and self-monitoring, (ii) remote monitoring of vital signs, (iii) emotional state recognition, (iv) food intake monitoring, activity and behaviour recognition, (v) activity and personal assistance, (vi) gesture recognition, (vii) fall detection and prevention, (viii) mobility assessment and frailty recognition, and (ix) cognitive and motor rehabilitation. For these application scenarios, the report illustrates the state of play in terms of scientific advances, available products and research project. The open challenges are also highlighted. The report ends with an overview of the challenges, the hindrances and the opportunities posed by the uptake in real world settings of AAL technologies. In this respect, the report illustrates the current procedural and technological approaches to cope with acceptability, usability and trust in the AAL technology, by surveying strategies and approaches to co-design, to privacy preservation in video and audio data, to transparency and explainability in data processing, and to data transmission and communication. User acceptance and ethical considerations are also debated. Finally, the potentials coming from the silver economy are overviewed.publishedVersio