Measuring frailty and detecting falls for elderly home care using depth camera

International audienceThis work concerns the development of low-cost am- bient systems for helping elderly to stay at home. Depth cameras allow a real-time analysis of the dis- placement of the person. We show that it is possible to recognize the activity of the person and to measure gait parameters from the analysis of simple features extracted from depth images. Activity recognition is based on Hidden Markov Models and performs fall detection. When a person is walking, the analysis of the trajectory of her centre of mass allows to mea- sure gait parameters that can then be used for frailty evaluation. We show that the proposed models are robust enough for activity classification, and that gait parameters measurement is accurate. We believe that such a system could be installed in the home of the elderly, while respecting privacy, since it relies on a local processing of depth images. Our system would be able to provide daily information on the person’s activity, the evolution of her gait parameters, and her habits, information that is useful for securing her and evaluating her frailty

State of the art of audio- and video based solutions for AAL

Working Group 3. Audio- and Video-based AAL ApplicationsIt is a matter of fact that Europe is facing more and more crucial challenges regarding health and social care due to the demographic change and the current economic context. The recent COVID-19 pandemic has stressed this situation even further, thus highlighting the need for taking action. Active and Assisted Living (AAL) technologies come as a viable approach to help facing these challenges, thanks to the high potential they have in enabling remote care and support. Broadly speaking, AAL can be referred to as the use of innovative and advanced Information and Communication Technologies to create supportive, inclusive and empowering applications and environments that enable older, impaired or frail people to live independently and stay active longer in society. AAL capitalizes on the growing pervasiveness and effectiveness of sensing and computing facilities to supply the persons in need with smart assistance, by responding to their necessities of autonomy, independence, comfort, security and safety. The application scenarios addressed by AAL are complex, due to the inherent heterogeneity of the end-user population, their living arrangements, and their physical conditions or impairment. Despite aiming at diverse goals, AAL systems should share some common characteristics. They are designed to provide support in daily life in an invisible, unobtrusive and user-friendly manner. Moreover, they are conceived to be intelligent, to be able to learn and adapt to the requirements and requests of the assisted people, and to synchronise with their specific needs. Nevertheless, to ensure the uptake of AAL in society, potential users must be willing to use AAL applications and to integrate them in their daily environments and lives. In this respect, video- and audio-based AAL applications have several advantages, in terms of unobtrusiveness and information richness. Indeed, cameras and microphones are far less obtrusive with respect to the hindrance other wearable sensors may cause to one’s activities. In addition, a single camera placed in a room can record most of the activities performed in the room, thus replacing many other non-visual sensors. Currently, video-based applications are effective in recognising and monitoring the activities, the movements, and the overall conditions of the assisted individuals as well as to assess their vital parameters (e.g., heart rate, respiratory rate). Similarly, audio sensors have the potential to become one of the most important modalities for interaction with AAL systems, as they can have a large range of sensing, do not require physical presence at a particular location and are physically intangible. Moreover, relevant information about individuals’ activities and health status can derive from processing audio signals (e.g., speech recordings). Nevertheless, as the other side of the coin, cameras and microphones are often perceived as the most intrusive technologies from the viewpoint of the privacy of the monitored individuals. This is due to the richness of the information these technologies convey and the intimate setting where they may be deployed. Solutions able to ensure privacy preservation by context and by design, as well as to ensure high legal and ethical standards are in high demand. After the review of the current state of play and the discussion in GoodBrother, we may claim that the first solutions in this direction are starting to appear in the literature. A multidisciplinary 4 debate among experts and stakeholders is paving the way towards AAL ensuring ergonomics, usability, acceptance and privacy preservation. The DIANA, PAAL, and VisuAAL projects are examples of this fresh approach. This report provides the reader with a review of the most recent advances in audio- and video-based monitoring technologies for AAL. It has been drafted as a collective effort of WG3 to supply an introduction to AAL, its evolution over time and its main functional and technological underpinnings. In this respect, the report contributes to the field with the outline of a new generation of ethical-aware AAL technologies and a proposal for a novel comprehensive taxonomy of AAL systems and applications. Moreover, the report allows non-technical readers to gather an overview of the main components of an AAL system and how these function and interact with the end-users. The report illustrates the state of the art of the most successful AAL applications and functions based on audio and video data, namely (i) lifelogging and self-monitoring, (ii) remote monitoring of vital signs, (iii) emotional state recognition, (iv) food intake monitoring, activity and behaviour recognition, (v) activity and personal assistance, (vi) gesture recognition, (vii) fall detection and prevention, (viii) mobility assessment and frailty recognition, and (ix) cognitive and motor rehabilitation. For these application scenarios, the report illustrates the state of play in terms of scientific advances, available products and research project. The open challenges are also highlighted. The report ends with an overview of the challenges, the hindrances and the opportunities posed by the uptake in real world settings of AAL technologies. In this respect, the report illustrates the current procedural and technological approaches to cope with acceptability, usability and trust in the AAL technology, by surveying strategies and approaches to co-design, to privacy preservation in video and audio data, to transparency and explainability in data processing, and to data transmission and communication. User acceptance and ethical considerations are also debated. Finally, the potentials coming from the silver economy are overviewed.publishedVersio

Patient Frailty: A Review

According to Geriator (2011), frailty is a common clinical syndrome in older adults that carries an increased risk for poor health outcomes such as falls, incident disability, hospitalization, and mortality. An operational definition of frailty is the existence of at least three out of five observable criteria indicating compromised energetics: low grip strength; low energy; slowed walking speed; low physical activity; and unintentional weight loss (Fried, et al. 2001). According to Dubois and Charpillet (2017), most transitions into frailty are due to unintentional weight loss which, in turn, impacts gait performance and subsequent general mobility. In this literature review, the focus is on those patients whose gait performance is compromised. As the common chain of events in this syndrome proceeds, poor gait performance results in low physical activity, decreasing muscular activity, and causing poor balance. Sarcopenia, exhaustion, poor gait performance and depression are primary risk factors for falls. This review also focuses on patients who are cared for at the home by the family physician, nurses, and family members. Research Question: How do existing methods identify patient frailty and what interventions can reduce adverse health outcomes and health care costs as well as maintain living at home for frail patients

Distributed Computing and Monitoring Technologies for Older Patients

This book summarizes various approaches for the automatic detection of health threats to older patients at home living alone. The text begins by briefly describing those who would most benefit from healthcare supervision. The book then summarizes possible scenarios for monitoring an older patient at home, deriving the common functional requirements for monitoring technology. Next, the work identifies the state of the art of technological monitoring approaches that are practically applicable to geriatric patients. A survey is presented on a range of such interdisciplinary fields as smart homes, telemonitoring, ambient intelligence, ambient assisted living, gerontechnology, and aging-in-place technology. The book discusses relevant experimental studies, highlighting the application of sensor fusion, signal processing and machine learning techniques. Finally, the text discusses future challenges, offering a number of suggestions for further research directions

Early diagnosis of frailty: Technological and non-intrusive devices for clinical detection

This work analyses different concepts for frailty diagnosis based on affordable standard technology such as smartphones or wearable devices. The goal is to provide ideas that go beyond classical diagnostic tools such as magnetic resonance imaging or tomography, thus changing the paradigm; enabling the detection of frailty without expensive facilities, in an ecological way for both patients and medical staff and even with continuous monitoring. Fried's five-point phenotype model of frailty along with a model based on trials and several classical physical tests were used for device classification. This work provides a starting point for future researchers who will have to try to bridge the gap separating elderly people from technology and medical tests in order to provide feasible, accurate and affordable tools for frailty monitoring for a wide range of users.This work was sponsored by the Spanish Ministry of Science, Innovation and Universities and the European Regional Development Fund (ERDF) across projects RTC-2017-6321-1 AEI/FEDER, UE, TEC2016-76021-C2-2-R AEI/FEDER, UE and PID2019-107270RB-C21/AEI/10.13039/501100011033, UE

State of the Art of Audio- and Video-Based Solutions for AAL

It is a matter of fact that Europe is facing more and more crucial challenges regarding health and social care due to the demographic change and the current economic context. The recent COVID-19 pandemic has stressed this situation even further, thus highlighting the need for taking action. Active and Assisted Living technologies come as a viable approach to help facing these challenges, thanks to the high potential they have in enabling remote care and support. Broadly speaking, AAL can be referred to as the use of innovative and advanced Information and Communication Technologies to create supportive, inclusive and empowering applications and environments that enable older, impaired or frail people to live independently and stay active longer in society. AAL capitalizes on the growing pervasiveness and effectiveness of sensing and computing facilities to supply the persons in need with smart assistance, by responding to their necessities of autonomy, independence, comfort, security and safety. The application scenarios addressed by AAL are complex, due to the inherent heterogeneity of the end-user population, their living arrangements, and their physical conditions or impairment. Despite aiming at diverse goals, AAL systems should share some common characteristics. They are designed to provide support in daily life in an invisible, unobtrusive and user-friendly manner. Moreover, they are conceived to be intelligent, to be able to learn and adapt to the requirements and requests of the assisted people, and to synchronise with their specific needs. Nevertheless, to ensure the uptake of AAL in society, potential users must be willing to use AAL applications and to integrate them in their daily environments and lives. In this respect, video- and audio-based AAL applications have several advantages, in terms of unobtrusiveness and information richness. Indeed, cameras and microphones are far less obtrusive with respect to the hindrance other wearable sensors may cause to one’s activities. In addition, a single camera placed in a room can record most of the activities performed in the room, thus replacing many other non-visual sensors. Currently, video-based applications are effective in recognising and monitoring the activities, the movements, and the overall conditions of the assisted individuals as well as to assess their vital parameters. Similarly, audio sensors have the potential to become one of the most important modalities for interaction with AAL systems, as they can have a large range of sensing, do not require physical presence at a particular location and are physically intangible. Moreover, relevant information about individuals’ activities and health status can derive from processing audio signals. Nevertheless, as the other side of the coin, cameras and microphones are often perceived as the most intrusive technologies from the viewpoint of the privacy of the monitored individuals. This is due to the richness of the information these technologies convey and the intimate setting where they may be deployed. Solutions able to ensure privacy preservation by context and by design, as well as to ensure high legal and ethical standards are in high demand. After the review of the current state of play and the discussion in GoodBrother, we may claim that the first solutions in this direction are starting to appear in the literature. A multidisciplinary debate among experts and stakeholders is paving the way towards AAL ensuring ergonomics, usability, acceptance and privacy preservation. The DIANA, PAAL, and VisuAAL projects are examples of this fresh approach. This report provides the reader with a review of the most recent advances in audio- and video-based monitoring technologies for AAL. It has been drafted as a collective effort of WG3 to supply an introduction to AAL, its evolution over time and its main functional and technological underpinnings. In this respect, the report contributes to the field with the outline of a new generation of ethical-aware AAL technologies and a proposal for a novel comprehensive taxonomy of AAL systems and applications. Moreover, the report allows non-technical readers to gather an overview of the main components of an AAL system and how these function and interact with the end-users. The report illustrates the state of the art of the most successful AAL applications and functions based on audio and video data, namely lifelogging and self-monitoring, remote monitoring of vital signs, emotional state recognition, food intake monitoring, activity and behaviour recognition, activity and personal assistance, gesture recognition, fall detection and prevention, mobility assessment and frailty recognition, and cognitive and motor rehabilitation. For these application scenarios, the report illustrates the state of play in terms of scientific advances, available products and research project. The open challenges are also highlighted. The report ends with an overview of the challenges, the hindrances and the opportunities posed by the uptake in real world settings of AAL technologies. In this respect, the report illustrates the current procedural and technological approaches to cope with acceptability, usability and trust in the AAL technology, by surveying strategies and approaches to co-design, to privacy preservation in video and audio data, to transparency and explainability in data processing, and to data transmission and communication. User acceptance and ethical considerations are also debated. Finally, the potentials coming from the silver economy are overviewed

A protected discharge facility for the elderly: design and validation of a working proof-of-concept

With the increasing share of elderly population worldwide, the need for assistive technologies to support clinicians in monitoring their health conditions is becoming more and more relevant. As a quantitative tool, geriatricians recently proposed the notion of frail elderly, which rapidly became a key element of clinical practices for the estimation of well-being in aging population. The evaluation of frailty is commonly based on self-reported outcomes and occasional physicians evaluations, and may therefore contain biased results. Another important aspect in the elderly population is hospitalization as a risk factor for patient\u2019s well being and public costs. Hospitalization is the main cause of functional decline, especially in older adults. The reduction of hospitalization time may allow an improvement of elderly health conditions and a reduction of hospital costs. Furthermore, a gradual transition from a hospital environment to a home-like one, can contribute to the weaning of the patient from a condition of hospitalization to a condition of discharge to his home. The advent of new technologies allows for the design and implementation of smart environments to monitor elderly health status and activities, fulfilling all the requirements of health and safety of the patients. From these starting points, in this thesis I present data-driven methodologies to automatically evaluate one of the main aspects contributing to the frailty estimation, i.e., the motility of the subject. First I will describe a model of protected discharge facility, realized in collaboration and within the E.O. Ospedali Galliera (Genoa, Italy), where patients can be monitored by a system of sensors while physicians and nurses have the opportunity to monitor them remotely. This sensorised facility is being developed to assist elderly users after they have been dismissed from the hospital and before they are ready to go back home, with the perspective of coaching them towards a healthy lifestyle. The facility is equipped with a variety of sensors (vision, depth, ambient and wearable sensors and medical devices), but in my thesis I primarily focus on RGB-D sensors and present visual computing tools to automatically estimate motility features. I provide an extensive system assessment I carried out onthree different experimental sessions with help of young as well as healthy aging volunteers. The results I present are in agreement with the assessment manually performed by physicians, showing the potential capability of my approach to complement current protocols of evaluation

Personalized functional health and fall risk prediction using electronic health records and in-home sensor data

Research has shown the importance of Electronic Health Records (EHR) and in-home sensor data for continuous health tracking and health risk predictions. With the increased computational capabilities and advances in machine learning techniques, we have new opportunities to use multi-modal health big data to develop accurate health tracking models. This dissertation describes the development, evaluation, and testing of systems for predicting functional health and fall risks in community-dwelling older adults using health data and machine learning techniques. In an initial study, we focused on organizing and de-identifying EHR data for analysis using HIPAA regulations. The dataset contained nine years of structured and unstructured EHR data obtained from TigerPlace, a senior living facility at Columbia, MO. The de-identification of this data was done using custom automated algorithms. The de-identified EHR data was used in several studies described in this dissertation. We then developed personalized functional health tracking models using geriatric assessments in the EHR data. Studies show that higher levels of functional health in older adults lead to a higher quality of life and improves the ability to age-in-place. Even though several geriatric assessments capture several aspects of functional health, there is limited research in longitudinally tracking the personalized functional health of older adults using a combination of these assessments. In this study, data from 150 older adult residents were used to develop a composite functional health prediction model using Activities of Daily Living (ADL), Instrumental Activities of Daily Living (IADL), Mini-Mental State Examination (MMSE), Geriatric Depression Scale (GDS), and Short Form 12 (SF12). Tracking functional health objectively could help clinicians to make decisions for interventions in case of functional health deterioration. We next constructed models for fall risk prediction in older adults using geriatric assessments, demographic data, and GAITRite assessment data. A 6-month fall risk prediction model was developed with data from 93 older adult residents. Explainable AI techniques were used to provide explanations to the model predictions, such as which specific features increased the risk of fall in a particular model prediction. Such explanations to model predictions provide valuable insights for targeted interventions. In another study, we developed deep neural network models to predict fall risk from de-identified nursing notes data from 162 older adult residents from TigerPlace. Clinical nursing notes have been shown to contain valuable information related to fall risk factors. This analysis provides the groundwork for future experiments to predict fall risk in older adults using clinical notes. In addition to using EHR data to predict functional health and fall risk in older adults, two studies were conducted to predict fall and functional health from in-home sensor data. Models for in-home fall prediction using depth sensor imagery have been successfully used at TigerPlace. However, the model is prone to false fall alarms in several scenarios, such as pillows thrown on the floor and pets jumping from couches. A secondary fall analysis was performed by analyzing fall alert videos to further identify and remove false alarms. In the final study, we used in-home sensor data streaming from depth sensors and bed sensors to predict functional health and absolute geriatric assessment values. These prediction models can be used to predict the functional health of residents in absence of sparse and infrequent geriatric assessments. This can also provide continuous tracking of functional health in older adults using the streaming in-home sensor data