State of the art of audio- and video based solutions for AAL

Working Group 3. Audio- and Video-based AAL ApplicationsIt is a matter of fact that Europe is facing more and more crucial challenges regarding health and social care due to the demographic change and the current economic context. The recent COVID-19 pandemic has stressed this situation even further, thus highlighting the need for taking action. Active and Assisted Living (AAL) technologies come as a viable approach to help facing these challenges, thanks to the high potential they have in enabling remote care and support. Broadly speaking, AAL can be referred to as the use of innovative and advanced Information and Communication Technologies to create supportive, inclusive and empowering applications and environments that enable older, impaired or frail people to live independently and stay active longer in society. AAL capitalizes on the growing pervasiveness and effectiveness of sensing and computing facilities to supply the persons in need with smart assistance, by responding to their necessities of autonomy, independence, comfort, security and safety. The application scenarios addressed by AAL are complex, due to the inherent heterogeneity of the end-user population, their living arrangements, and their physical conditions or impairment. Despite aiming at diverse goals, AAL systems should share some common characteristics. They are designed to provide support in daily life in an invisible, unobtrusive and user-friendly manner. Moreover, they are conceived to be intelligent, to be able to learn and adapt to the requirements and requests of the assisted people, and to synchronise with their specific needs. Nevertheless, to ensure the uptake of AAL in society, potential users must be willing to use AAL applications and to integrate them in their daily environments and lives. In this respect, video- and audio-based AAL applications have several advantages, in terms of unobtrusiveness and information richness. Indeed, cameras and microphones are far less obtrusive with respect to the hindrance other wearable sensors may cause to one’s activities. In addition, a single camera placed in a room can record most of the activities performed in the room, thus replacing many other non-visual sensors. Currently, video-based applications are effective in recognising and monitoring the activities, the movements, and the overall conditions of the assisted individuals as well as to assess their vital parameters (e.g., heart rate, respiratory rate). Similarly, audio sensors have the potential to become one of the most important modalities for interaction with AAL systems, as they can have a large range of sensing, do not require physical presence at a particular location and are physically intangible. Moreover, relevant information about individuals’ activities and health status can derive from processing audio signals (e.g., speech recordings). Nevertheless, as the other side of the coin, cameras and microphones are often perceived as the most intrusive technologies from the viewpoint of the privacy of the monitored individuals. This is due to the richness of the information these technologies convey and the intimate setting where they may be deployed. Solutions able to ensure privacy preservation by context and by design, as well as to ensure high legal and ethical standards are in high demand. After the review of the current state of play and the discussion in GoodBrother, we may claim that the first solutions in this direction are starting to appear in the literature. A multidisciplinary 4 debate among experts and stakeholders is paving the way towards AAL ensuring ergonomics, usability, acceptance and privacy preservation. The DIANA, PAAL, and VisuAAL projects are examples of this fresh approach. This report provides the reader with a review of the most recent advances in audio- and video-based monitoring technologies for AAL. It has been drafted as a collective effort of WG3 to supply an introduction to AAL, its evolution over time and its main functional and technological underpinnings. In this respect, the report contributes to the field with the outline of a new generation of ethical-aware AAL technologies and a proposal for a novel comprehensive taxonomy of AAL systems and applications. Moreover, the report allows non-technical readers to gather an overview of the main components of an AAL system and how these function and interact with the end-users. The report illustrates the state of the art of the most successful AAL applications and functions based on audio and video data, namely (i) lifelogging and self-monitoring, (ii) remote monitoring of vital signs, (iii) emotional state recognition, (iv) food intake monitoring, activity and behaviour recognition, (v) activity and personal assistance, (vi) gesture recognition, (vii) fall detection and prevention, (viii) mobility assessment and frailty recognition, and (ix) cognitive and motor rehabilitation. For these application scenarios, the report illustrates the state of play in terms of scientific advances, available products and research project. The open challenges are also highlighted. The report ends with an overview of the challenges, the hindrances and the opportunities posed by the uptake in real world settings of AAL technologies. In this respect, the report illustrates the current procedural and technological approaches to cope with acceptability, usability and trust in the AAL technology, by surveying strategies and approaches to co-design, to privacy preservation in video and audio data, to transparency and explainability in data processing, and to data transmission and communication. User acceptance and ethical considerations are also debated. Finally, the potentials coming from the silver economy are overviewed.publishedVersio

Computer vision based posture estimation and fall detection.

Falls are a major health problem, especially in the elderly population. Increasing fall events demands a high quality of service and dedicated medical treatment which is an economic burden. Serious injuries due to fall can cost lives in the absence of immediate care and support. There- fore, a monitoring system that can accurately detect fall events and generate instant alerts for immediate care is extremely necessary. To address this problem, this research aims to develop a computer vision-based fall detection system. This study proposes fall detection in three stages: (A) Detection of human silhouette and recognition of the pose, (B) Detection of the human as three regions for different postures including fall and (C) Recognise fall and non-fall using locations of human body regions as distinguishing features. The first stages of work comprise human silhouette detection and identification of activities in the form of different poses. Identifying a pose is important to understand a fall event where a change of pose defines its characteristics. A fall event comprises of sequential change of poses and ends up in a lying pose. Initial pose during a fall can be standing, sitting or bending but the final pose is usually a lying pose. It would, therefore, be beneficial if lying pose is recognised more accurately than other normal activities such as standing, sitting, bending or crawling to address a fall. Hence in the first stage, Background Subtraction (BS) is used to detect human silhouette. After background subtraction, the foreground images were used in a Convolutional Neural Network (CNN) to recognise different poses. The RGB and the Depth images were captured from a Kinect Sensor. The fusion of RGB and Depth images were explored for feeding to a convolutional neural net- work. Depth together with RGB complimented each other to overcome their weakness respectively and proved to be a significant strategy. The classification was performed using CNN to recognise different activities with 81% accuracy on validation. The other challenge in fall detection is the tracking of a person during a fall. Background Subtraction is not sufficient to track a fallen person especially when there are lighting and viewpoint variations in the environment and present of another object like furniture, a pet or even another person. Furthermore, tracking be- comes tougher during the fall in comparison to normal activities like walking or sitting because the rate of change pose is higher during a fall. To overcome this, the idea is to locate the regions in the body in every frame and consider it as a stable tracking strategy. The location of the body parts provides crucial information to distinguish falls from the other normal activities as the person is detected all the time during these activities. Hence the second stage of this research consists of posture detection using the pose estimation technique. This research proposes to use CNN based pose estimation using simplified human postures. The available joints are grouped according to three regions: Head, Torso and Leg and then finally fed to the CNN model with just three inputs instead of several available joints. This strategy added stability in pose detection and proved to be more effective against complex poses observed during a fall. To train the CNN model, transfer learning technique was used. The model was able to achieve 96.7% accuracy in detecting the three regions on different human postures on the publicly available dataset. A system which considers all the lying poses as falls can also generate a higher false alarm. Lying on bed or sofa can easily generate a fall alarm if they are recognised as falls. Hence, it is important to recognise actual fall by considering a sequence of frames that defines a fall and not just the lying pose. In the third and final stage, this study proposes Long Short-Term Memory (LSTM) recurrent networks-based fall detection. The proposed LSTM model uses the detected three region’s location as input features. LSTM is capable of using contextual information from the sequential input patterns. Therefore, the LSTM model was fed with location features of different postures in a sequence for training. The model was able to learn fall patterns and distinguish them from other activities with 88.33% accuracy. Furthermore, the precision of the fall class was 1.0. This is highly desirable in the case of fall detection as there is no false alarm and this means that the cost incurred in calling medical support for a false alarm can be completely avoided

State of the Art of Audio- and Video-Based Solutions for AAL

It is a matter of fact that Europe is facing more and more crucial challenges regarding health and social care due to the demographic change and the current economic context. The recent COVID-19 pandemic has stressed this situation even further, thus highlighting the need for taking action. Active and Assisted Living technologies come as a viable approach to help facing these challenges, thanks to the high potential they have in enabling remote care and support. Broadly speaking, AAL can be referred to as the use of innovative and advanced Information and Communication Technologies to create supportive, inclusive and empowering applications and environments that enable older, impaired or frail people to live independently and stay active longer in society. AAL capitalizes on the growing pervasiveness and effectiveness of sensing and computing facilities to supply the persons in need with smart assistance, by responding to their necessities of autonomy, independence, comfort, security and safety. The application scenarios addressed by AAL are complex, due to the inherent heterogeneity of the end-user population, their living arrangements, and their physical conditions or impairment. Despite aiming at diverse goals, AAL systems should share some common characteristics. They are designed to provide support in daily life in an invisible, unobtrusive and user-friendly manner. Moreover, they are conceived to be intelligent, to be able to learn and adapt to the requirements and requests of the assisted people, and to synchronise with their specific needs. Nevertheless, to ensure the uptake of AAL in society, potential users must be willing to use AAL applications and to integrate them in their daily environments and lives. In this respect, video- and audio-based AAL applications have several advantages, in terms of unobtrusiveness and information richness. Indeed, cameras and microphones are far less obtrusive with respect to the hindrance other wearable sensors may cause to one’s activities. In addition, a single camera placed in a room can record most of the activities performed in the room, thus replacing many other non-visual sensors. Currently, video-based applications are effective in recognising and monitoring the activities, the movements, and the overall conditions of the assisted individuals as well as to assess their vital parameters. Similarly, audio sensors have the potential to become one of the most important modalities for interaction with AAL systems, as they can have a large range of sensing, do not require physical presence at a particular location and are physically intangible. Moreover, relevant information about individuals’ activities and health status can derive from processing audio signals. Nevertheless, as the other side of the coin, cameras and microphones are often perceived as the most intrusive technologies from the viewpoint of the privacy of the monitored individuals. This is due to the richness of the information these technologies convey and the intimate setting where they may be deployed. Solutions able to ensure privacy preservation by context and by design, as well as to ensure high legal and ethical standards are in high demand. After the review of the current state of play and the discussion in GoodBrother, we may claim that the first solutions in this direction are starting to appear in the literature. A multidisciplinary debate among experts and stakeholders is paving the way towards AAL ensuring ergonomics, usability, acceptance and privacy preservation. The DIANA, PAAL, and VisuAAL projects are examples of this fresh approach. This report provides the reader with a review of the most recent advances in audio- and video-based monitoring technologies for AAL. It has been drafted as a collective effort of WG3 to supply an introduction to AAL, its evolution over time and its main functional and technological underpinnings. In this respect, the report contributes to the field with the outline of a new generation of ethical-aware AAL technologies and a proposal for a novel comprehensive taxonomy of AAL systems and applications. Moreover, the report allows non-technical readers to gather an overview of the main components of an AAL system and how these function and interact with the end-users. The report illustrates the state of the art of the most successful AAL applications and functions based on audio and video data, namely lifelogging and self-monitoring, remote monitoring of vital signs, emotional state recognition, food intake monitoring, activity and behaviour recognition, activity and personal assistance, gesture recognition, fall detection and prevention, mobility assessment and frailty recognition, and cognitive and motor rehabilitation. For these application scenarios, the report illustrates the state of play in terms of scientific advances, available products and research project. The open challenges are also highlighted. The report ends with an overview of the challenges, the hindrances and the opportunities posed by the uptake in real world settings of AAL technologies. In this respect, the report illustrates the current procedural and technological approaches to cope with acceptability, usability and trust in the AAL technology, by surveying strategies and approaches to co-design, to privacy preservation in video and audio data, to transparency and explainability in data processing, and to data transmission and communication. User acceptance and ethical considerations are also debated. Finally, the potentials coming from the silver economy are overviewed

Automatic Fall Risk Detection based on Imbalanced Data

In recent years, the declining birthrate and aging population have gradually brought countries into an ageing society. Regarding accidents that occur amongst the elderly, falls are an essential problem that quickly causes indirect physical loss. In this paper, we propose a pose estimation-based fall detection algorithm to detect fall risks. We use body ratio, acceleration and deflection as key features instead of using the body keypoints coordinates. Since fall data is rare in real-world situations, we train and evaluate our approach in a highly imbalanced data setting. We assess not only different imbalanced data handling methods but also different machine learning algorithms. After oversampling on our training data, the K-Nearest Neighbors (KNN) algorithm achieves the best performance. The F1 scores for three different classes, Normal, Fall, and Lying, are 1.00, 0.85 and 0.96, which is comparable to previous research. The experiment shows that our approach is more interpretable with the key feature from skeleton information. Moreover, it can apply in multi-people scenarios and has robustness on medium occlusion

Proposition of a checklist to assist in the development and evaluation of virtual reality games for the virtual rehabilitation of the elderly

Recentemente, os jogos digitais e a tecnologia de realidade virtual vêm sendo utilizados como um recurso complementar para a reabilitação. É importante destacar que diversos fatores devem ser levados em consideração ao desenvolver um artefato digital para auxiliar na reabilitação virtual de idosos. Sendo assim, o objetivo desse artigo foi propor um checklist para auxiliar no desenvolvimento e avaliação de jogos digitais em realidade virtual para a reabilitação virtual de idosos. A metodologia foi dividida em quatro etapas: revisão sistemática de literatura, proposição do checklist, coleta e análise de dados. A pesquisa pode ser classificada como um estudo quanti-qualitativo. Foi feito um grupo focal confirmatório com dois participantes para verificar o checklist proposto. O instrumento para a coleta de dados foi a entrevista semiestruturada. Como resultado, foi feita uma revisão de literatura a fim de levantar um conjunto de diretrizes para a reabilitação virtual deidosos. Foi aplicada a atividade de card sorting para sistematizar esse conjunto de diretrizes. A partir disso, foi apresentado um checklist que visa orientar o desenvolvimento e a avaliação de um artefato digital para auxiliar na reabilitação virtual dos idosos. O checklist proposto inclui 29 itens, organizados nas seguintes categorias: terapia, motivação, interação e segurança. Foram sugeridas duas versões do checklist. A primeira é voltadapara os desenvolvedores e visa orientar a construção do artefato digital. Já a segunda pode ser aplicada naetapa de avaliação para verificar se o jogo digital em realidade virtual atende às necessidades e preferênciasdo público sênior.Recently, digital games and virtual reality technology have been used as a complementary feature for rehabilitation. It is important to emphasize that several factors should be taken into account by developing a digital artifact to assist in the virtual rehabilitation of the elderly. In this way, the aim of this paper was to propose a checklist to assist in the development and evaluation of virtual reality games for the virtual rehabilitation of the elderly. The research method was divided in four stages: systematic review, checklist proposition, data collection and analyses. This research can be classified as quanti-qualitative study. A confirmatory focus group with two participants was carried out to verify the proposed checklist. The assessment tool was a semi-structured interview. As a result, a systematic review was conducted in order to investigate a set of guidelines for the virtual rehabilitation of the elderly. The card sorting activity was applied to systematize this set of guidelines. From this, a checklist was proposed that aimed to orientate the development and evaluation of a digital artifact to assist in the virtual rehabilitation of the elderly. The proposed checklist includes 29 items, organized in the following categories: therapy, motivation, interaction and safety. Two versions of the checklist were suggested. The first one is focused on the developers and aims to orientate the construction of a digital artifact. The second one can be applied in the evaluation phase to verify if the virtual reality game meets needs and preferences of the senior users

Contextual game design: from interface development to human activity recognition

A dissertação focasse nos seguintes pontos: Criação de diferentes interfaces usando o sensor Kinect, para reablitação de pacientes com cancro de mama. E, reconhecimento de atividade humana (problema derivado aquando a criação das interfaces)

Recent Advances in Motion Analysis

The advances in the technology and methodology for human movement capture and analysis over the last decade have been remarkable. Besides acknowledged approaches for kinematic, dynamic, and electromyographic (EMG) analysis carried out in the laboratory, more recently developed devices, such as wearables, inertial measurement units, ambient sensors, and cameras or depth sensors, have been adopted on a wide scale. Furthermore, computational intelligence (CI) methods, such as artificial neural networks, have recently emerged as promising tools for the development and application of intelligent systems in motion analysis. Thus, the synergy of classic instrumentation and novel smart devices and techniques has created unique capabilities in the continuous monitoring of motor behaviors in different fields, such as clinics, sports, and ergonomics. However, real-time sensing, signal processing, human activity recognition, and characterization and interpretation of motion metrics and behaviors from sensor data still representing a challenging problem not only in laboratories but also at home and in the community. This book addresses open research issues related to the improvement of classic approaches and the development of novel technologies and techniques in the domain of motion analysis in all the various fields of application

SHELDON Smart habitat for the elderly.

An insightful document concerning active and assisted living under different perspectives: Furniture and habitat, ICT solutions and Healthcare

A framework and serious game to support those with mild cognitive impairment

This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonCognitive decline is common in the elderly. As a result, a range of cognitive rehabilitation games have been proposed to supplement or replace traditional rehabilitative training by offering benefits such as improved engagement. This research project focuses on mild cognitive impairment (MCI), an initial stage of cognitive decline that does not affect functioning in daily life, but which may progress towards more serious cognitive deteriorations, notably dementia. Unfortunately, while a variety of serious game frameworks and rehabilitative serious games have been proposed, there is a distinct lack of those which support the distinctive characteristics of MCI patients. Consequently, to optimise the advantages of serious games for MCI, the research proposes the MCI-GaTE (MCI-Game Therapy Experience) framework that may be used to develop serious games as effective cognitive and physical rehabilitation tools. The framework is derived from a combination of a survey of related research literature in the area, analysis of resident profiles from a nursing home, and in-depth interviews with occupational therapists (OTs) who work with MCI patients on a daily basis to help them overcome the disabling effects so that they can carry out everyday tasks. The conceptual framework comprises four sectors that may be used to guide game design and development: an MCI player profile that represents the capabilities of a player with MCI, core gaming elements that support gameful and playful activities, therapeutic elements that support cognitive and physical rehabilitation through tasks and scenarios according to the player’s abilities, and motivational elements to enhance the player’s attitude towards the serious tasks. Together, they provide tailored support for rehabilitation needs and may also serve as a set of comprehensive and established criteria by which an MCI serious game may be evaluated. To demonstrate the use of MCI-GaTE, an immersive and gesture-based serious game, A go!, is designed that exploits the framework to enable MCI-diagnosed players to undertake a series of tailored therapeutic tasks supported by an assigned OT. To this end, a goal-directed design approach is employed, whereby personas, scenarios and journey maps are developed that satisfy the goals of both the MCI player and their OT, and enable the derivation of functional requirements leading to a visual design. A-go! is realised as a responsive and interactive high-fidelity prototype that supports gesture recognition and 3D game objects from a first-person perspective to facilitate immersion without the need for additional worn devices, such as headsets, which would prove impractical for the targeted elderly players. Evaluation with OTs revealed that the immersive game potentially offers more effective and tailored support to MCI patients than traditional methods, contributing new possibilities for enhancing MCI rehabilitative training, while a comparative assessment of MCI-GaTE demonstrated that it provides a comprehensive approach not currently offered by state-of-the-art rehabilitative frameworks.GRACE Healthcare Ltd. & Hong Kong Caritas in Evergreen Home and Integrated Home Care Services, TWGHs Fung Yiu King Hospital HA, Kwai Chung Hospital HA, and TWGHs Jockey Club Rehabilitation Comple

Providing Real-Time Exercise Feedback to Patients Undergoing Physical Therapy

Musculoskeletal conditions, often requiring rehabilitation, affect one-third of the U.S. population annually. RehabBuddy is a rehabilitation assistance system that extends the reach of a physical rehabilitation specialist beyond the clinic. This thesis presents a system that uses body-worn motion sensors and a mobile application that provides the patient with assistance to ensure that home exercises are performed with the same precision as under clinical supervision. Assisted by a specialist in the clinic, the wearable sensors and user interface developed allow the capture of individualized exercises unique to the patient's physical abilities. Beyond the clinical setting, the system can assist patients by providing real-time corrective feedback to repeat these exercises through a correct and complete arc of motion for the prescribed number of repetitions. An inertial measurement unit (IMU) is used on the body part to be exercised to capture its pose. Presented is a kinematics data processing approach to defining custom exercises with flexibility in terms of where it is worn and the nature of the exercise, as well as real-time corrective feedback parameters. This thesis goes through the engineering approach, initial student investigator trials, and presents new preliminary subject data from subject trials currently ongoing at the University of Kentucky. The system is tested on multiple exercises performed by multiple subjects. It is then demonstrated how it can improve exercise adherence by assisting patients in reaching the full prescribed range of motion and avoid overextension, assist in adherence to the ideal plane of motion, and affect hold time.MSComputer Engineering, College of Engineering & Computer ScienceUniversity of Michigan-Dearbornhttp://deepblue.lib.umich.edu/bitstream/2027.42/169159/1/Ella Reimann Final Thesis.pd