State of the art of audio- and video based solutions for AAL

Working Group 3. Audio- and Video-based AAL ApplicationsIt is a matter of fact that Europe is facing more and more crucial challenges regarding health and social care due to the demographic change and the current economic context. The recent COVID-19 pandemic has stressed this situation even further, thus highlighting the need for taking action. Active and Assisted Living (AAL) technologies come as a viable approach to help facing these challenges, thanks to the high potential they have in enabling remote care and support. Broadly speaking, AAL can be referred to as the use of innovative and advanced Information and Communication Technologies to create supportive, inclusive and empowering applications and environments that enable older, impaired or frail people to live independently and stay active longer in society. AAL capitalizes on the growing pervasiveness and effectiveness of sensing and computing facilities to supply the persons in need with smart assistance, by responding to their necessities of autonomy, independence, comfort, security and safety. The application scenarios addressed by AAL are complex, due to the inherent heterogeneity of the end-user population, their living arrangements, and their physical conditions or impairment. Despite aiming at diverse goals, AAL systems should share some common characteristics. They are designed to provide support in daily life in an invisible, unobtrusive and user-friendly manner. Moreover, they are conceived to be intelligent, to be able to learn and adapt to the requirements and requests of the assisted people, and to synchronise with their specific needs. Nevertheless, to ensure the uptake of AAL in society, potential users must be willing to use AAL applications and to integrate them in their daily environments and lives. In this respect, video- and audio-based AAL applications have several advantages, in terms of unobtrusiveness and information richness. Indeed, cameras and microphones are far less obtrusive with respect to the hindrance other wearable sensors may cause to one’s activities. In addition, a single camera placed in a room can record most of the activities performed in the room, thus replacing many other non-visual sensors. Currently, video-based applications are effective in recognising and monitoring the activities, the movements, and the overall conditions of the assisted individuals as well as to assess their vital parameters (e.g., heart rate, respiratory rate). Similarly, audio sensors have the potential to become one of the most important modalities for interaction with AAL systems, as they can have a large range of sensing, do not require physical presence at a particular location and are physically intangible. Moreover, relevant information about individuals’ activities and health status can derive from processing audio signals (e.g., speech recordings). Nevertheless, as the other side of the coin, cameras and microphones are often perceived as the most intrusive technologies from the viewpoint of the privacy of the monitored individuals. This is due to the richness of the information these technologies convey and the intimate setting where they may be deployed. Solutions able to ensure privacy preservation by context and by design, as well as to ensure high legal and ethical standards are in high demand. After the review of the current state of play and the discussion in GoodBrother, we may claim that the first solutions in this direction are starting to appear in the literature. A multidisciplinary 4 debate among experts and stakeholders is paving the way towards AAL ensuring ergonomics, usability, acceptance and privacy preservation. The DIANA, PAAL, and VisuAAL projects are examples of this fresh approach. This report provides the reader with a review of the most recent advances in audio- and video-based monitoring technologies for AAL. It has been drafted as a collective effort of WG3 to supply an introduction to AAL, its evolution over time and its main functional and technological underpinnings. In this respect, the report contributes to the field with the outline of a new generation of ethical-aware AAL technologies and a proposal for a novel comprehensive taxonomy of AAL systems and applications. Moreover, the report allows non-technical readers to gather an overview of the main components of an AAL system and how these function and interact with the end-users. The report illustrates the state of the art of the most successful AAL applications and functions based on audio and video data, namely (i) lifelogging and self-monitoring, (ii) remote monitoring of vital signs, (iii) emotional state recognition, (iv) food intake monitoring, activity and behaviour recognition, (v) activity and personal assistance, (vi) gesture recognition, (vii) fall detection and prevention, (viii) mobility assessment and frailty recognition, and (ix) cognitive and motor rehabilitation. For these application scenarios, the report illustrates the state of play in terms of scientific advances, available products and research project. The open challenges are also highlighted. The report ends with an overview of the challenges, the hindrances and the opportunities posed by the uptake in real world settings of AAL technologies. In this respect, the report illustrates the current procedural and technological approaches to cope with acceptability, usability and trust in the AAL technology, by surveying strategies and approaches to co-design, to privacy preservation in video and audio data, to transparency and explainability in data processing, and to data transmission and communication. User acceptance and ethical considerations are also debated. Finally, the potentials coming from the silver economy are overviewed.publishedVersio

Evaluation of Elderly Fall Detection Systems using Data Analysis

As the global population continues to age, it is more important to ensure the safety and wellbeing of senior citizens. Falls are a frequent and dangerous concern for elders, frequently resulting in fatalities or very serious injuries. Modern technology, such as the Elderly Fall Detection System, have been created to solve this issue. This technology attempts to detect falls quickly and react to them, offering aid right away and maybe saving lives. These studies identify a fall when the tri-axial accelerometer reading from a wearable device exceeds the predetermined threshold. Less complexity and computational expense compared to other approaches is one of the main benefits of adopting threshold-based methods. Finding the right threshold value, however, to accurately identify all falls without confounding them with certain ADL, has proven to be a challenging issue. Early Intervention and Medical Aid: Being independent and able to live in one's own house are two things that many elderly people enjoy. Support for carers and peace of mind: detection systems not only aid the elderly but also provide their carers peace of mind. Cost-cutting: Falls among the elderly frequently lead to hospital stays, rehab, and higher healthcare expenses. The Weighted Product Model (WPM) is a technique for ranking and evaluating alternatives based on a variety of factors. It is an easy and obvious method that enables decision-makers to weigh the relative weight of many variables and make wise decisions. Smart surveillance system, Smart cane, Smart Carpet, Smart phone/watch and app. Unobtrusiveness, reliability, privacy, Cost. As the global population continues to age, it is more important to ensure the safety and wellbeing of senior citizens. Falls are a frequent and dangerous concern for elders, frequently resulting in fatalities or very serious injuries The Weighted Product Model (WPM) is a technique for ranking and evaluating alternatives based on a variety of factors Unobtrusiveness, reliability, privacy, Cost, smart phone is the highest and smart carpet is the lowes

Investigation of Low-Cost Wearable Internet of Things Enabled Technology for Physical Activity Recognition in the Elderly

Technological advances in mobile sensing technologies has produced new opportunities for the monitoring of the elderly in uncontrolled environments by researchers. Sensors have become smaller, cheaper and can be worn on the body, potentially creating a network of sensors. Smart phones are also more common in the average household and can also provide some behavioural analysis due to the built-in sensors. As a result of this, researchers are able to monitor behaviours in a more naturalistic setting, which can lead to more contextually meaningful data. For those suffering with a mental illness, non-invasive and continuous monitoring can be achieved. Applying sensors to real world environments can aid in improving the quality of life of an elderly person with a mental illness and monitor their condition through behavioural analysis. In order to achieve this, selected classifiers must be able to accurately detect when an activity has taken place. In this thesis we aim to provide a framework for the investigation of activity recognition in the elderly using low-cost wearable sensors, which has resulted in the following contributions: 1. Classification of eighteen activities which were broken down into three disparate categories typical in a home setting: dynamic, sedentary and transitional. These were detected using two Shimmer3 IMU devices that we have located on the participants’ wrist and waist to create a low-cost, contextually deployable solution for elderly care monitoring. 2. Through the categorisation of performed Extracted time-domain and frequency-domain features from the Shimmer devices accelerometer and gyroscope were used as inputs, we achieved a high accuracy classification from a Convolutional Neural Network (CNN) model applied to the data set gained from participants recruited to the study through Join Dementia Research. The model was evaluated by variable adjustments to the model, tracking changes in its performance. Performance statistics were generated by the model for comparison and evaluation. Our results indicate that a low epoch of 200 using the ReLu activation function can display a high accuracy of 86% on the wrist data set and 85% on the waist data set, using only two low-cost wearable devices

Comprehensive review of vision-based fall detection systems

Vision-based fall detection systems have experienced fast development over the last years. To determine the course of its evolution and help new researchers, the main audience of this paper, a comprehensive revision of all published articles in the main scientific databases regarding this area during the last five years has been made. After a selection process, detailed in the Materials and Methods Section, eighty-one systems were thoroughly reviewed. Their characterization and classification techniques were analyzed and categorized. Their performance data were also studied, and comparisons were made to determine which classifying methods best work in this field. The evolution of artificial vision technology, very positively influenced by the incorporation of artificial neural networks, has allowed fall characterization to become more resistant to noise resultant from illumination phenomena or occlusion. The classification has also taken advantage of these networks, and the field starts using robots to make these systems mobile. However, datasets used to train them lack real-world data, raising doubts about their performances facing real elderly falls. In addition, there is no evidence of strong connections between the elderly and the communities of researchers

State of the Art of Audio- and Video-Based Solutions for AAL

It is a matter of fact that Europe is facing more and more crucial challenges regarding health and social care due to the demographic change and the current economic context. The recent COVID-19 pandemic has stressed this situation even further, thus highlighting the need for taking action. Active and Assisted Living technologies come as a viable approach to help facing these challenges, thanks to the high potential they have in enabling remote care and support. Broadly speaking, AAL can be referred to as the use of innovative and advanced Information and Communication Technologies to create supportive, inclusive and empowering applications and environments that enable older, impaired or frail people to live independently and stay active longer in society. AAL capitalizes on the growing pervasiveness and effectiveness of sensing and computing facilities to supply the persons in need with smart assistance, by responding to their necessities of autonomy, independence, comfort, security and safety. The application scenarios addressed by AAL are complex, due to the inherent heterogeneity of the end-user population, their living arrangements, and their physical conditions or impairment. Despite aiming at diverse goals, AAL systems should share some common characteristics. They are designed to provide support in daily life in an invisible, unobtrusive and user-friendly manner. Moreover, they are conceived to be intelligent, to be able to learn and adapt to the requirements and requests of the assisted people, and to synchronise with their specific needs. Nevertheless, to ensure the uptake of AAL in society, potential users must be willing to use AAL applications and to integrate them in their daily environments and lives. In this respect, video- and audio-based AAL applications have several advantages, in terms of unobtrusiveness and information richness. Indeed, cameras and microphones are far less obtrusive with respect to the hindrance other wearable sensors may cause to one’s activities. In addition, a single camera placed in a room can record most of the activities performed in the room, thus replacing many other non-visual sensors. Currently, video-based applications are effective in recognising and monitoring the activities, the movements, and the overall conditions of the assisted individuals as well as to assess their vital parameters. Similarly, audio sensors have the potential to become one of the most important modalities for interaction with AAL systems, as they can have a large range of sensing, do not require physical presence at a particular location and are physically intangible. Moreover, relevant information about individuals’ activities and health status can derive from processing audio signals. Nevertheless, as the other side of the coin, cameras and microphones are often perceived as the most intrusive technologies from the viewpoint of the privacy of the monitored individuals. This is due to the richness of the information these technologies convey and the intimate setting where they may be deployed. Solutions able to ensure privacy preservation by context and by design, as well as to ensure high legal and ethical standards are in high demand. After the review of the current state of play and the discussion in GoodBrother, we may claim that the first solutions in this direction are starting to appear in the literature. A multidisciplinary debate among experts and stakeholders is paving the way towards AAL ensuring ergonomics, usability, acceptance and privacy preservation. The DIANA, PAAL, and VisuAAL projects are examples of this fresh approach. This report provides the reader with a review of the most recent advances in audio- and video-based monitoring technologies for AAL. It has been drafted as a collective effort of WG3 to supply an introduction to AAL, its evolution over time and its main functional and technological underpinnings. In this respect, the report contributes to the field with the outline of a new generation of ethical-aware AAL technologies and a proposal for a novel comprehensive taxonomy of AAL systems and applications. Moreover, the report allows non-technical readers to gather an overview of the main components of an AAL system and how these function and interact with the end-users. The report illustrates the state of the art of the most successful AAL applications and functions based on audio and video data, namely lifelogging and self-monitoring, remote monitoring of vital signs, emotional state recognition, food intake monitoring, activity and behaviour recognition, activity and personal assistance, gesture recognition, fall detection and prevention, mobility assessment and frailty recognition, and cognitive and motor rehabilitation. For these application scenarios, the report illustrates the state of play in terms of scientific advances, available products and research project. The open challenges are also highlighted. The report ends with an overview of the challenges, the hindrances and the opportunities posed by the uptake in real world settings of AAL technologies. In this respect, the report illustrates the current procedural and technological approaches to cope with acceptability, usability and trust in the AAL technology, by surveying strategies and approaches to co-design, to privacy preservation in video and audio data, to transparency and explainability in data processing, and to data transmission and communication. User acceptance and ethical considerations are also debated. Finally, the potentials coming from the silver economy are overviewed

Modelos de aprendizaje automático en la detección e identificación de personas: una revisión de literatura

Introduction: This article is the result of research entitled "Development of a prototype to optimize access conditions to the SENA-Pescadero using artificial intelligence and open-source tools", developed at the Servicio Nacional de Aprendizaje in 2020.   Problem: How to identify Machine Learning Techniques applied to computer vision processes through a literature review? Objective: Determine the application, as well as advantages and disadvantages of machine learning techniques focused on the detection and identification of people. Methodology: Systematic literature review in 4 high-impact bibliographic and scientific databases, using search filters and information selection criteria. Results: Machine Learning techniques defined as Principal Component Analysis, Weak Label Regularized Local Coordinate Coding, Support Vector Machines, Haar Cascade Classifiers and EigenFaces and FisherFaces, as well as their applicability in detection and identification processes.   Conclusion: The research led to the identification of the main computational intelligence techniques based on machine learning, applied to the detection and identification of people. Their influence was shown in several application cases, but most of them were focused on the implementation and optimization of access control systems, or tasks in which the identification of people was required for the execution of processes. Originality: Through this research, we studied and defined the main machine learning techniques currently used for the detection and identification of people. Limitations: The systematic review is limited to information available in the 4 databases consulted, and the amount of information is variable as articles are deposited in the databases.Introducción: Este artículo es el resultado de la investigación titulada " Desarrollo de un prototipo para optimizar las condiciones de acceso al SENA-Pescadero utilizando inteligencia artificial y herramientas de código abierto", desarrollada en el Servicio Nacional de Aprendizaje en 2020. Problema: ¿Cómo identificar las técnicas de aprendizaje automático aplicadas a los procesos de visión por computador a través de una revisión bibliográfica? Objetivo: Determinar la aplicación, así como las ventajas y desventajas de las técnicas de aprendizaje automático enfocadas a la detección e identificación de personas. Metodología: Revisión sistemática de la literatura en 4 bases de datos bibliográficas y científicas de alto impacto, utilizando filtros de búsqueda y criterios de selección de información. Resultados: Técnicas de aprendizaje automático definidas como Análisis de Componentes Principales, Codificación Local de Coordenadas Regularizada de Etiquetas Débiles, Máquinas de Vectores de Soporte, Clasificadores en Cascada de Haar y EigenFaces y FisherFaces, así como su aplicabilidad en procesos de detección e identificación. Conclusiones: La investigación permitió identificar las principales técnicas de inteligencia computacional basadas en machine learning aplicadas a la detección e identificación de personas. Su influencia se mostró en varios casos de aplicación, pero la mayoría de ellos se centraron en la implementación y optimización de sistemas de control de acceso, o tareas en las que se requería la identificación de personas para la ejecución de procesos Originalidad: A través de esta investigación se estudiaron y definieron las principales técnicas de machine learning utilizadas actualmente para la detección e identificación de personas

Pedestrian soft biometrics recognition using deep learning on thermal images in smart cities

With technological advancement and the rise of the Internet of Things, our society is becoming more interconnected than ever before. Our computers and devices are getting smaller, and their computing power and memory has been increasing. These advances coupled with the leaps in artificial intelligence caused by the deep learning revolution in recent yearshave led to an increasingly rising interest in the field of pervasive intelligence. Intelligence in the environment has been used in smart homes in order to bring assistance to semi-autonomous people by performing activity recognition based on sensor data. As technology keeps improving, we may start to investigate the extension of assistive technologies beyond the boundaries of smart homes and into our smart cities. In order to bring assistance to semi-autonomous people, the first step is to be able to recognize profiles of vulnerable people. In order to leverage technology and artificial intelligence to make our cities smarter, safer and more accessible, this thesis investigates the use of environmental sensors such as thermal cameras to perform pedestrian soft biometrics recognition (age, gender and mobility) in the city. In this thesis, the process of building prototypes from scratch in order to collect thermal gait data in the city is explored, and the use and optimization of deep learning algorithms to perform soft biometrics recognition, as well as the feasibility of implementing these algorithms on limited resource boards are explored. The use of unprocessed thermal images allows a higher degree of privacy for the citizens, and it is novel in the field of human profile recognition. This thesis aims to set the foundation of future work, both in the field of thermal images-based soft biometrics recognition and pervasive intelligence in our cities in order to make them smarter, and move towards an interconnected society. Les progrès technologiques et le développement de l’Internet des Objets nous mènent vers une société de plus en plus interconnectée. Nos ordinateurs et nos appareils deviennent de plus en plus petits et leur puissance de calcul et leur mémoire ne cesse de s’améliorer. Ces avancées combinées aux récents progrès dans le domaine de l’intelligence artificielle avec la révolution de l’apprentissage profond ont mené à un intérêt grandissant dans le domaine de l’intelligence ambiante. L’intelligence ambiante a été utilisée dans le domaine des maisons intelligentes sous forme de reconnaissance d’activités, permettant d’assister les personnes semi-autonomes en utilisant des données collectées par des capteurs. Alors que le progrès technologique continue, nous arrivons à un point où l’hypothèse d’étendre ces stratégies d’assistance des maisons aux villes intelligentes devient de plus en plus réaliste. Afin d’étendre cette assistance aux villes, la première étape est d’identifier les personnes vulnérables, qui sont celles qui pourraient bénéficier de cette assistance. Dans le but d’utiliser la technologie pour rendre nos villes plus intelligentes, plus sûres et plus accessibles, cette thèse explore l’utilisation de capteurs environnementaux tels que des caméras thermiques pour effectuer de la reconnaissance de profils dans la ville (âge, genre et mobilité). Dans cette thèse, le processus de construction de prototypes pour récolter des données thermales dans la ville est présenté, et l’utilisation ainsi que l’optimisation d’algorithmes d’apprentissage profond pour la reconnaissance de profils est explorée. L’implémentation des algorithmes sur un système embarqué est également abordée. L’utilisation d’images thermiques garantit un plus grand degré d’anonymat pour les citoyens que l’utilisation de caméras RGB, et cette thèse représente les premiers travaux de reconnaissance de profils multiples en utilisant uniquement des images thermiques sans pré-traitement. Cette thèse a pour objectif de poser les bases pour des travaux futurs dans le domaine de la reconnaissance de profils en utilisant des images thermiques, ainsi que dans le domaine de l’intelligence ambiante dans nos villes, afin de les rendre plus intelligentes et de se diriger vers une société interconnectée

Privacy aware human action recognition: an exploration of temporal salience modelling and neuromorphic vision sensing

Solving the issue of privacy in the application of vision-based home monitoring has emerged as a significant demand. The state-of-the-art studies contain advanced privacy protection by filtering/covering the most sensitive content, which is the identity in this scenario. However, going beyond privacy remains a challenge for the machine to explore the obfuscated data, i.e., utility. Thanks for the usefulness of exploring the human visual system to solve the problem of visual data. Nowadays, a high level of visual abstraction can be obtained from the visual scene by constructing saliency maps that highlight the most useful content in the scene and attenuate others. One way of maintaining privacy with keeping useful information about the action is by discovering the most significant region and removing the redundancy. Another solution to address the privacy is motivated by the new visual sensor technology, i.e., neuromorphic vision sensor. In this thesis, we first introduce a novel method for vision-based privacy preservation. Particularly, we propose a new temporal salience-based anonymisation method to preserve privacy with maintaining the usefulness of the anonymity domain-based data. This anonymisation method has achieved a high level of privacy compared to the current work. The second contribution involves the development of a new descriptor for human action recognition (HAR) based on exploring the anonymity domain of the temporal salience method. The proposed descriptor tests the utility of the anonymised data without referring to RGB intensities of the original data. The extracted features using our proposed descriptor have shown an improvement with accuracies of the human actions, outperforming the existing methods. The proposed method has shown improvements by 3.04%, 3.14%, 0.83%, 3.67%, and 16.71% for DHA, KTH, UIUC1, UCF sports, and HMDB51 datasets, respectively, compared to state-of-the-art methods. The third contribution focuses on proposing a new method to deal with the new neuromorphic vision domain, which has come up to the application, since the issue of privacy has been already solved by the sensor itself. The output of this new domain is exploited by further exploring the local and global details of the log intensity changes. The empirical evaluation shows that exploring the neuromorphic domain provides useful details that have demonstrated increasing accuracy rates for E-KTH, E-UCF11 and E-HMDB5 by 0.54%, 19.42% and 25.61%, respectively