Behavioural modelling for ambient assisted living

Tese de doutoramento - MAP-i (University of Minho, Aveiro, and Porto)A mudança incomum na rotina diária ao nível da mobilidade de um idoso em sua casa, pode ser um sinal ou sintoma precoce para a possibilidade de vir a desenvolver um problema de saúde. O recurso a diferentes sensores pode ser um meio para complementar os sistemas de cuidados de saúde tradicionais, de forma a obter uma visão mais detalhada da mobilidade diária do individuo em sua casa, enquanto realiza as suas tarefas diárias. Acreditamos, que os dados recolhidos a partir de sensores de baixo custo, como sensores de presença e ocupação, podem ser utilizados para fornecer evidências sobre os hábitos diários de mobilidade dos idosos que vivem sozinhos em casa e detetar desta forma mudanças nas suas rotinas. Neste trabalho, validamos esta hipótese, desenvolvendo um sistema que aprende automaticamente as transições diárias entre divisões da habitação e hábitos de estadia em cada uma dessas divisões em cada momento do dia e consequentemente gera alarmes sempre que os desvios são detetados. Apresentamos neste trabalho um algoritmo que processa os fluxos de dados dos diferentes sensores e identifica características que descrevem a rotina diária de mobilidade de um idoso que vive sozinho em casa. Para isso foi definido um conjunto de dimensões baseadas nos dados extraídos dos sensores, como parte do nosso Behaviour Monitoring System (BMS). Fomos capazes de detetar com um atraso mínimo os comportamentos incomuns e ao mesmo tempo, durações de confirmação da deteção elevadas, de tal modo suficientes para um conjunto comum de situações anormais. Apresentamos e avaliamos o BMS com dados sintetizados, produzidos por um gerador de dados desenvolvido para este efeito e projetado para simular diferentes perfis de mobilidade de indivíduos em casa, e também com dados reais obtidos de trabalhos de investigação anteriores. Os resultados indicam que o BMS deteta várias mudanças de mobilidade que podem ser sintomas para problemas de saúde comuns. O sistema proposto é uma abordagem útil para a aprendizagem dos hábitos de mobilidade em ambientes domésticos, com potencial para detetar alterações comportamentais que ocorrem devido a problemas de saúde, e assim encorajar a monitorização dos comportamentos e dos cuidados de saúde dos idosos.Unusual changes in the regular daily mobility routine of an elderly at home can be an indicator or early symptoms for developing a health problem. Sensor technology can be utilised to complement the traditional healthcare systems to gain a more detailed view of the daily mobility of a person at home when performing everyday tasks. We hypothesise that data collected from low-cost sensors such as presence and occupancy sensors can be analysed to provide insights on the daily mobility habits of the elderly living alone at home and to detect routine changes. We validate this hypothesis by designing a system that automatically learns the daily room-to-room transitions and stays habits in each room at each time of the day and generates alarm notifications when deviations are detected. We present an algorithm to process the sensor data streams and compute features that describe the daily mobility routine of an elderly living alone at home. This was done by defining a set of sensor-driven dimensions extracted from the sensor data as part of our Behaviour Monitoring System (BMS). We are able to achieve low detection delay with confirmation time that is high enough to convey the detection of a set of common abnormal situations. We illustrate and evaluate BMS with synthetic data, generated by a developed data generator that was designed to mimic different users’ mobility profiles at home, and also with real-life dataset collected from prior research work. Results indicate BMS detects several mobility changes that can be symptoms of common health problems. The proposed system is a useful approach for learning the mobility habits at home environments, with the potential to detect behaviour changes that occur due to health problems, and therefore, motivating progress toward behaviour monitoring and elder’s care

Deep learning-based automatic analysis of social interactions from wearable data for healthcare applications

PhD ThesisSocial interactions of people with Late Life Depression (LLD) could be an objective measure of social functioning due to the association between LLD and poor social functioning. The utilisation of wearable computing technologies is a relatively new approach within healthcare and well-being application sectors. Recently, the design and development of wearable technologies and systems for health and well-being monitoring have attracted attention both of the clinical and scientific communities. Mainly because the current clinical practice of – typically rather sporadic – clinical behaviour assessments are often administered in artificial settings. As a result, it does not provide a realistic impression of a patient’s condition and thus does not lead to sufficient diagnosis and care. However, wearable behaviour monitors have the potential for continuous, objective assessment of behaviour and wider social interactions and thereby allowing for capturing naturalistic data without any constraints on the place of recording or any typical limitations of the lab-setting research. Such data from naturalistic ambient environments would facilitate automated transmission and analysis by having no constraints on the recordings, allowing for a more timely and accurate assessment of depressive symptoms. In response to this artificial setting issue, this thesis focuses on the analysis and assessment of the different aspects of social interactions in naturalistic environments using deep learning algorithms. That could lead to improvements in both diagnosis and treatment. The advantages of using deep learning are that there is no need for hand-crafted features engineering and this leads to using the raw data with minimal pre-processing compared to classical machine learning approaches and also its scalability and ability to generalise. The main dataset used in this thesis is recorded by a wrist worn device designed at Newcastle University. This device has multiple sensors including microphone, tri-axial accelerometer, light sensor and proximity sensor. In this thesis, only microphone and tri-axial accelerometer are used for the social interaction analysis. The other sensors are not used since they need more calibration from the user which in this will be the elderly people with depression. Hence, it was not feasible in this scenario. Novel deep learning models are proposed to automatically analyse two aspects of social interactions (the verbal interactions/acoustic communications and physical activities/movement patterns). Verbal Interactions include the total quantity of speech, who is talking to whom and when and how much engagement the wearer contributed in the conversations. The physical activity analysis includes activity recognition and the quantity of each activity and sleep patterns. This thesis is composed of three main stages, two of them discuss the acoustic analysis and the third stage describes the movement pattern analysis. The acoustic analysis starts with speech detection in which each segment of the recording is categorised as speech or non-speech. This segment classification is achieved by a novel deep learning model that leverages bi-directional Long Short-Term Memory with gated activation units combined with Maxout Networks as well as a combination of two optimisers. After detecting speech segments from audio data, the next stage is detecting how much engagement the wearer has in any conversation throughout these speech events based on detecting the wearer of the device using a variant model of the previous one that combines the convolutional autoencoder with bi-directional Long Short-Term Memory. Following this, the system then detects the spoken parts of the main speaker/wearer and therefore detects the conversational turn-taking but only includes the turn taking between the wearer and other speakers and not every speaker in the conversation. This stage did not take into account the semantics of the speakers due to the ethical constraints of the main dataset (Depression dataset) and therefore it was not possible to listen to the data by any means or even have any information about the contents. So, it is a good idea to be considered for future work. Stage 3 involves the physical activity analysis that is inferring the elementary physical activities and movement patterns. These elementary patterns include sedentary actions, walking, mixed activities, cycling, using vehicles as well as the sleep patterns. The predictive model used is based on Random Forests and Hidden Markov Models. In all stages the methods presented in this thesis have been compared to the state-of-the-art in processing audio, accelerometer data, respectively, to thoroughly assess their contribution. Following these stages is a thorough analysis of the interplay between acoustic interaction and physical movement patterns and the depression key clinical variables resulting to the outcomes of the previous stages. The main reason for not using deep learning in this stage unlike the previous stages is that the main dataset (Depression dataset) did not have any annotations for the speech or even the activity due to the ethical constraints as mentioned. Furthermore, the training dataset (Discussion dataset) did not have any annotations for the accelerometer data where the data is recorded freely and there is no camera attached to device to make it possible to be annotated afterwards.Newton-Mosharafa Fund and the mission sector and cultural affairs, ministry of Higher Education in Egypt