Explainable pattern modelling and summarization in sensor equipped smart homes of elderly

In the next several decades, the proportion of the elderly population is expected to increase significantly. This has led to various efforts to help live them independently for longer periods of time. Smart homes equipped with sensors provide a potential solution by capturing various behavioral and physiological patterns of the residents. In this work, we develop techniques to model and detect changes in these patterns. The focus is on methods that are explainable in nature and allow for generating natural language descriptions. We propose a comprehensive change description framework that can detect unusual changes in the sensor parameters and describe the data leading to those changes in natural language. An approach that models and detects variations in physiological and behavioral routines of the elderly forms one part of the change description framework. The second part comes from a natural language generation system in which we identify important health-relevant features from the sensor parameters. Throughout this dissertation, we validate the developed techniques using both synthetic and real data obtained from the homes of the elderly living in sensor-equipped facilities. Using multiple real data retrospective case studies, we show that our methods are able to detect variations in the sensor data that are correlated with important health events in the elderly as recorded in their Electronic Health Records.Includes bibliographical reference

A Big Bang–Big Crunch Type-2 Fuzzy Logic System for Machine-Vision-Based Event Detection and Summarization in Real-World Ambient-Assisted Living

The area of ambient-assisted living (AAL) focuses on developing new technologies, which can improve the quality of life and care provided to elderly and disabled people. In this paper, we propose a novel system based on 3-D RGB-D vision sensors and interval type-2 fuzzy-logic-based systems (IT2FLSs) employing the big bang-big crunch algorithm for the real-time automatic detection and summarization of important events and human behaviors from the large-scale data. We will present several real-world experiments, which were conducted for AAL-related behaviors with various users. It will be shown that the proposed BB-BC IT2FLSs outperform the type-1 fuzzy logic system counterparts as well as other conventional nonfuzzy methods, and the performance improves when the number of subjects increases

A Big Bang Big Crunch Type-2 Fuzzy Logic System for Machine Vision-Based Event Detection and Summarization in Real-world Ambient Assisted Living

The recent years have witnessed the prevalence and abundance of vision sensors in various applications such as security surveillance, healthcare and Ambient Assisted Living (AAL) among others. This is so as to realize intelligent environments which are capable of detecting users’ actions and gestures so that the needed services can be provided automatically and instantly to maximize user comfort and safety as well as to minimize energy. However, it is very challenging to automatically detect important events and human behaviour from vision sensors and summarize them in real time. This is due to the massive data sizes related to video analysis applications and the high level of uncertainties associated with the real world unstructured environments occupied by various users. Machine vision based systems can help detect and summarize important information which cannot be detected by any other sensor; for example, how much water a candidate drank and whether or not they had something to eat. However, conventional non-fuzzy based methods are not robust enough to recognize the various complex types of behaviour in AAL applications. Fuzzy logic system (FLS) is an established field of research to robustly handle uncertainties in complicated real-world problems. In this thesis, we will present a general recognition and classification framework based on fuzzy logic systems which allows for behaviour recognition and event summarisation using 2D/3D video sensors in AAL applications. I started by investigating the use of 2D CCTV camera based system where I proposed and developed novel IT2FLS-based methods for silhouette extraction and 2D behaviour recognition which outperform the traditional on the publicly available Weizmann human action dataset. I will also present a novel system based on 3D RGB-D vision sensors and Interval Type-2 Fuzzy Logic based Systems (IT2FLSs) ) generated by the Big Bang Big Crunch (BB-BC) algorithm for the real time automatic detection and summarization of important events and human behaviour. I will present several real world experiments which were conducted for AAL related behaviour with various users. It will be shown that the proposed BB-BC IT2FLSs outperforms its Type-1 FLSs (T1FLSs) counterpart as well as other conventional non-fuzzy methods, and that performance improvement rises when the number of subjects increases. It will be shown that by utilizing the recognized output activity together with relevant event descriptions (such as video data, timestamp, location and user identification) detailed events are efficiently summarized and stored in our back-end SQL event database, which provides services including event searching, activity retrieval and high-definition video playback to the front-end user interfaces

Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation

This paper surveys the current state of the art in Natural Language Generation (NLG), defined as the task of generating text or speech from non-linguistic input. A survey of NLG is timely in view of the changes that the field has undergone over the past decade or so, especially in relation to new (usually data-driven) methods, as well as new applications of NLG technology. This survey therefore aims to (a) give an up-to-date synthesis of research on the core tasks in NLG and the architectures adopted in which such tasks are organised; (b) highlight a number of relatively recent research topics that have arisen partly as a result of growing synergies between NLG and other areas of artificial intelligence; (c) draw attention to the challenges in NLG evaluation, relating them to similar challenges faced in other areas of Natural Language Processing, with an emphasis on different evaluation methods and the relationships between them.Comment: Published in Journal of AI Research (JAIR), volume 61, pp 75-170. 118 pages, 8 figures, 1 tabl

Early detection of health changes in the elderly using in-home multi-sensor data streams

The rapid aging of the population worldwide requires increased attention from health care providers and the entire society. For the elderly to live independently, many health issues related to old age, such as frailty and risk of falling, need increased attention and monitoring. When monitoring daily routines for older adults, it is desirable to detect the early signs of health changes before serious health events, such as hospitalizations, happen, so that timely and adequate preventive care may be provided. By deploying multi-sensor systems in homes of the elderly, we can track trajectories of daily behaviors in a feature space defined using the sensor data. In this work, we investigate a methodology for learning data distribution from streaming data and tracking the evolution of the behavior trajectories over long periods (years) using high dimensional streaming clustering and provide very early indicators of changes in health. If we assume that habitual behaviors correspond to clusters in feature space and diseases produce a change in behavior, albeit not highly specific, tracking trajectory deviations can provide hints of early illness. Retrospectively, we visualize the streaming clustering results and track how the behavior clusters evolve in feature space with the help of two dimension-reduction algorithms, Principal Component Analysis (PCA) and t-distributed Stochastic Neighbor Embedding (t-SNE). Moreover, our tracking algorithm in the original high dimensional feature space generates early health warning alerts if a negative trend is detected in the behavior trajectory. We validated our algorithm on synthetic data, real-world data and tested it on a pilot dataset of four TigerPlace residents monitored with a collection of motion, bed, and depth sensors over ten years. We used the TigerPlace electronic health records (EHR) to understand the residents' behavior patterns and to evaluate and explain the health warnings generated by our algorithm. The results obtained on the TigerPlace dataset show that most of the warnings produced by our algorithm can be linked to health events documented in the EHR, providing strong support for a prospective deployment of the approach.Includes bibliographical references

Addendum to Informatics for Health 2017: Advancing both science and practice

This article presents presentation and poster abstracts that were mistakenly omitted from the original publication

Visual Sensor Based Abnormal Event Detection with Moving Shadow Removal in Home Healthcare Applications

Vision-based abnormal event detection for home healthcare systems can be greatly improved using visual sensor-based techniques able to detect, track and recognize objects in the scene. However, in moving object detection and tracking processes, moving cast shadows can be misclassified as part of objects or moving objects. Shadow removal is an essential step for developing video surveillance systems. The goal of the primary is to design novel computer vision techniques that can extract objects more accurately and discriminate between abnormal and normal activities. To improve the accuracy of object detection and tracking, our proposed shadow removal algorithm is employed. Abnormal event detection based on visual sensor by using shape features variation and 3-D trajectory is presented to overcome the low fall detection rate. The experimental results showed that the success rate of detecting abnormal events was 97% with a false positive rate of 2%. Our proposed algorithm can allow distinguishing diverse fall activities such as forward falls, backward falls, and falling asides from normal activities