Combining ontological and temporal formalisms for composite activity modelling and recognition in smart homes

The file attached to this record is the author's final peer reviewed version. The Publisher's final version can be found by following the DOI link.Activity recognition is essential in providing activity assistance for users in smart homes. While significant progress has been made for single-user single-activity recognition, it still remains a challenge to carry out real-time progressive composite activity recognition. This paper introduces a hybrid ontological and temporal approach to composite activity modelling and recognition by extending existing ontology-based knowledge-driven approach. The compelling feature of the approach is that it combines ontological and temporal knowledge representation formalisms to provide powerful representation capabilities for activity modelling. The paper describes in detail ontological activity modelling which establishes relationships between activities and their involved entities, and temporal activity modelling which defines relationships between constituent activities of a composite activity. As an essential part of the model, the paper also presents methods for developing temporal entailment rules to support the interpretation and inference of composite activities. In addition, this paper outlines an integrated architecture for composite activity recognition and elaborated a unified activity recognition algorithm which can support the recognition of simple and composite activities. The approach has been implemented in a feature-rich prototype system upon which testing and evaluation have been conducted. Initial experimental results have shown average recognition accuracy of 100% and 88.26% for simple and composite activities, respectively

Multioccupant Activity Recognition in Pervasive Smart Home Environments

been the center of lot of research for many years now. The aim is to recognize the sequence of actions by a specific person using sensor readings. Most of the research has been devoted to activity recognition of single occupants in the environment. However, living environments are usually inhabited by more than one person and possibly with pets. Hence, human activity recognition in the context of multi-occupancy is more general, but also more challenging. The difficulty comes from mainly two aspects: resident identification, known as data association, and diversity of human activities. The present survey paper provides an overview of existing approaches and current practices for activity recognition in multi-occupant smart homes. It presents the latest developments and highlights the open issues in this field

A knowledge-based approach towards human activity recognition in smart environments

For many years it is known that the population of older persons is on the rise. A recent report estimates that globally, the share of the population aged 65 years or over is expected to increase from 9.3 percent in 2020 to around 16.0 percent in 2050 [1]. This point has been one of the main sources of motivation for active research in the domain of human activity recognition in smart-homes. The ability to perform ADL without assistance from other people can be considered as a reference for the estimation of the independent living level of the older person. Conventionally, this has been assessed by health-care domain experts via a qualitative evaluation of the ADL. Since this evaluation is qualitative, it can vary based on the person being monitored and the caregiver\u2019s experience. A significant amount of research work is implicitly or explicitly aimed at augmenting the health-care domain expert\u2019s qualitative evaluation with quantitative data or knowledge obtained from HAR. From a medical perspective, there is a lack of evidence about the technology readiness level of smart home architectures supporting older persons by recognizing ADL [2]. We hypothesize that this may be due to a lack of effective collaboration between smart-home researchers/developers and health-care domain experts, especially when considering HAR. We foresee an increase in HAR systems being developed in close collaboration with caregivers and geriatricians to support their qualitative evaluation of ADL with explainable quantitative outcomes of the HAR systems. This has been a motivation for the work in this thesis. The recognition of human activities \u2013 in particular ADL \u2013 may not only be limited to support the health and well-being of older people. It can be relevant to home users in general. For instance, HAR could support digital assistants or companion robots to provide contextually relevant and proactive support to the home users, whether young adults or old. This has also been a motivation for the work in this thesis. Given our motivations, namely, (i) facilitation of iterative development and ease in collaboration between HAR system researchers/developers and health-care domain experts in ADL, and (ii) robust HAR that can support digital assistants or companion robots. There is a need for the development of a HAR framework that at its core is modular and flexible to facilitate an iterative development process [3], which is an integral part of collaborative work that involves develop-test-improve phases. At the same time, the framework should be intelligible for the sake of enriched collaboration with health-care domain experts. Furthermore, it should be scalable, online, and accurate for having robust HAR, which can enable many smart-home applications. The goal of this thesis is to design and evaluate such a framework. This thesis contributes to the domain of HAR in smart-homes. Particularly the contribution can be divided into three parts. The first contribution is Arianna+, a framework to develop networks of ontologies - for knowledge representation and reasoning - that enables smart homes to perform human activity recognition online. The second contribution is OWLOOP, an API that supports the development of HAR system architectures based on Arianna+. It enables the usage of Ontology Web Language (OWL) by the means of Object-Oriented Programming (OOP). The third contribution is the evaluation and exploitation of Arianna+ using OWLOOP API. The exploitation of Arianna+ using OWLOOP API has resulted in four HAR system implementations. The evaluations and results of these HAR systems emphasize the novelty of Arianna+

On Leveraging Statistical and Relational Information for the Representation and Recognition of Complex Human Activities

Machine activity recognition aims to automatically predict human activities from a series of sensor signals. It is a key aspect to several emerging applications, especially in the pervasive computing field. However, this problem faces several challenges due to the complex, relational and ambiguous nature of human activities. These challenges still defy the majority of traditional pattern recognition approaches, whether they are knowledge-based or data-driven. Concretely, the current approaches to activity recognition in sensor environments fall short to represent, reason or learn under uncertainty, complex relational structure, rich temporal context and abundant common-sense knowledge. Motivated by these shortcomings, our work focuses on the combination of both data-driven and knowledge-based paradigms in order to address this problem. In particular, we propose two logic-based statistical relational activity recognition frameworks which we describe in two different parts. The first part presents a Markov logic-based framework addressing the recognition of complex human activities under realistic settings. Markov logic is a highly flexible statistical relational formalism combining the power of first-order logic with Markov networks by attaching real-valued weights to formulas in first-order logic. Thus, it unites both symbolic and probabilistic reasoning and allows to model the complex relational structure as well as the inherent uncertainty underlying human activities and sensor data. We focus on addressing the challenge of recognizing interleaved and concurrent activities while preserving the intuitiveness and flexibility of the modelling task. Using three different models we evaluate and prove the viability of using Markov logic networks for that problem statement. We also demonstrate the crucial impact of domain knowledge on the recognition outcome. Implementing an exhaustive model including heterogeneous information sources comes, however, at considerable knowledge engineering efforts. Hence, employing a standard, widely used formalism can alleviate that by enhancing the portability, the re-usability and the extension of the model. In the second part of this document, we apply a hybrid approach that goes one step further than Markov logic network towards a formal, yet intuitive conceptualization of the domain of discourse. Concretely, we propose an activity recognition framework based on log-linear description logic, a probabilistic variant of description logics. Log-linear description logic leverages the principles of Markov logic while allowing for a formal conceptualization of the domain of discourse, backed up with powerful reasoning and consistency check tools. Based on principles from the activity theory, we focus on addressing the challenge of representing and recognizing human activities at three levels of granularity: operations, actions and activities. Complying with real-life scenarios, we assess and discuss the viability of the proposed framework. In particular, we show the positive impact of augmenting the proposed multi-level activity ontology with weights compared to using its conventional weight-free variant

Sensor-based early activity recognition inside buildings to support energy and comfort management systems

Building Energy and Comfort Management (BECM) systems have the potential to considerably reduce costs related to energy consumption and improve the efficiency of resource exploitation, by implementing strategies for resource management and control and policies for Demand-Side Management (DSM). One of the main requirements for such systems is to be able to adapt their management decisions to the users’ specific habits and preferences, even when they change over time. This feature is fundamental to prevent users’ disaffection and the gradual abandonment of the system. In this paper, a sensor-based system for analysis of user habits and early detection and prediction of user activities is presented. To improve the resulting accuracy, the system incorporates statistics related to other relevant external conditions that have been observed to be correlated (e.g., time of the day). Performance evaluation on a real use case proves that the proposed system enables early recognition of activities after only 10 sensor events with an accuracy of 81%. Furthermore, the correlation between activities can be used to predict the next activity with an accuracy of about 60%

Measurement of Multitasking with Focus Shift Analysis

Perhaps one of the most prevalent, yet least understood, forms of modern behavior is human multitasking aided by technology. Multitasking is both difficult to define and to measure. Based on Activity Theory and the concept of focus shifts, this study defines multitasking as shifts that occur when the flow of work is interrupted and tasks are interchanged during a session. The analysis of focus shift diversity, with respect to the tasks that receive attention, enables researchers to measure multitasking. With this approach, several indices of diversity are examined in order to provide recommendations about alternative measurement choices. This methodological study advances our understanding of the possibilities and limitations of using diversity indices for measuring multitasking

A situation-driven framework for relearning of activities of daily living in smart home environments

Activities of Daily Living (ADLs) are sine qua non for self-care and improved quality of life. Self-efficacy is major challenge for seniors with early-stage dementia (ED) when performing daily living activities. ED causes deterioration of cognitive functions and thus impacts aging adults’ functioning initiative and performance of instrumental activities of daily living (IADLs). Generally, IADLs requires certain skills in both planning and execution and may involve sequence of steps for aging adults to accomplish their goals. These intricate procedures in IADLs potentially predispose older adults to safety-critical situations with life-threatening consequences. A safety-critical situation is a state or event that potentially constitutes a risk with life-threatening injuries or accidents. To address this problem, a situation-driven framework for relearning of daily living activities in smart home environment is proposed. The framework is composed of three (3) major units namely: a) goal inference unit – leverages a deep learning model to infer human goal in a smart home, b) situation-context generator – responsible for risk mitigation in IADLs, and c) a recommendation unit – to support decision making of aging adults in safety-critical situations. The proposed framework was validated against IADLs dataset collected from a smart home research prototype and the results obtained are promising