ProCAVIAR: Hybrid Data-Driven and Probabilistic Knowledge-Based Activity Recognition

The recognition of physical activities using sensors on mobile devices has been mainly addressed with supervised and semi-supervised learning. The state-of-the-art methods are mainly based on the analysis of the user\u2019s movement patterns that emerge from inertial sensors data. While the literature on this topic is quite mature, existing approaches are still not adequate to discriminate activities characterized by similar physical movements. The context that surrounds the user (e.g., semantic location) could be used as additional information to significantly extend the set of recognizable activities. Since collecting a comprehensive training set with activities performed in every possible context condition is too costly, if possible at all, existing works proposed knowledge-based reasoning over ontological representation of context data to refine the predictions obtained from machine learning. A problem with this approach is the rigidity of the underlying logic formalism that cannot capture the intrinsic uncertainty of the relationships between activities and context. In this work, we propose a novel activity recognition method that combines semisupervised learning and probabilistic ontological reasoning. We model the relationships between activities and context as a combination of soft and hard ontological axioms. For each activity, we use a probabilistic ontology to compute its compatibility with the current context conditions. The output of probabilistic semantic reasoning is combined with the output of a machine learning classifier based on inertial sensor data to obtain the most likely activity performed by the user. The evaluation of our system on a dataset with 13 types of activities performed by 26 subjects shows that our probabilistic framework outperforms both a pure machine learning approach and previous hybrid approaches based on classic ontological reasoning

Integration of Legacy Appliances into Home Energy Management Systems

The progressive installation of renewable energy sources requires the coordination of energy consuming devices. At consumer level, this coordination can be done by a home energy management system (HEMS). Interoperability issues need to be solved among smart appliances as well as between smart and non-smart, i.e., legacy devices. We expect current standardization efforts to soon provide technologies to design smart appliances in order to cope with the current interoperability issues. Nevertheless, common electrical devices affect energy consumption significantly and therefore deserve consideration within energy management applications. This paper discusses the integration of smart and legacy devices into a generic system architecture and, subsequently, elaborates the requirements and components which are necessary to realize such an architecture including an application of load detection for the identification of running loads and their integration into existing HEM systems. We assess the feasibility of such an approach with a case study based on a measurement campaign on real households. We show how the information of detected appliances can be extracted in order to create device profiles allowing for their integration and management within a HEMS

Ontology-based Activity Recognition Framework and Services

This paper introduces an ontology-based integrated framework for activity modeling, activity recognition and activity model evolution. Central to the framework is ontological activity modeling and semantic-based activity recognition, which is supported by an iterative process that incrementally improves the completeness and accuracy of activity models. In addition, the paper presents a service-oriented architecture for the realization of the proposed framework which can provide activity context-aware services in a scalable distributed manner. The paper further describes and discusses the implementation and testing experience of the framework and services in the context of smart home based assistive living

Context-Aware Personalized Activity Modeling in Concurrent Environment

Activity recognition, having endemic impact on smart homes, faces one of the biggest challenges in learning a personalized activity model completely by using a generic model especially for parallel and interleaved activities. Furthermore, inhabitant’s mistaken object interaction may entail in another spurious activity at smart homes. Identifying and removing such spurious activities is another challenging task. Knowledge driven techniques used for recognizing activity models are static in nature, lack contextual representation and may not comprehend spurious actions for parallel/interleaved activities. In this paper, a novel approach for completing the personalized model specific to each inhabitant at smart homes using generic model (incomplete) is presented that can recognize the sequential, parallel, and interleaved activities dynamically while removing the spurious activities semantically. A comprehensive set of experiments and results based upon number of correct (true positivity) or incorrect (false negativity) recognition of activities assert effectiveness of presented approach within a smart hom

Combining ontological and temporal formalisms for composite activity modelling and recognition in smart homes

The file attached to this record is the author's final peer reviewed version. The Publisher's final version can be found by following the DOI link.Activity recognition is essential in providing activity assistance for users in smart homes. While significant progress has been made for single-user single-activity recognition, it still remains a challenge to carry out real-time progressive composite activity recognition. This paper introduces a hybrid ontological and temporal approach to composite activity modelling and recognition by extending existing ontology-based knowledge-driven approach. The compelling feature of the approach is that it combines ontological and temporal knowledge representation formalisms to provide powerful representation capabilities for activity modelling. The paper describes in detail ontological activity modelling which establishes relationships between activities and their involved entities, and temporal activity modelling which defines relationships between constituent activities of a composite activity. As an essential part of the model, the paper also presents methods for developing temporal entailment rules to support the interpretation and inference of composite activities. In addition, this paper outlines an integrated architecture for composite activity recognition and elaborated a unified activity recognition algorithm which can support the recognition of simple and composite activities. The approach has been implemented in a feature-rich prototype system upon which testing and evaluation have been conducted. Initial experimental results have shown average recognition accuracy of 100% and 88.26% for simple and composite activities, respectively

A Modified KNN Algorithm for Activity Recognition in Smart Home

Nowadays, more and more elderly people cannot take care of themselves, and feel uncomfortable in daily activities. Smart home systems can help to improve daily life of elderly people. A smart home can bring residents a more comfortable living environment by recognizing the daily activities automatically. In this paper, in order to improve the accuracy of activity recognition in smart homes, we conduct some improvements in data preprocess and recognition phase, and more importantly, a novel sensor segmentation method and a modified KNN algorithm are proposed. The segmentation algorithm employs segment sensor data into fragments based on predefined activity knowledge, and then the proposed modified KNN algorithm uses center distances as a measure for classification. We also conduct comprehensive experiments, and the results demonstrate that the proposed method outperforms the other classifiers

Handling Real-World Context Awareness, Uncertainty and Vagueness in Real-Time Human Activity Tracking and Recognition with a Fuzzy Ontology-Based Hybrid Method

Human activity recognition is a key task in ambient intelligence applications to achieve proper ambient assisted living. There has been remarkable progress in this domain, but some challenges still remain to obtain robust methods. Our goal in this work is to provide a system that allows the modeling and recognition of a set of complex activities in real life scenarios involving interaction with the environment. The proposed framework is a hybrid model that comprises two main modules: a low level sub-activity recognizer, based on data-driven methods, and a high-level activity recognizer, implemented with a fuzzy ontology to include the semantic interpretation of actions performed by users. The fuzzy ontology is fed by the sub-activities recognized by the low level data-driven component and provides fuzzy ontological reasoning to recognize both the activities and their influence in the environment with semantics. An additional benefit of the approach is the ability to handle vagueness and uncertainty in the knowledge-based module, which substantially outperforms the treatment of incomplete and/or imprecise data with respect to classic crisp ontologies. We validate these advantages with the public CAD-120 dataset (Cornell Activity Dataset), achieving an accuracy of 90.1% and 91.07% for low-level and high-level activities, respectively. This entails an improvement over fully data-driven or ontology-based approaches.This work was funded by TUCS (Turku Centre for Computer Science), Finnish Cultural Foundation, Nokia Foundation, Google Anita Borg Scholarship, CEI BioTIC Project CEI2013-P-3, Contrato-Programa of Faculty of Education, Economy and Technology of Ceuta and Project TIN2012-30939 from National I+D Research Program (Spain). We also thank Fernando Bobillo for his support with FuzzyOWL and FuzzyDL tools

Inferring Complex Activities for Context-aware Systems within Smart Environments

The rising ageing population worldwide and the prevalence of age-related conditions such as physical fragility, mental impairments and chronic diseases have significantly impacted the quality of life and caused a shortage of health and care services. Over-stretched healthcare providers are leading to a paradigm shift in public healthcare provisioning. Thus, Ambient Assisted Living (AAL) using Smart Homes (SH) technologies has been rigorously investigated to help address the aforementioned problems. Human Activity Recognition (HAR) is a critical component in AAL systems which enables applications such as just-in-time assistance, behaviour analysis, anomalies detection and emergency notifications. This thesis is aimed at investigating challenges faced in accurately recognising Activities of Daily Living (ADLs) performed by single or multiple inhabitants within smart environments. Specifically, this thesis explores five complementary research challenges in HAR. The first study contributes to knowledge by developing a semantic-enabled data segmentation approach with user-preferences. The second study takes the segmented set of sensor data to investigate and recognise human ADLs at multi-granular action level; coarse- and fine-grained action level. At the coarse-grained actions level, semantic relationships between the sensor, object and ADLs are deduced, whereas, at fine-grained action level, object usage at the satisfactory threshold with the evidence fused from multimodal sensor data is leveraged to verify the intended actions. Moreover, due to imprecise/vague interpretations of multimodal sensors and data fusion challenges, fuzzy set theory and fuzzy web ontology language (fuzzy-OWL) are leveraged. The third study focuses on incorporating uncertainties caused in HAR due to factors such as technological failure, object malfunction, and human errors. Hence, existing studies uncertainty theories and approaches are analysed and based on the findings, probabilistic ontology (PR-OWL) based HAR approach is proposed. The fourth study extends the first three studies to distinguish activities conducted by more than one inhabitant in a shared smart environment with the use of discriminative sensor-based techniques and time-series pattern analysis. The final study investigates in a suitable system architecture with a real-time smart environment tailored to AAL system and proposes microservices architecture with sensor-based off-the-shelf and bespoke sensing methods. The initial semantic-enabled data segmentation study was evaluated with 100% and 97.8% accuracy to segment sensor events under single and mixed activities scenarios. However, the average classification time taken to segment each sensor events have suffered from 3971ms and 62183ms for single and mixed activities scenarios, respectively. The second study to detect fine-grained-level user actions was evaluated with 30 and 153 fuzzy rules to detect two fine-grained movements with a pre-collected dataset from the real-time smart environment. The result of the second study indicate good average accuracy of 83.33% and 100% but with the high average duration of 24648ms and 105318ms, and posing further challenges for the scalability of fusion rule creations. The third study was evaluated by incorporating PR-OWL ontology with ADL ontologies and Semantic-Sensor-Network (SSN) ontology to define four types of uncertainties presented in the kitchen-based activity. The fourth study illustrated a case study to extended single-user AR to multi-user AR by combining RFID tags and fingerprint sensors discriminative sensors to identify and associate user actions with the aid of time-series analysis. The last study responds to the computations and performance requirements for the four studies by analysing and proposing microservices-based system architecture for AAL system. A future research investigation towards adopting fog/edge computing paradigms from cloud computing is discussed for higher availability, reduced network traffic/energy, cost, and creating a decentralised system. As a result of the five studies, this thesis develops a knowledge-driven framework to estimate and recognise multi-user activities at fine-grained level user actions. This framework integrates three complementary ontologies to conceptualise factual, fuzzy and uncertainties in the environment/ADLs, time-series analysis and discriminative sensing environment. Moreover, a distributed software architecture, multimodal sensor-based hardware prototypes, and other supportive utility tools such as simulator and synthetic ADL data generator for the experimentation were developed to support the evaluation of the proposed approaches. The distributed system is platform-independent and currently supported by an Android mobile application and web-browser based client interfaces for retrieving information such as live sensor events and HAR results

Proceedings of the 2nd IUI Workshop on Interacting with Smart Objects

These are the Proceedings of the 2nd IUI Workshop on Interacting with Smart Objects. Objects that we use in our everyday life are expanding their restricted interaction capabilities and provide functionalities that go far beyond their original functionality. They feature computing capabilities and are thus able to capture information, process and store it and interact with their environments, turning them into smart objects