Inferring Complex Activities for Context-aware Systems within Smart Environments

The rising ageing population worldwide and the prevalence of age-related conditions such as physical fragility, mental impairments and chronic diseases have significantly impacted the quality of life and caused a shortage of health and care services. Over-stretched healthcare providers are leading to a paradigm shift in public healthcare provisioning. Thus, Ambient Assisted Living (AAL) using Smart Homes (SH) technologies has been rigorously investigated to help address the aforementioned problems. Human Activity Recognition (HAR) is a critical component in AAL systems which enables applications such as just-in-time assistance, behaviour analysis, anomalies detection and emergency notifications. This thesis is aimed at investigating challenges faced in accurately recognising Activities of Daily Living (ADLs) performed by single or multiple inhabitants within smart environments. Specifically, this thesis explores five complementary research challenges in HAR. The first study contributes to knowledge by developing a semantic-enabled data segmentation approach with user-preferences. The second study takes the segmented set of sensor data to investigate and recognise human ADLs at multi-granular action level; coarse- and fine-grained action level. At the coarse-grained actions level, semantic relationships between the sensor, object and ADLs are deduced, whereas, at fine-grained action level, object usage at the satisfactory threshold with the evidence fused from multimodal sensor data is leveraged to verify the intended actions. Moreover, due to imprecise/vague interpretations of multimodal sensors and data fusion challenges, fuzzy set theory and fuzzy web ontology language (fuzzy-OWL) are leveraged. The third study focuses on incorporating uncertainties caused in HAR due to factors such as technological failure, object malfunction, and human errors. Hence, existing studies uncertainty theories and approaches are analysed and based on the findings, probabilistic ontology (PR-OWL) based HAR approach is proposed. The fourth study extends the first three studies to distinguish activities conducted by more than one inhabitant in a shared smart environment with the use of discriminative sensor-based techniques and time-series pattern analysis. The final study investigates in a suitable system architecture with a real-time smart environment tailored to AAL system and proposes microservices architecture with sensor-based off-the-shelf and bespoke sensing methods. The initial semantic-enabled data segmentation study was evaluated with 100% and 97.8% accuracy to segment sensor events under single and mixed activities scenarios. However, the average classification time taken to segment each sensor events have suffered from 3971ms and 62183ms for single and mixed activities scenarios, respectively. The second study to detect fine-grained-level user actions was evaluated with 30 and 153 fuzzy rules to detect two fine-grained movements with a pre-collected dataset from the real-time smart environment. The result of the second study indicate good average accuracy of 83.33% and 100% but with the high average duration of 24648ms and 105318ms, and posing further challenges for the scalability of fusion rule creations. The third study was evaluated by incorporating PR-OWL ontology with ADL ontologies and Semantic-Sensor-Network (SSN) ontology to define four types of uncertainties presented in the kitchen-based activity. The fourth study illustrated a case study to extended single-user AR to multi-user AR by combining RFID tags and fingerprint sensors discriminative sensors to identify and associate user actions with the aid of time-series analysis. The last study responds to the computations and performance requirements for the four studies by analysing and proposing microservices-based system architecture for AAL system. A future research investigation towards adopting fog/edge computing paradigms from cloud computing is discussed for higher availability, reduced network traffic/energy, cost, and creating a decentralised system. As a result of the five studies, this thesis develops a knowledge-driven framework to estimate and recognise multi-user activities at fine-grained level user actions. This framework integrates three complementary ontologies to conceptualise factual, fuzzy and uncertainties in the environment/ADLs, time-series analysis and discriminative sensing environment. Moreover, a distributed software architecture, multimodal sensor-based hardware prototypes, and other supportive utility tools such as simulator and synthetic ADL data generator for the experimentation were developed to support the evaluation of the proposed approaches. The distributed system is platform-independent and currently supported by an Android mobile application and web-browser based client interfaces for retrieving information such as live sensor events and HAR results

Learning preferences for personalisation in a pervasive environment

With ever increasing accessibility to technological devices, services and applications there is also an increasing burden on the end user to manage and configure such resources. This burden will continue to increase as the vision of pervasive environments, with ubiquitous access to a plethora of resources, continues to become a reality. It is key that appropriate mechanisms to relieve the user of such burdens are developed and provided. These mechanisms include personalisation systems that can adapt resources on behalf of the user in an appropriate way based on the user's current context and goals. The key knowledge base of many personalisation systems is the set of user preferences that indicate what adaptations should be performed under which contextual situations. This thesis investigates the challenges of developing a system that can learn such preferences by monitoring user behaviour within a pervasive environment. Based on the findings of related works and experience from EU project research, several key design requirements for such a system are identified. These requirements are used to drive the design of a system that can learn accurate and up to date preferences for personalisation in a pervasive environment. A standalone prototype of the preference learning system has been developed. In addition the preference learning system has been integrated into a pervasive platform developed through an EU research project. The preference learning system is fully evaluated in terms of its machine learning performance and also its utility in a pervasive environment with real end users

Context Aware Computing for The Internet of Things: A Survey

As we are moving towards the Internet of Things (IoT), the number of sensors deployed around the world is growing at a rapid pace. Market research has shown a significant growth of sensor deployments over the past decade and has predicted a significant increment of the growth rate in the future. These sensors continuously generate enormous amounts of data. However, in order to add value to raw sensor data we need to understand it. Collection, modelling, reasoning, and distribution of context in relation to sensor data plays critical role in this challenge. Context-aware computing has proven to be successful in understanding sensor data. In this paper, we survey context awareness from an IoT perspective. We present the necessary background by introducing the IoT paradigm and context-aware fundamentals at the beginning. Then we provide an in-depth analysis of context life cycle. We evaluate a subset of projects (50) which represent the majority of research and commercial solutions proposed in the field of context-aware computing conducted over the last decade (2001-2011) based on our own taxonomy. Finally, based on our evaluation, we highlight the lessons to be learnt from the past and some possible directions for future research. The survey addresses a broad range of techniques, methods, models, functionalities, systems, applications, and middleware solutions related to context awareness and IoT. Our goal is not only to analyse, compare and consolidate past research work but also to appreciate their findings and discuss their applicability towards the IoT.Comment: IEEE Communications Surveys & Tutorials Journal, 201

Multi-Dimensional-Personalization in mobile contexts

During the dot com era the word "personalisation” was a hot buzzword. With the fall of the dot com companies the topic has lost momentum. As the killer application for UMTS or the mobile internet has yet to be identified, the concept of Multi-Dimensional-Personalisation (MDP) could be a candidate. Using this approach, a recommendation of mobile advertisement or marketing (i.e., recommendations or notifications), online content, as well as offline events, can be offered to the user based on their known interests and current location. Instead of having to request or pull this information, the new service concept would proactively provide the information and services – with the consequence that the right information or service could therefore be offered at the right place, at the right time. The growing availability of "Location-based Services“ for mobile phones is a new target for the use of personalisation. "Location-based Services“ are information, for example, about restaurants, hotels or shopping malls with offers which are in close range / short distance to the user. The lack of acceptance for such services in the past is based on the fact that early implementations required the user to pull the information from the service provider. A more promising approach is to actively push information to the user. This information must be from interest to the user and has to reach the user at the right time and at the right place. This raises new requirements on personalisation which will go far beyond present requirements. It will reach out from personalisation based only on the interest of the user. Besides the interest, the enhanced personalisation has to cover the location and movement patterns, the usage and the past, present and future schedule of the user. This new personalisation paradigm has to protect the user’s privacy so that an approach supporting anonymous recommendations through an extended "Chinese Wall“ will be described

PROFILING - CONCEPTS AND APPLICATIONS

Profiling is an approach to put a label or a set of labels on a subject, considering the characteristics of this subject. The New Oxford American Dictionary defines profiling as: “recording and analysis of a person’s psychological and behavioral characteristics, so as to assess or predict his/her capabilities in a certain sphere or to assist in identifying a particular subgroup of people”. This research extends this definition towards things demonstrating that many methods used for profiling of people may be applied for a different type of subjects, namely things. The goal of this research concerns proposing methods for discovery of profiles of users and things with application of Data Science methods. The profiles are utilized in vertical and 2 horizontal scenarios and concern such domains as smart grid and telecommunication (vertical scenarios), and support provided both for the needs of authorization and personalization (horizontal usage).:The thesis consists of eight chapters including an introduction and a summary. First chapter describes motivation for work that was carried out for the last 8 years together with discussion on its importance both for research and business practice. The motivation for this work is much broader and emerges also from business importance of profiling and personalization. The introduction summarizes major research directions, provides research questions, goals and supplementary objectives addressed in the thesis. Research methodology is also described, showing impact of methodological aspects on the work undertaken. Chapter 2 provides introduction to the notion of profiling. The definition of profiling is introduced. Here, also a relation of a user profile to an identity is discussed. The papers included in this chapter show not only how broadly a profile may be understood, but also how a profile may be constructed considering different data sources. Profiling methods are introduced in Chapter 3. This chapter refers to the notion of a profile developed using the BFI-44 personality test and outcomes of a survey related to color preferences of people with a specific personality. Moreover, insights into profiling of relations between people are provided, with a focus on quality of a relation emerging from contacts between two entities. Chapters from 4 to 7 present different scenarios that benefit from application of profiling methods. Chapter 4 starts with introducing the notion of a public utility company that in the thesis is discussed using examples from smart grid and telecommunication. Then, in chapter 4 follows a description of research results regarding profiling for the smart grid, focusing on a profile of a prosumer and forecasting demand and production of the electric energy in the smart grid what can be influenced e.g. by weather or profiles of appliances. Chapter 5 presents application of profiling techniques in the field of telecommunication. Besides presenting profiling methods based on telecommunication data, in particular on Call Detail Records, also scenarios and issues related to privacy and trust are addressed. Chapter 6 and Chapter 7 target at horizontal applications of profiling that may be of benefit for multiple domains. Chapter 6 concerns profiling for authentication using un-typical data sources such as Call Detail Records or data from a mobile phone describing the user behavior. Besides proposing methods, also limitations are discussed. In addition, as a side research effect a methodology for evaluation of authentication methods is proposed. Chapter 7 concerns personalization and consists of two diverse parts. Firstly, behavioral profiles to change interface and behavior of the system are proposed and applied. The performance of solutions personalizing content either locally or on the server is studied. Then, profiles of customers of shopping centers are created based on paths identified using Call Detail Records. The analysis demonstrates that the data that is collected for one purpose, may significantly influence other business scenarios. Chapter 8 summarizes the research results achieved by the author of this document. It presents contribution over state of the art as well as some insights into the future work planned

Personalizing Human-Robot Dialogue Interactions using Face and Name Recognition

Task-oriented dialogue systems are computer systems that aim to provide an interaction indistinguishable from ordinary human conversation with the goal of completing user- defined tasks. They are achieving this by analyzing the intents of users and choosing respective responses. Recent studies show that by personalizing the conversations with this systems one can positevely affect their perception and long-term acceptance. Personalised social robots have been widely applied in different fields to provide assistance. In this thesis we are working on development of a scientific conference assistant. The goal of this assistant is to provide the conference participants with conference information and inform about the activities for their spare time during conference. Moreover, to increase the engagement with the robot our team has worked on personalizing the human-robot interaction by means of face and name recognition. To achieve this personalisation, first the name recognition ability of available physical robot was improved, next by the concent of the participants their pictures were taken and used for memorization of returning users. As acquiring the consent for personal data storage is not an optimal solution, an alternative method for participants recognition using QR Codes on their badges was developed and compared to pre-trained model in terms of speed. Lastly, the personal details of each participant, as unviversity, country of origin, was acquired prior to conference or during the conversation and used in dialogues. The developed robot, called DAGFINN was displayed at two conferences happened this year in Stavanger, where the first time installment did not involve personalization feature. Hence, we conclude this thesis by discussing the influence of personalisation on dialogues with the robot and participants satisfaction with developed social robot