Inferring Complex Activities for Context-aware Systems within Smart Environments

The rising ageing population worldwide and the prevalence of age-related conditions such as physical fragility, mental impairments and chronic diseases have significantly impacted the quality of life and caused a shortage of health and care services. Over-stretched healthcare providers are leading to a paradigm shift in public healthcare provisioning. Thus, Ambient Assisted Living (AAL) using Smart Homes (SH) technologies has been rigorously investigated to help address the aforementioned problems. Human Activity Recognition (HAR) is a critical component in AAL systems which enables applications such as just-in-time assistance, behaviour analysis, anomalies detection and emergency notifications. This thesis is aimed at investigating challenges faced in accurately recognising Activities of Daily Living (ADLs) performed by single or multiple inhabitants within smart environments. Specifically, this thesis explores five complementary research challenges in HAR. The first study contributes to knowledge by developing a semantic-enabled data segmentation approach with user-preferences. The second study takes the segmented set of sensor data to investigate and recognise human ADLs at multi-granular action level; coarse- and fine-grained action level. At the coarse-grained actions level, semantic relationships between the sensor, object and ADLs are deduced, whereas, at fine-grained action level, object usage at the satisfactory threshold with the evidence fused from multimodal sensor data is leveraged to verify the intended actions. Moreover, due to imprecise/vague interpretations of multimodal sensors and data fusion challenges, fuzzy set theory and fuzzy web ontology language (fuzzy-OWL) are leveraged. The third study focuses on incorporating uncertainties caused in HAR due to factors such as technological failure, object malfunction, and human errors. Hence, existing studies uncertainty theories and approaches are analysed and based on the findings, probabilistic ontology (PR-OWL) based HAR approach is proposed. The fourth study extends the first three studies to distinguish activities conducted by more than one inhabitant in a shared smart environment with the use of discriminative sensor-based techniques and time-series pattern analysis. The final study investigates in a suitable system architecture with a real-time smart environment tailored to AAL system and proposes microservices architecture with sensor-based off-the-shelf and bespoke sensing methods. The initial semantic-enabled data segmentation study was evaluated with 100% and 97.8% accuracy to segment sensor events under single and mixed activities scenarios. However, the average classification time taken to segment each sensor events have suffered from 3971ms and 62183ms for single and mixed activities scenarios, respectively. The second study to detect fine-grained-level user actions was evaluated with 30 and 153 fuzzy rules to detect two fine-grained movements with a pre-collected dataset from the real-time smart environment. The result of the second study indicate good average accuracy of 83.33% and 100% but with the high average duration of 24648ms and 105318ms, and posing further challenges for the scalability of fusion rule creations. The third study was evaluated by incorporating PR-OWL ontology with ADL ontologies and Semantic-Sensor-Network (SSN) ontology to define four types of uncertainties presented in the kitchen-based activity. The fourth study illustrated a case study to extended single-user AR to multi-user AR by combining RFID tags and fingerprint sensors discriminative sensors to identify and associate user actions with the aid of time-series analysis. The last study responds to the computations and performance requirements for the four studies by analysing and proposing microservices-based system architecture for AAL system. A future research investigation towards adopting fog/edge computing paradigms from cloud computing is discussed for higher availability, reduced network traffic/energy, cost, and creating a decentralised system. As a result of the five studies, this thesis develops a knowledge-driven framework to estimate and recognise multi-user activities at fine-grained level user actions. This framework integrates three complementary ontologies to conceptualise factual, fuzzy and uncertainties in the environment/ADLs, time-series analysis and discriminative sensing environment. Moreover, a distributed software architecture, multimodal sensor-based hardware prototypes, and other supportive utility tools such as simulator and synthetic ADL data generator for the experimentation were developed to support the evaluation of the proposed approaches. The distributed system is platform-independent and currently supported by an Android mobile application and web-browser based client interfaces for retrieving information such as live sensor events and HAR results

A semantics-based approach to sensor data segmentation in real-time Activity Recognition

Department of Information Engineering, Dalian University, China The file attached to this record is the author's final peer reviewed version. The Publisher's final version can be found by following the DOI link.Activity Recognition (AR) is key in context-aware assistive living systems. One challenge in AR is the segmentation of observed sensor events when interleaved or concurrent activities of daily living (ADLs) are performed. Several studies have proposed methods of separating and organising sensor observations and recognise generic ADLs performed in a simple or composite manner. However, little has been explored in semantically distinguishing individual sensor events directly and passing it to the relevant ongoing/new atomic activities. This paper proposes Semiotic theory inspired ontological model, capturing generic knowledge and inhabitant-specific preferences for conducting ADLs to support the segmentation process. A multithreaded decision algorithm and system prototype were developed and evaluated against 30 use case scenarios where each event was simulated at 10sec interval on a machine with i7 2.60GHz CPU, 2 cores and 8GB RAM. The result suggests that all sensor events were adequately segmented with 100% accuracy for single ADL scenarios and minor improvement of 97.8% accuracy for composite ADL scenario. However, the performance has suffered to segment each event with the average classification time of 3971ms and 62183ms for single and composite ADL scenarios, respectively

Semantic segmentation of real-time sensor data stream for complex activity recognition

The file attached to this record is the author's final peer reviewed version. The Publisher's final version can be found by following the DOI link.Data segmentation plays a critical role in performing human activity recognition in the ambient assistant living systems. It is particularly important for complex activity recognition when the events occur in short bursts with attributes of multiple sub-tasks. Although substantial efforts have been made in segmenting the real-time sensor data stream such as static/dynamic window sizing approaches, little has been explored to exploit object semantic for discerning sensor data into multiple threads of activity of daily living. This paper proposes a semantic-based approach for segmenting sensor data series using ontologies to perform terminology box and assertion box reasoning, along with logical rules to infer whether the incoming sensor event is related to a given sequences of the activity. The proposed approach is illustrated using a use-case scenario which conducts semantic segmentation of a real-time sensor data stream to recognise an elderly persons complex activities

Towards a service-oriented architecture for a mobile assistive system with real-time environmental sensing

Dalian Key Lab. of Smart Medical and Healthcare, Computer Science Department, Dalian University, China,With the growing aging population, age-related diseases have increased considerably over the years. In response to these, ambient assistive living (AAL) systems are being developed and are continually evolving to enrich and support independent living. While most researchers investigate robust activity recognition (AR) techniques, this paper focuses on some of the architectural challenges of the AAL systems. This work proposes a system architecture that fuses varying software design patterns and integrates readily available hardware devices to create wireless sensor networks for real-time applications. The system architecture brings together the service-oriented architecture (SOA), semantic web technologies, and other methods to address some of the shortcomings of the preceding system implementations using off-the-shelf and open source components. In order to validate the proposed architecture, a prototype is developed and tested positively to recognize basic user activities in real time. The system provides a base that can be further extended in many areas of AAL systems, including composite AR

Real-Time Sensor Observation Segmentation For Complex Activity Recognition Within Smart Environments

The file attached to this record is the author's final peer reviewed versionActivity Recognition (AR) is at the heart of any types of assistive living systems. One of the key challenges faced in AR is segmentation of the sensor events when inhabitant performs simple or composite activities of daily living (ADLs). In addition, each inhabitant may follow a particular ritual or a tradition in performing different ADLs and their patterns may change overtime. Many recent studies apply methods to segment and recognise generic ADLs performed in a composite manner. However, little has been explored in semantically distinguishing individual sensor events and directly passing it to the relevant ongoing/new atomic activities. This paper proposes to use the ontological model to capture generic knowledge of ADLs and methods which also takes inhabitant-specific preferences into considerations when segmenting sensor events. The system implementation was developed, deployed and evaluated against 84 use case scenarios. The result suggests that all sensor events were adequately segmented with 98% accuracy and the average classification time of 3971ms and 62183ms for single and composite ADL scenarios were recorded, respectively

Reality and Perception: Activity monitoring and data collection within a real-world smart home

The file attached to this record is the author's final peer reviewed version.Smart home technologies have been developing rapidly in the last few years. However, there is still a lack of annotated rich datasets that can be used for different analysis purposes by researchers. The motivation for this study is driven by the need of self-management for chronic disease patients and the often neglected privacy aspects. The study describes the extension of an existing smart home environment at Great Northern Haven (GHN) with ambient and wearable devices. The discussed principles include the design of the experiment, data collection strategies and encountered challenges in regards to the sensors, connection problems and occupation with multiple inhabitants

East Midlands Research into Ageing Network (EMRAN) Discussion Paper Series

Academic geriatric medicine in Leicester . There has never been a better time to consider joining us. We have recently appointed a Professor in Geriatric Medicine, alongside Tom Robinson in stroke and Victoria Haunton, who has just joined as a Senior Lecturer in Geriatric Medicine. We have fantastic opportunities to support students in their academic pursuits through a well-established intercalated BSc programme, and routes on through such as ACF posts, and a successful track-record in delivering higher degrees leading to ACL post. We collaborate strongly with Health Sciences, including academic primary care. See below for more detail on our existing academic set-up. Leicester Academy for the Study of Ageing We are also collaborating on a grander scale, through a joint academic venture focusing on ageing, the ‘Leicester Academy for the Study of Ageing’ (LASA), which involves the local health service providers (acute and community), De Montfort University; University of Leicester; Leicester City Council; Leicestershire County Council and Leicester Age UK. Professors Jayne Brown and Simon Conroy jointly Chair LASA and have recently been joined by two further Chairs, Professors Kay de Vries and Bertha Ochieng. Karen Harrison Dening has also recently been appointed an Honorary Chair. LASA aims to improve outcomes for older people and those that care for them that takes a person-centred, whole system perspective. Our research will take a global perspective, but will seek to maximise benefits for the people of Leicester, Leicestershire and Rutland, including building capacity. We are undertaking applied, translational, interdisciplinary research, focused on older people, which will deliver research outcomes that address domains from: physical/medical; functional ability, cognitive/psychological; social or environmental factors. LASA also seeks to support commissioners and providers alike for advice on how to improve care for older people, whether by research, education or service delivery. Examples of recent research projects include: ‘Local History Café’ project specifically undertaking an evaluation on loneliness and social isolation; ‘Better Visits’ project focused on improving visiting for family members of people with dementia resident in care homes; and a study on health issues for older LGBT people in Leicester. Clinical Geriatric Medicine in Leicester We have developed a service which recognises the complexity of managing frail older people at the interface (acute care, emergency care and links with community services). There are presently 17 consultant geriatricians supported by existing multidisciplinary teams, including the largest complement of Advance Nurse Practitioners in the country. Together we deliver Comprehensive Geriatric Assessment to frail older people with urgent care needs in acute and community settings. The acute and emergency frailty units – Leicester Royal Infirmary This development aims at delivering Comprehensive Geriatric Assessment to frail older people in the acute setting. Patients are screened for frailty in the Emergency Department and then undergo a multidisciplinary assessment including a consultant geriatrician, before being triaged to the most appropriate setting. This might include admission to in-patient care in the acute or community setting, intermediate care (residential or home based), or occasionally other specialist care (e.g. cardiorespiratory). Our new emergency department is the county’s first frail friendly build and includes fantastic facilities aimed at promoting early recovering and reducing the risk of hospital associated harms. There is also a daily liaison service jointly run with the psychogeriatricians (FOPAL); we have been examining geriatric outreach to oncology and surgery as part of an NIHR funded study. We are home to the Acute Frailty Network, and those interested in service developments at the national scale would be welcome to get involved. Orthogeriatrics There are now dedicated hip fracture wards and joint care with anaesthetists, orthopaedic surgeons and geriatricians. There are also consultants in metabolic bone disease that run clinics. Community work Community work will consist of reviewing patients in clinic who have been triaged to return to the community setting following an acute assessment described above. Additionally, primary care colleagues refer to outpatients for sub-acute reviews. You will work closely with local GPs with support from consultants to deliver post-acute, subacute, intermediate and rehabilitation care services. Stroke Medicine 24/7 thrombolysis and TIA services. The latter is considered one of the best in the UK and along with the high standard of vascular surgery locally means one of the best performances regarding carotid intervention

Towards a Mobile Assistive System Using Service-Oriented Architecture

As the aging population grows, age-related diseases show a cohesive increase. Ambient Assistive Living (AAL) systems are being developed and are continually evolving in various areas. While most researchers focus on robust Activity Recognition techniques, this paper investigates some of the architectural challenges of the AAL systems. This study proposes a new system architecture that brings together the service-oriented architecture (SOA), Semantic Web technologies and other methods to address some of the shortfalls in the predecessor system implementations using off-the-shelf and open source components. A partial system implementation is then presented using the proposed system architecture. The system takes some of the key design aspects such as extensibility, reusability, scalability, and maintainability into consideration that can then be seen as a base to further extend the capability of monitoring, collecting, processing and accurately recognising complex or concurrent activities in its overall aim to support assistive living

Fuzzy-Based Fine-Grained Human Activity Recognition within Smart Environments

The Publisher's final version can be found by following the DOI link.With the increasing ageing population, Smart Home (SH) has been under vigorous investigation to enable Ambient Assisted Living (AAL) and foster independent living. Human Activity Recognition (HAR) is the backbone of AAL systems in order to detect Activities of Daily Living (ADL) and provide timely, context-aware assistance. Existing SH based AAL systems primarily focus on coarse-grained activity recognition (AR) and assume successful usage of everyday objects using binary sensors. Limited attention is given to fined-grained AR by verifying the intended object interactions with evidence from multiple heterogeneous sensor data. This paper proposes a fine-grained AR approach which fuses multimodal data from single objects and handles the imprecise nature of non-binary sensor measurements. This approach leverages the fuzzy ontology to model fine-grained actions with imprecise membership states of the sensors in relation to object and fuzzyDL reasoning tool to classify action completion. In addition, a microservice architecture is proposed with a non-intrusive heterogeneous ambient and embedded object based sensing method. The sensing method integrates both off-the-shelf and bespoke devices to collect fine-grained object level interactions. A case study is provided to illustrate the use of the fine-grained AR approach to recognize kitchen-based activities