Location-enhanced activity recognition in indoor environments using off the shelf smart watch technology and BLE beacons

Activity recognition in indoor spaces benefits context awareness and improves the efficiency of applications related to personalised health monitoring, building energy management, security and safety. The majority of activity recognition frameworks, however, employ a network of specialised building sensors or a network of body-worn sensors. As this approach suffers with respect to practicality, we propose the use of commercial off-the-shelf devices. In this work, we design and evaluate an activity recognition system composed of a smart watch, which is enhanced with location information coming from Bluetooth Low Energy (BLE) beacons. We evaluate the performance of this approach for a variety of activities performed in an indoor laboratory environment, using four supervised machine learning algorithms. Our experimental results indicate that our location-enhanced activity recognition system is able to reach a classification accuracy ranging from 92% to 100%, while without location information classification accuracy it can drop to as low as 50% in some cases, depending on the window size chosen for data segmentation

A generalized model for indoor location estimation using environmental sound from human activity recognition

The indoor location of individuals is a key contextual variable for commercial and assisted location-based services and applications. Commercial centers and medical buildings (eg, hospitals) require location information of their users/patients to offer the services that are needed at the correct moment. Several approaches have been proposed to tackle this problem. In this paper, we present the development of an indoor location system which relies on the human activity recognition approach, using sound as an information source to infer the indoor location based on the contextual information of the activity that is realized at the moment. In this work, we analyze the sound information to estimate the location using the contextual information of the activity. A feature extraction approach to the sound signal is performed to feed a random forest algorithm in order to generate a model to estimate the location of the user. We evaluate the quality of the resulting model in terms of sensitivity and specificity for each location, and we also perform out-of-bag error estimation. Our experiments were carried out in five representative residential homes. Each home had four individual indoor rooms. Eleven activities (brewing coffee, cooking, eggs, taking a shower, etc.) were performed to provide the contextual information. Experimental results show that developing an indoor location system (ILS) that uses contextual information from human activities (identified with data provided from the environmental sound) can achieve an estimation that is 95% correct

Expand reality in-company project: A proximity technology business model research in support of healthcare management

The in-company project takes place in the 20s pandemic atmosphere, where the Customer-Journey has undergone various modifications, and investigates how the proximity-digital technology, proposed differently after 8 years1 of existence, could take off yet again in a new industry, in support of the healthcare one. Accordingly, this prospect of re-proposing proximity technology channels in the market raises a range of challenges to be faced, such as the citizen’s scepticism about the probable storage and theft of personal data. Yet, it offers unique stimulating opportunities for the project success, in terms of Customer Service, Administrative and Building Management – multiple types of studies to establish a definitive strategy aimed at disrupting and enhancing the market. For instance, leveraging the new Tech-Customer path may be complex on one hand, but it may also be a source of new value development on the other. Finally, the research will be mean for shaping a strategic Business Model Canvas for ExpandReality®. As a result, the Final Research aims to assist the start-up in understanding how the launch of the Beacons-based products and platform can work and be marketed, as well as ensuring an overcome of initial consumer’s scepticism. In conclusion the investigation will outbreak in an ultimate Business Model Canvas for the start-up, first analysed by a group of professionals and then re-shaped. “The innovation and entrepreneurship journey is about turning ideas into value propositions that customers care about and business models that can scale”. (Osterwalder, 2020)O projeto in-company ocorre na atmosfera pandémica do ano de 2020 e seguintes, em que o “Customer Journey” sofreu algumas alterações, e desta forma investiga como a tecnologia digital de proximidade, proposta de maneira diferente depois de 8 anos2 de existência, pode desenvolver-se mais uma vez numa nova indústria e numa nova realidade. Nesse sentido, a possível proposta de canais de tecnologia de proximidade no mercado, levanta alguns desafios. Como por exemplo, o ceticismo do cidadão quanto ao provável armazenamento e roubo de dados pessoais. Ainda assim, oferece possibilidades estimulantes de sucesso do projecto, em apoio ao cliente, gestão administrativa e arquitetónica - diferentes tipos de estudos direcionados no sentido de desenvolver uma estratégia com o objetivo de agitar o mercado e lucrar com ele. Por exemplo, pode ser arriscado explorar o novo caminho do cliente técnico, mas ao mesmo tempo pode ser uma possibilidade de criação de novo valor. Finalmente, a própria pesquisa será um meio para a intenção final de moldar um Modelo de Negócios estratégico para a start-up ExpandReality®. Desta forma, a Pesquisa Final tem como objetivo ajudar os conselheiros a perceber como o lançamento de produtos baseados em Beacons pode funcionar e como estes podem ser comercializados, com um uso seguro dos dados extraídos.. Concluindo a investigação surgirá no formato de um plano de modelo de negócios final para a start-up, primeiramente analisado por um grupo de profissionais e posteriormente reformulado. “Inovação e empreendedorismo consiste em transformar ideias em propostas de valor com as quais os clientes se preocupam, e modelos de negócios que podem ser escalados”. (Osterwalder, 2020

A bluetooth low energy indoor positioning system with channel diversity, weighted trilateration and Kalman filtering

Indoor Positioning Systems (IPS) using Bluetooth Low Energy (BLE) technology are currently becoming real and available, which has made them grow in popularity and use. However, there are still plenty of challenges related to this technology, especially in terms of Received Signal Strength Indicator (RSSI) fluctuations due to the behaviour of the channels and the multipath effect, that lead to poor precision. In order to mitigate these effects, in this paper we propose and implement a real Indoor Positioning System based on Bluetooth Low Energy, that improves accuracy while reducing power consumption and costs. The three main proposals are: frequency diversity, Kalman filtering and a trilateration method what we have denominated “weighted trilateration”. The analysis of the results proves that all the proposals improve the precision of the system, which goes up to 1.82 m 90% of the time for a device moving in a middle-size room and 0.7 m for static devices. Furthermore, we have proved that the system is scalable and efficient in terms of cost and power consumption. The implemented approach allows using a very simple device (like a SensorTag) on the items to locate. The system enables a very low density of anchor points or references and with a precision better than existing solutionsPeer ReviewedPostprint (published version

Ambient assisted living deployment aims to empower people living with dementia (AnAbEL)

Ambient Assisted Living aims to support the wellbeing of people with special needs by offering assistive solutions. Those systems focused on dementia claim to increase the autonomy of people living with dementia by monitoring their activities. Thus, topics such as Activity Recognition related to dementia and specific solutions such as reminders and tracking users by Global Positioning System offer great advances that seek users' safety and to preserve their healthier lifestyle. However, these solutions address secondary parties by providing useful activities logs or alerts but excluding the main interested user: the person living with dementia. Although primary users are taken into consideration at some design stages by using user-centred design frameworks, final products tend not to fully address the user's needs. This paper presents an Ambient Intelligent system aimed to reduce this limitation by developing a final solution more strongly focused on enhancing a healthy lifestyle by empowering the user's autonomy. Through continued activities monitoring in real-time, the system can provide reminders to the users by coaching them to keep healthy routines. Continuous monitoring also provides a complete user's behaviour tracking and the context-awareness logic used involves the caregivers through alerts when necessary to ensure the user's safety. This article describes the process followed to develop the system aimed to cover the previous concerns and the practical feedback from health professionals over the system deployment working in a real environment

A survey on wearable sensor modality centred human activity recognition in health care

Increased life expectancy coupled with declining birth rates is leading to an aging population structure. Aging-caused changes, such as physical or cognitive decline, could affect people's quality of life, result in injuries, mental health or the lack of physical activity. Sensor-based human activity recognition (HAR) is one of the most promising assistive technologies to support older people's daily life, which has enabled enormous potential in human-centred applications. Recent surveys in HAR either only focus on the deep learning approaches or one specific sensor modality. This survey aims to provide a more comprehensive introduction for newcomers and researchers to HAR. We first introduce the state-of-art sensor modalities in HAR. We look more into the techniques involved in each step of wearable sensor modality centred HAR in terms of sensors, activities, data pre-processing, feature learning and classification, including both conventional approaches and deep learning methods. In the feature learning section, we focus on both hand-crafted features and automatically learned features using deep networks. We also present the ambient-sensor-based HAR, including camera-based systems, and the systems which combine the wearable and ambient sensors. Finally, we identify the corresponding challenges in HAR to pose research problems for further improvement in HAR

Deep learning-based automatic analysis of social interactions from wearable data for healthcare applications

PhD ThesisSocial interactions of people with Late Life Depression (LLD) could be an objective measure of social functioning due to the association between LLD and poor social functioning. The utilisation of wearable computing technologies is a relatively new approach within healthcare and well-being application sectors. Recently, the design and development of wearable technologies and systems for health and well-being monitoring have attracted attention both of the clinical and scientific communities. Mainly because the current clinical practice of – typically rather sporadic – clinical behaviour assessments are often administered in artificial settings. As a result, it does not provide a realistic impression of a patient’s condition and thus does not lead to sufficient diagnosis and care. However, wearable behaviour monitors have the potential for continuous, objective assessment of behaviour and wider social interactions and thereby allowing for capturing naturalistic data without any constraints on the place of recording or any typical limitations of the lab-setting research. Such data from naturalistic ambient environments would facilitate automated transmission and analysis by having no constraints on the recordings, allowing for a more timely and accurate assessment of depressive symptoms. In response to this artificial setting issue, this thesis focuses on the analysis and assessment of the different aspects of social interactions in naturalistic environments using deep learning algorithms. That could lead to improvements in both diagnosis and treatment. The advantages of using deep learning are that there is no need for hand-crafted features engineering and this leads to using the raw data with minimal pre-processing compared to classical machine learning approaches and also its scalability and ability to generalise. The main dataset used in this thesis is recorded by a wrist worn device designed at Newcastle University. This device has multiple sensors including microphone, tri-axial accelerometer, light sensor and proximity sensor. In this thesis, only microphone and tri-axial accelerometer are used for the social interaction analysis. The other sensors are not used since they need more calibration from the user which in this will be the elderly people with depression. Hence, it was not feasible in this scenario. Novel deep learning models are proposed to automatically analyse two aspects of social interactions (the verbal interactions/acoustic communications and physical activities/movement patterns). Verbal Interactions include the total quantity of speech, who is talking to whom and when and how much engagement the wearer contributed in the conversations. The physical activity analysis includes activity recognition and the quantity of each activity and sleep patterns. This thesis is composed of three main stages, two of them discuss the acoustic analysis and the third stage describes the movement pattern analysis. The acoustic analysis starts with speech detection in which each segment of the recording is categorised as speech or non-speech. This segment classification is achieved by a novel deep learning model that leverages bi-directional Long Short-Term Memory with gated activation units combined with Maxout Networks as well as a combination of two optimisers. After detecting speech segments from audio data, the next stage is detecting how much engagement the wearer has in any conversation throughout these speech events based on detecting the wearer of the device using a variant model of the previous one that combines the convolutional autoencoder with bi-directional Long Short-Term Memory. Following this, the system then detects the spoken parts of the main speaker/wearer and therefore detects the conversational turn-taking but only includes the turn taking between the wearer and other speakers and not every speaker in the conversation. This stage did not take into account the semantics of the speakers due to the ethical constraints of the main dataset (Depression dataset) and therefore it was not possible to listen to the data by any means or even have any information about the contents. So, it is a good idea to be considered for future work. Stage 3 involves the physical activity analysis that is inferring the elementary physical activities and movement patterns. These elementary patterns include sedentary actions, walking, mixed activities, cycling, using vehicles as well as the sleep patterns. The predictive model used is based on Random Forests and Hidden Markov Models. In all stages the methods presented in this thesis have been compared to the state-of-the-art in processing audio, accelerometer data, respectively, to thoroughly assess their contribution. Following these stages is a thorough analysis of the interplay between acoustic interaction and physical movement patterns and the depression key clinical variables resulting to the outcomes of the previous stages. The main reason for not using deep learning in this stage unlike the previous stages is that the main dataset (Depression dataset) did not have any annotations for the speech or even the activity due to the ethical constraints as mentioned. Furthermore, the training dataset (Discussion dataset) did not have any annotations for the accelerometer data where the data is recorded freely and there is no camera attached to device to make it possible to be annotated afterwards.Newton-Mosharafa Fund and the mission sector and cultural affairs, ministry of Higher Education in Egypt

A data fusion-based hybrid sensory system for older people’s daily activity recognition.

Population aged 60 and over is growing faster. Ageing-caused changes, such as physical or cognitive decline, could affect people’s quality of life, resulting in injuries, mental health or the lack of physical activity. Sensor-based human activity recognition (HAR) has become one of the most promising assistive technologies for older people’s daily life. Literature in HAR suggests that each sensor modality has its strengths and limitations and single sensor modalities may not cope with complex situations in practice. This research aims to design and implement a hybrid sensory HAR system to provide more comprehensive, practical and accurate surveillance for older people to assist them living independently. This reseach: 1) designs and develops a hybrid HAR system which provides a spatio- temporal surveillance system for older people by combining the wrist-worn sensors and the room-mounted ambient sensors (passive infrared); the wearable data are used to recognize the defined specific daily activities, and the ambient information is used to infer the occupant’s room-level daily routine; 2): proposes a unique and effective data fusion method to hybridize the two-source sensory data, in which the captured room-level location information from the ambient sensors is also utilized to trigger the sub classification models pretrained by room-assigned wearable data; 3): implements augmented features which are extracted from the attitude angles of the wearable device and explores the contribution of the new features to HAR; 4:) proposes a feature selection (FS) method in the view of kernel canonical correlation analysis (KCCA) to maximize the relevance between the feature candidate and the target class labels and simultaneously minimizes the joint redundancy between the already selected features and the feature candidate, named mRMJR-KCCA; 5:) demonstrates all the proposed methods above with the ground-truth data collected from recruited participants in home settings. The proposed system has three function modes: 1) the pure wearable sensing mode (the whole classification model) which can identify all the defined specific daily activities together and function alone when the ambient sensing fails; 2) the pure ambient sensing mode which can deliver the occupant’s room-level daily routine without wearable sensing; and 3) the data fusion mode (room-based sub classification mode) which provides a more comprehensive and accurate surveillance HAR when both the wearable sensing and ambient sensing function properly. The research also applies the mutual information (MI)-based FS methods for feature selection, Support Vector Machine (SVM) and Random Forest (RF) for classification. The experimental results demonstrate that the proposed hybrid sensory system improves the recognition accuracy to 98.96% after applying data fusion using Random Forest (RF) classification and mRMJR-KCCA feature selection. Furthermore, the improved results are achieved with a much smaller number of features compared with the scenario of recognizing all the defined activities using wearable data alone. The research work conducted in the thesis is unique, which is not directly compared with others since there are few other similar existing works in terms of the proposed data fusion method and the introduced new feature set