Affective Computing for Emotion Detection using Vision and Wearable Sensors

The research explores the opportunities, challenges, limitations, and presents advancements in computing that relates to, arises from, or deliberately influences emotions (Picard, 1997). The field is referred to as Affective Computing (AC) and is expected to play a major role in the engineering and development of computationally and cognitively intelligent systems, processors and applications in the future. Today the field of AC is bolstered by the emergence of multiple sources of affective data and is fuelled on by developments under various Internet of Things (IoTs) projects and the fusion potential of multiple sensory affective data streams. The core focus of this thesis involves investigation into whether the sensitivity and specificity (predictive performance) of AC, based on the fusion of multi-sensor data streams, is fit for purpose? Can such AC powered technologies and techniques truly deliver increasingly accurate emotion predictions of subjects in the real world? The thesis begins by presenting a number of research justifications and AC research questions that are used to formulate the original thesis hypothesis and thesis objectives. As part of the research conducted, a detailed state of the art investigations explored many aspects of AC from both a scientific and technological perspective. The complexity of AC as a multi-sensor, multi-modality, data fusion problem unfolded during the state of the art research and this ultimately led to novel thinking and origination in the form of the creation of an AC conceptualised architecture that will act as a practical and theoretical foundation for the engineering of future AC platforms and solutions. The AC conceptual architecture developed as a result of this research, was applied to the engineering of a series of software artifacts that were combined to create a prototypical AC multi-sensor platform known as the Emotion Fusion Server (EFS) to be used in the thesis hypothesis AC experimentation phases of the research. The thesis research used the EFS platform to conduct a detailed series of AC experiments to investigate if the fusion of multiple sensory sources of affective data from sensory devices can significantly increase the accuracy of emotion prediction by computationally intelligent means. The research involved conducting numerous controlled experiments along with the statistical analysis of the performance of sensors for the purposes of AC, the findings of which serve to assess the feasibility of AC in various domains and points to future directions for the AC field. The AC experiments data investigations conducted in relation to the thesis hypothesis used applied statistical methods and techniques, and the results, analytics and evaluations are presented throughout the two thesis research volumes. The thesis concludes by providing a detailed set of formal findings, conclusions and decisions in relation to the overarching research hypothesis on the sensitivity and specificity of the fusion of vision and wearables sensor modalities and offers foresights and guidance into the many problems, challenges and projections for the AC field into the future

Continuous and transparent multimodal authentication: reviewing the state of the art

Individuals, businesses and governments undertake an ever-growing range of activities online and via various Internet-enabled digital devices. Unfortunately, these activities, services, information and devices are the targets of cybercrimes. Verifying the user legitimacy to use/access a digital device or service has become of the utmost importance. Authentication is the frontline countermeasure of ensuring only the authorized user is granted access; however, it has historically suffered from a range of issues related to the security and usability of the approaches. They are also still mostly functioning at the point of entry and those performing sort of re-authentication executing it in an intrusive manner. Thus, it is apparent that a more innovative, convenient and secure user authentication solution is vital. This paper reviews the authentication methods along with the current use of authentication technologies, aiming at developing a current state-of-the-art and identifying the open problems to be tackled and available solutions to be adopted. It also investigates whether these authentication technologies have the capability to fill the gap between high security and user satisfaction. This is followed by a literature review of the existing research on continuous and transparent multimodal authentication. It concludes that providing users with adequate protection and convenience requires innovative robust authentication mechanisms to be utilized in a universal level. Ultimately, a potential federated biometric authentication solution is presented; however it needs to be developed and extensively evaluated, thus operating in a transparent, continuous and user-friendly manner

Novel Machine Learning and Wearable Sensor Based Solutions for Smart Healthcare Monitoring

The advent of IoT has enabled the design of connected and integrated smart health monitoring systems. These health monitoring systems can be utilized for monitoring the mental and physical wellbeing of a person. Stress, anxiety, and hypertension are the major elements responsible for the plethora of physical and mental illnesses. In this context, the older population demands special attention because of the several age-related complications that exacerbate the effects of stress, anxiety, and hypertension. Monitoring stress, anxiety, and blood pressure regularly can prevent long-term damage by initiating necessary intervention or clinical treatment beforehand. This will improve the quality of life and reduce the burden on caregivers and the cost of healthcare. Therefore, this thesis explores novel technological solutions for real-time monitoring of stress, anxiety, and blood pressure using unobtrusive wearable sensors and machine learning techniques. The first contribution of this thesis is the experimental data collection of 50 healthy older adults, based on which, the works on stress detection and anxiety detection have been developed. The data collection procedure lasted for more than a year. We have collected physiological signals, salivary cortisol, and self-reported questionnaire feedback during the study. Salivary cortisol is an established clinical biomarker for physiological stress. Hence, a stress detection model that is trained to distinguish between the stressed and not-stressed states as indicated by the increase in cortisol level has the potential to facilitate clinical level diagnosis of stress from the comfort of their own home. The second contribution of the thesis is the development of a stress detection model based on fingertip sensors. We have extracted features from Electrodermal Activity (EDA) and Blood Volume Pulse (BVP) signals obtained from fingertip EDA and Photoplethysmogram (PPG) sensors to train machine learning algorithms for distinguishing between stressed and not-stressed states. We have evaluated the performance of four traditional machine learning algorithms and one deep-learning-based Long Short-Term Memory (LSTM) classifier. Results and analysis showed that the proposed LSTM classifier performed equally well as the traditional machine learning models. The third contribution of the thesis is to evaluate an integrated system of wrist-worn sensors for stress detection. We have evaluated four signal streams, EDA, BVP, Inter-Beat Interval (IBI), and Skin Temperature (ST) signals from EDA, PPG, and ST sensors. A random forest classifier was used for distinguishing between the stressed and not-stressed states. Results and analysis showed that incorporating features from different signals was able to reduce the misclassification rate of the classifier. Further, we have also prototyped the integration of the proposed wristband-based stress detection system in a consumer end device with voice capabilities. The fourth contribution of the thesis is the design of an anxiety detection model that uses features from a single wearable sensor and a context feature to improve the performance of the classification model. Using a context feature instead of integrating other physiological features for improving the performance of the model can reduce the complexity and cost of the anxiety detection model. In our proposed work, we have used a simple experimental context feature to highlight the importance of context in the accurate detection of anxious states. Our results and analysis have shown that with the addition of the context-based feature, the classifier was able to reduce misclassification by increasing the confidence of the decision. The final and the fifth contribution of the thesis is the validation of a proposed computational framework for the blood pressure estimation model. The proposed framework uses features from the PPG signal to estimate the systolic and diastolic blood pressure values using advanced regression techniques

Insomnia : the affordance of hybrid media in visualising a sleep disorder

The integration of visual and numerical abstraction in contemporary audio-visual communication has become increasingly prevalent. This increase reflects the evolution of computational machines from simple data processors. Computation and interface have augmented our senses and converged algorithmic logic with cultural techniques to form hybrid channels of communication. These channels are fluid and mutable, allowing creatives to explore and disseminate knowledge through iterative media practice. Insomnia is an auto-ethnographic case study that examines the affordance of merging Brain-Computer Interfaces (BCIs) and node- based programming software (TouchDesigner), as a hybrid media system (McMullan, 2020). As a system, Insomnia compiles my archived brain activity data and processes it through a custom designed generative visualisation interface. Documenting and ‘processing’ a sleep disorder is filtered through key concepts of media archaeology, cultural techniques, and practice-led research allowing Insomnia to inform discussion of the affordance of hybrid media. Insomnia is presented as a virtual exhibition with a supporting exegesis. The methodology and outcomes of the project form a framework that bridges science communication and creative practice and points to continued development for interactive installation design

Elderly Fall Detection Systems: A Literature Survey

Falling is among the most damaging event elderly people may experience. With the ever-growing aging population, there is an urgent need for the development of fall detection systems. Thanks to the rapid development of sensor networks and the Internet of Things (IoT), human-computer interaction using sensor fusion has been regarded as an effective method to address the problem of fall detection. In this paper, we provide a literature survey of work conducted on elderly fall detection using sensor networks and IoT. Although there are various existing studies which focus on the fall detection with individual sensors, such as wearable ones and depth cameras, the performance of these systems are still not satisfying as they suffer mostly from high false alarms. Literature shows that fusing the signals of different sensors could result in higher accuracy and lower false alarms, while improving the robustness of such systems. We approach this survey from different perspectives, including data collection, data transmission, sensor fusion, data analysis, security, and privacy. We also review the benchmark data sets available that have been used to quantify the performance of the proposed methods. The survey is meant to provide researchers in the field of elderly fall detection using sensor networks with a summary of progress achieved up to date and to identify areas where further effort would be beneficial

On driver behavior recognition for increased safety:A roadmap

Advanced Driver-Assistance Systems (ADASs) are used for increasing safety in the automotive domain, yet current ADASs notably operate without taking into account drivers’ states, e.g., whether she/he is emotionally apt to drive. In this paper, we first review the state-of-the-art of emotional and cognitive analysis for ADAS: we consider psychological models, the sensors needed for capturing physiological signals, and the typical algorithms used for human emotion classification. Our investigation highlights a lack of advanced Driver Monitoring Systems (DMSs) for ADASs, which could increase driving quality and security for both drivers and passengers. We then provide our view on a novel perception architecture for driver monitoring, built around the concept of Driver Complex State (DCS). DCS relies on multiple non-obtrusive sensors and Artificial Intelligence (AI) for uncovering the driver state and uses it to implement innovative Human–Machine Interface (HMI) functionalities. This concept will be implemented and validated in the recently EU-funded NextPerception project, which is briefly introduced

Home healthcare using ubiquitous computing and robot technologies

The rapid increase of senior population worldwide is challenging the existing healthcare and support systems. Recently, smart home environments are utilized for ubiquitous health monitoring, allowing patients to stay in the comfort of their homes. In this dissertation, a Cloud-based Smart Home Environment (CoSHE) for home healthcare is presented, which consists of ambient intelligence, wearable computing, and robot technologies. The system includes a smart home which is embedded with distributed environmental sensors to support human localization. Wearable units are developed to collect physiological, motion and audio signals through non-invasive wearable sensors and provide contextual information in terms of the resident's daily activity and location in the home. This enables healthcare professionals to study daily activities, behavioral changes and monitor rehabilitation and recovery processes. The sensor data are processed in a smart home gateway and sent to a private cloud, which provides real-time data access for remote caregivers. Our case studies show that contextual information provided by ubiquitous computing can help better understand the patient's health status. With a robot assistant in the loop, we demonstrated that the CoSHE can facilitate healthcare delivery via interaction between human and robot

Wiki-health: from quantified self to self-understanding

Today, healthcare providers are experiencing explosive growth in data, and medical imaging represents a significant portion of that data. Meanwhile, the pervasive use of mobile phones and the rising adoption of sensing devices, enabling people to collect data independently at any time or place is leading to a torrent of sensor data. The scale and richness of the sensor data currently being collected and analysed is rapidly growing. The key challenges that we will be facing are how to effectively manage and make use of this abundance of easily-generated and diverse health data. This thesis investigates the challenges posed by the explosive growth of available healthcare data and proposes a number of potential solutions to the problem. As a result, a big data service platform, named Wiki-Health, is presented to provide a unified solution for collecting, storing, tagging, retrieving, searching and analysing personal health sensor data. Additionally, it allows users to reuse and remix data, along with analysis results and analysis models, to make health-related knowledge discovery more available to individual users on a massive scale. To tackle the challenge of efficiently managing the high volume and diversity of big data, Wiki-Health introduces a hybrid data storage approach capable of storing structured, semi-structured and unstructured sensor data and sensor metadata separately. A multi-tier cloud storage system—CACSS has been developed and serves as a component for the Wiki-Health platform, allowing it to manage the storage of unstructured data and semi-structured data, such as medical imaging files. CACSS has enabled comprehensive features such as global data de-duplication, performance-awareness and data caching services. The design of such a hybrid approach allows Wiki-Health to potentially handle heterogeneous formats of sensor data. To evaluate the proposed approach, we have developed an ECG-based health monitoring service and a virtual sensing service on top of the Wiki-Health platform. The two services demonstrate the feasibility and potential of using the Wiki-Health framework to enable better utilisation and comprehension of the vast amounts of sensor data available from different sources, and both show significant potential for real-world applications.Open Acces