EmoEcho: a tangible interface to convey and communicate emotions

An interactive tangible interface has been developed to capture and communicate emotions between people who are missing and longing for loved ones. EmoEcho measures the wearer’s pulse, touch and movement to provide varying vibration patterns on the partner device. During an informal evaluation of two prototype devices users acknowledged how EmoEcho could help counter the negative feeling of missing someone through the range of haptic feedback offered. In general, we believe, tangible interfaces appear to offer a non-obtrusive means towards interpreting and communicating emotions to others

LabelSens: enabling real-time sensor data labelling at the point of collection using an artificial intelligence-based approach

In recent years, machine learning has developed rapidly, enabling the development of applications with high levels of recognition accuracy relating to the use of speech and images. However, other types of data to which these models can be applied have not yet been explored as thoroughly. Labelling is an indispensable stage of data pre-processing that can be particularly challenging, especially when applied to single or multi-model real-time sensor data collection approaches. Currently, real-time sensor data labelling is an unwieldy process, with a limited range of tools available and poor performance characteristics, which can lead to the performance of the machine learning models being compromised. In this paper, we introduce new techniques for labelling at the point of collection coupled with a pilot study and a systematic performance comparison of two popular types of deep neural networks running on five custom built devices and a comparative mobile app (68.5-89% accuracy within-device GRU model, 92.8% highest LSTM model accuracy). These devices are designed to enable real-time labelling with various buttons, slide potentiometer and force sensors. This exploratory work illustrates several key features that inform the design of data collection tools that can help researchers select and apply appropriate labelling techniques to their work. We also identify common bottlenecks in each architecture and provide field tested guidelines to assist in building adaptive, high-performance edge solutions

iFidgetCube: Tangible Fidgeting Interfaces (TFIs) to monitor and improve mental wellbeing

The ability to unobtrusively measure mental wellbeing states using non-invasive sensors has the potential to greatly improve mental wellbeing by alleviating the effects of high stress levels. Multiple sensors, such as electrodermal activity, heart rate and accelerometers, embedded within tangible devices pave the way to continuously and non-invasively monitor wellbeing in real-world environments. On the other hand, fidgeting tools enable repetitive interaction methods that may help to tap into individual’s psychological need to feel occupied and engaged; hence potentially reducing stress. In this paper, we present the design, implementation, and deployment of Tangible Fidgeting Interfaces (TFIs) in the form of computerised iFidgetCubes. iFidgetCubes embed non-invasive sensors along with fidgeting mechanisms to aid relaxation and ease restlessness. We take advantage of our labeling techniques at the point of collection to implement multiple subject-independent deep learning classifiers to infer wellbeing. The obtained performance demonstrates that these new forms of tangible interfaces combined with deep learning classifiers have the potential to accurately infer wellbeing in addition to providing fidgeting tools

Tag in the Park: paving the way for proximity-based AI pervasive games

In this paper we introduce “Tag in the Park”, a mobile platform merging short-range wireless communication and Artificial Intelligence (AI) to create an interactive gamified experience through virtual treasure hunts. Central to this is a mobile app that revolves around the interaction with a network of Bluetooth Low Energy (BLE) beacons and Near Field Communication (NFC) tags. These miniature smart devices provide contextual information, AI challenges, quizzes and nudges for a gamified, and active visitor experience which can help increase engagement and promote physical activity. “Tag in the Park” is currently available for download on both Google Play and Apple Store and the platform is fully set up and open to visitors at Rufford Abbey Park, Nottinghamshire. Results indicate the mobile platform helps engage visitors, encourages exploration and increases visitor economy

Combining deep learning with signal-image encoding for multi-modal mental wellbeing classification

The quantification of emotional states is an important step to understanding wellbeing. Time series data from multiple modalities such as physiological and motion sensor data have proven to be integral for measuring and quantifying emotions. Monitoring emotional trajectories over long periods of time inherits some critical limitations in relation to the size of the training data. This shortcoming may hinder the development of reliable and accurate machine learning models. To address this problem, this paper proposes a framework to tackle the limitation in performing emotional state recognition: 1) encoding time series data into coloured images; 2) leveraging pre-trained object recognition models to apply a Transfer Learning (TL) approach using the images from step 1; 3) utilising a 1D Convolutional Neural Network (CNN) to perform emotion classification from physiological data; 4) concatenating the pre-trained TL model with the 1D CNN. We demonstrate that model performance when inferring real-world wellbeing rated on a 5-point Likert scale can be enhanced using our framework, resulting in up to 98.5% accuracy, outperforming a conventional CNN by 4.5%. Subject-independent models using the same approach resulted in an average of 72.3% accuracy (SD 0.038). The proposed methodology helps improve performance and overcome problems with small training datasets

Emotion on the edge: air quality sensors decoded as a real-world emotion indicator

As the research community increasingly focuses on quantifying emotional states in real-world scenarios, there is a growing need for edge computing. In this work, we present a novel approach to on-device emotion classification through the development of a low-cost hand-held device. This device incorporates a range of environmental air quality factors, including Particulate Matter, Nitrogen Dioxide, Carbon Monoxide, Ammonia, and Noise. Our research addresses the current limitations in the field of emotional state measurement by leveraging environmental air quality data, which has been previously linked to affective states. This on-device approach not only offers an alternative to resource-intensive emotion recognition methods but also contributes to the development of more practical and affordable solutions for emotion assessment. The preliminary results of our device's performance in real-world scenarios suggest its effectiveness in quantifying emotional states through air quality factors, with the model achieving 95% accuracy demonstrating accurate on-device classification without the need for external high-processing power

Combining multiple tinyML models for multimodal context-aware stress recognition on constrained microcontrollers

As stress continues to be a major health concern, there is growing interest in developing effective stress management systems that can detect and mitigate stress. Deep Neural Networks (DNNs) have shown their effectiveness in accurately classifying stress, but most existing solutions rely on the cloud or large obtrusive devices for inference. The emergence of tinyML provides an opportunity to bridge this gap and enable ubiquitous intelligent systems. In this paper, we propose a context-aware stress detection approach that uses a microcontroller to continuously infer physical activity to mitigate motion artifacts when inferring stress from heart rate and electrodermal activity. We deploy two DNNs onto a single resource-constrained microcontroller for real-world stress recognition, with the resultant stress and activity recognition models achieving 88% and 98% accuracy respectively. Our proposed context-aware approach improves the accuracy and privacy of stress detection systems while eliminating the need to store or transmit sensitive health data

Challenges of designing and developing tangible interfaces for mental well-being

Mental well-being technologies possess many qualities that give them the potential to help people receive assessment and treatment who may otherwise not receive help due to fear of stigma or lack of resources. The combination of advances in sensors, microcontrollers and machine learning is leading to the emergence of dedicated tangible interfaces to monitor and promote positive mental well-being. However, there are key technical, ergonomic and aesthetic challenges to be overcome in order to make these interfaces effective and respond to users’ needs. In this paper, the barriers to develop mental well-being tangible interfaces are discussed by identifying and examining the recent technological challenges machine learning, sensors, microcontrollers and batteries create. User-oriented challenges that face the development of mental well-being technologies are then considered ranging from user engagement during co-design and trials to ethical and privacy concern