Eating and drinking gesture spotting and recognition using a novel adaptive segmentation technique and a gesture discrepancy measure

Despite the increasing developments on human activity recognition using wearable technology, there are still many open challenges in spotting and recognising sporadic gestures. As opposed to activities, which exhibit continuous behaviour, the difficulty of spotting gestures lies in their rather sparse nature. This paper proposes a novel solution to spot and recognise a set of similar eating and drinking gestures from continuous inertial data streams. First, potential segments containing an eating or a drinking gesture are found using a Crossings-based Adaptive Segmentation Technique (CAST). Second, further to the long-established range of features employed in previous human activities recognition research work, a gesture discrepancy measure is proposed to improve the classification performance of the system. At the final step, a range of state-of-the-art classification models is employed for evaluation. Various conclusions can be drawn from the results obtained. First, given the 100% recall achieved at the segmentation step, the CAST can be considered a reliable segmentation technique for spotting drinking and eating gestures which may be employed in future gesture spotting work. Second, the addition of gesture discrepancy as a feature descriptor consistently improves the classification performance of the system. Third, the reliability of the food and drink intake monitoring approach proposed in this work finds support on the out-performance of previous similar work

Combination of Accumulated Motion and Color Segmentation for Human Activity Analysis

The automated analysis of activity in digital multimedia, and especially video, is gaining more and more importance due to the evolution of higher-level video processing systems and the development of relevant applications such as surveillance and sports. This paper presents a novel algorithm for the recognition and classification of human activities, which employs motion and color characteristics in a complementary manner, so as to extract the most information from both sources, and overcome their individual limitations. The proposed method accumulates the flow estimates in a video, and extracts “regions of activity†by processing their higher-order statistics. The shape of these activity areas can be used for the classification of the human activities and events taking place in a video and the subsequent extraction of higher-level semantics. Color segmentation of the active and static areas of each video frame is performed to complement this information. The color layers in the activity and background areas are compared using the earth mover's distance, in order to achieve accurate object segmentation. Thus, unlike much existing work on human activity analysis, the proposed approach is based on general color and motion processing methods, and not on specific models of the human body and its kinematics. The combined use of color and motion information increases the method robustness to illumination variations and measurement noise. Consequently, the proposed approach can lead to higher-level information about human activities, but its applicability is not limited to specific human actions. We present experiments with various real video sequences, from sports and surveillance domains, to demonstrate the effectiveness of our approach

Recognition of quotidian activities in support of independent living using a single wrist-worn inertial measurement unit

The field of Ambient Assisted Living (AAL) is gaining increasing attention from the research community in recent years with the rapid present and future ageing of the population worldwide. This problem has been widely recognised as has the need for it to be addressed both from an economic and societal perspective. Assisted living environments incorporate technological solutions to create a better condition of life for older adults. However, in order to create a better condition of life, it is crucial to understand the specific needs of each individual. To this regard, self-assessment of daily activities has shown to be subjective and variable, presenting important discrepancies with those performed by clinicians. The above challenges have fostered the search for alternative monitoring solutions, increasing the research efforts upon the field of Human Activity Recognition (HAR). A vast array of sensing devices, including ambient sensors, video cameras and wearable devices, has been employed for the automatic monitoring of a person in a home environment. However, the research focus is shifting towards wearable solutions, which avoid the privacy concerns related to the use of video cameras in a home environment while providing more intrinsic information about the user than ambient devices. The focus of this research is the investigation of signal processing and machine learning techniques for the recognition of quotidian activities concerning self-neglect (a behavioural condition in which individuals, generally older people, disregard the attention, intentionally or un intentionally, of their basic needs). More precisely, the aimed group of activities include those concerning personal hygiene, namely handswashing and teeth brushing, as well as those directly related to dietary behaviour, namely eating and drinking. The work undertaken in this thesis is divided into three different stages. First, given the continuous quasi-periodic behaviour of hands washing and teeth brushing, these are studied alongside a group of other quotidian activities which also exhibit continuity during their performance. These studies include the investigation of informative features for activity recognition as well as relevant classification models and signal processing techniques. In addition, a novel multi-level refinement approach is proposed as a way to improve the classification rate of those activities with lower inter-activity classification rate. Second, a novel framework for fluid and food intake gesture recognition is developed. As opposed to the above activities, the nature of eating and drinking activities is neither static nor quasi-periodic. Instead, they are composed of sparsely occurring motions or gestures in continuous data streams. Given this characteristic, a novel signal segmentation technique, namely the Crossings-based Adaptive Segmentation Technique (CAST), is proposed to identify potential eating and drinking gestures while filtering out the remaining unwanted segments of the signals. In addition, various feature descriptors, namely a Soft Dynamic Time Warping (DTW) gesture discrepancy measure and time series to image encoding techniques, as well as various deep learning architectures are explored to overcome the notable existing similarity between eating and drinking gestures. The third stage of the work aims at the identification of meal periods through the analysis of the distribution of eating gestures along time using low-computational cost signal processing techniques, including a moving average and an entropy measure. The novel computational solutions and the results presented in this thesis, demonstrate a significant contribution towards the recognition of quotidian activities in support of independent living

A deep learning based wearable system for food and drink intake recognition

Eating difficulties and the subsequent need for eating assistance are a prevalent issue within the elderly population. Besides, a poor diet is considered a confounding factor for developing chronic diseases and functional limitations. Driven by the above issues, this paper proposes a wrist-worn tri-axial accelerometer based food and drink intake recognition system. First, an adaptive segmentation technique is employed to identify potential eating and drinking gestures from the continuous accelerometer readings. A posteriori, a study upon the use of Convolutional Neural Networks for the recognition of eating and drinking gestures is carried out. This includes the employment of three time series to image encoding frameworks, namely the signal spectrogram, the Markov Transition Field and the Gramian Angular Field, as well as the development of various multi-input multi-domain networks. The recognition of the gestures is then tackled as a 3-class classification problem (‘Eat’, ‘Drink’ and ‘Null’), where the ‘Null’ class is composed of all the irrelevant gestures included in the post-segmentation gesture set. An average per-class classification accuracy of 97.10% was achieved by the proposed system. When compared to similar work, such accurate classification performance signifies a great contribution to the field of assisted living

A computational approach to gestural interactions of the upper limb on planar surfaces

There are many compelling reasons for proposing new gestural interactions: one might want to use a novel sensor that affords access to data that couldn’t be previously captured, or transpose a well-known task into a different unexplored scenario. After an initial design phase, the creation, optimisation or understanding of new interactions remains, however, a challenge. Models have been used to foresee interaction properties: Fitts’ law, for example, accurately predicts movement time in pointing and steering tasks. But what happens when no existing models apply? The core assertion to this work is that a computational approach provides frameworks and associated tools that are needed to model such interactions. This is supported through three research projects, in which discriminative models are used to enable interactions, optimisation is included as an integral part of their design and reinforcement learning is used to explore motions users produce in such interactions