Adaptive Body Gesture Representation for Automatic Emotion Recognition

We present a computational model and a system for the automated recognition of emotions starting from full-body movement. Three-dimensional motion data of full-body movements are obtained either from professional optical motion-capture systems (Qualisys) or from low-cost RGB-D sensors (Kinect and Kinect2). A number of features are then automatically extracted at different levels, from kinematics of a single joint to more global expressive features inspired by psychology and humanistic theories (e.g., contraction index, fluidity, and impulsiveness). An abstraction layer based on dictionary learning further processes these movement features to increase the model generality and to deal with intraclass variability, noise, and incomplete information characterizing emotion expression in human movement. The resulting feature vector is the input for a classifier performing real-time automatic emotion recognition based on linear support vector machines. The recognition performance of the proposed model is presented and discussed, including the tradeoff between precision of the tracking measures (we compare the Kinect RGB-D sensor and the Qualisys motion-capture system) versus dimension of the training dataset. The resulting model and system have been successfully applied in the development of serious games for helping autistic children learn to recognize and express emotions by means of their full-body movement

An original framework for understanding human actions and body language by using deep neural networks

The evolution of both fields of Computer Vision (CV) and Artificial Neural Networks (ANNs) has allowed the development of efficient automatic systems for the analysis of people's behaviour. By studying hand movements it is possible to recognize gestures, often used by people to communicate information in a non-verbal way. These gestures can also be used to control or interact with devices without physically touching them. In particular, sign language and semaphoric hand gestures are the two foremost areas of interest due to their importance in Human-Human Communication (HHC) and Human-Computer Interaction (HCI), respectively. While the processing of body movements play a key role in the action recognition and affective computing fields. The former is essential to understand how people act in an environment, while the latter tries to interpret people's emotions based on their poses and movements; both are essential tasks in many computer vision applications, including event recognition, and video surveillance. In this Ph.D. thesis, an original framework for understanding Actions and body language is presented. The framework is composed of three main modules: in the first one, a Long Short Term Memory Recurrent Neural Networks (LSTM-RNNs) based method for the Recognition of Sign Language and Semaphoric Hand Gestures is proposed; the second module presents a solution based on 2D skeleton and two-branch stacked LSTM-RNNs for action recognition in video sequences; finally, in the last module, a solution for basic non-acted emotion recognition by using 3D skeleton and Deep Neural Networks (DNNs) is provided. The performances of RNN-LSTMs are explored in depth, due to their ability to model the long term contextual information of temporal sequences, making them suitable for analysing body movements. All the modules were tested by using challenging datasets, well known in the state of the art, showing remarkable results compared to the current literature methods

Deep Multi Temporal Scale Networks for Human Motion Analysis

The movement of human beings appears to respond to a complex motor system that contains signals at different hierarchical levels. For example, an action such as ``grasping a glass on a table'' represents a high-level action, but to perform this task, the body needs several motor inputs that include the activation of different joints of the body (shoulder, arm, hand, fingers, etc.). Each of these different joints/muscles have a different size, responsiveness, and precision with a complex non-linearly stratified temporal dimension where every muscle has its temporal scale. Parts such as the fingers responds much faster to brain input than more voluminous body parts such as the shoulder. The cooperation we have when we perform an action produces smooth, effective, and expressive movement in a complex multiple temporal scale cognitive task. Following this layered structure, the human body can be described as a kinematic tree, consisting of joints connected. Although it is nowadays well known that human movement and its perception are characterised by multiple temporal scales, very few works in the literature are focused on studying this particular property. In this thesis, we will focus on the analysis of human movement using data-driven techniques. In particular, we will focus on the non-verbal aspects of human movement, with an emphasis on full-body movements. The data-driven methods can interpret the information in the data by searching for rules, associations or patterns that can represent the relationships between input (e.g. the human action acquired with sensors) and output (e.g. the type of action performed). Furthermore, these models may represent a new research frontier as they can analyse large masses of data and focus on aspects that even an expert user might miss. The literature on data-driven models proposes two families of methods that can process time series and human movement. The first family, called shallow models, extract features from the time series that can help the learning algorithm find associations in the data. These features are identified and designed by domain experts who can identify the best ones for the problem faced. On the other hand, the second family avoids this phase of extraction by the human expert since the models themselves can identify the best set of features to optimise the learning of the model. In this thesis, we will provide a method that can apply the multi-temporal scales property of the human motion domain to deep learning models, the only data-driven models that can be extended to handle this property. We will ask ourselves two questions: what happens if we apply knowledge about how human movements are performed to deep learning models? Can this knowledge improve current automatic recognition standards? In order to prove the validity of our study, we collected data and tested our hypothesis in specially designed experiments. Results support both the proposal and the need for the use of deep multi-scale models as a tool to better understand human movement and its multiple time-scale nature

Visual Tracking Based on Human Feature Extraction from Surveillance Video for Human Recognition

A multimodal human identification system based on face and body recognition may be made available for effective biometric authentication. The outcomes are achieved by extracting facial recognition characteristics using several extraction techniques, including Eigen-face and Principle Component Analysis (PCA). Systems for authenticating people using their bodies and faces are implemented using artificial neural networks (ANN) and genetic optimization techniques as classifiers. Through feature fusion and scores fusion, the biometric systems for the human body and face are merged to create a single multimodal biometric system. Human bodies may be identified with astonishing accuracy and effectiveness thanks to the SDK for the Kinect sensor. To identify people, biometrics aims to mimic the pattern recognition process. In comparison to traditional authentication methods based on secrets and tokens, it is a more dependable and safe option. Human physiological and behavioral traits are used by biometric technologies to identify people automatically. These characteristics must fulfill many criteria, especially those that relate to universality, efficacy, and applicability

Construction Ergonomic Risk and Productivity Assessment Using Mobile Technology and Machine Learning

The construction industry has one of the lowest productivity rates of all industries. To remedy this problem, project managers tend to increase personnel\u27s workload (growing output), or assign more (often insufficiently trained) workers to certain tasks (reducing time). This, however, can expose personnel to work-related musculoskeletal disorders which if sustained over time, lead to health problems and financial loss. This Thesis presents a scientific methodology for collecting time-motion data via smartphone sensors, and analyzing the data for rigorous health and productivity assessment, thus creating new opportunities in research and development within the architecture, engineering, and construction (AEC) domain. In particular, first, a novel hypothesis is proposed for predicting features of a given body posture, followed by an equation for measuring trunk and shoulder flexions. Experimental results demonstrate that for eleven of the thirteen postures, calculated risk levels are identical to true values. Next, a machine learning-based methodology was designed and tested to calculate workers\u27 productivity as well as ergonomic risks due to overexertion. Results show that calculated productivity values are in very close agreement with true values, and all calculated risk levels are identical to actual values. The presented data collection and analysis framework has a great potential to improve existing practices in construction and other domains by overcoming challenges associated with manual observations and direct measurement techniques

From pixels to affect : a study on games and player experience

Is it possible to predict the affect of a user just by observing her behavioral interaction through a video? How can we, for instance, predict a user’s arousal in games by merely looking at the screen during play? In this paper we address these questions by employing three dissimilar deep convolutional neural network architectures in our attempt to learn the underlying mapping between video streams of gameplay and the player’s arousal. We test the algorithms in an annotated dataset of 50 gameplay videos of a survival shooter game and evaluate the deep learned models’ capacity to classify high vs low arousal levels. Our key findings with the demanding leave-onevideo- out validation method reveal accuracies of over 78% on average and 98% at best. While this study focuses on games and player experience as a test domain, the findings and methodology are directly relevant to any affective computing area, introducing a general and user-agnostic approach for modeling affect.This paper is funded, in part, by the H2020 project Com N Play Science (project no: 787476).peer-reviewe

Deep Learning-Based Action Recognition

The classification of human action or behavior patterns is very important for analyzing situations in the field and maintaining social safety. This book focuses on recent research findings on recognizing human action patterns. Technology for the recognition of human action pattern includes the processing technology of human behavior data for learning, technology of expressing feature values ​​of images, technology of extracting spatiotemporal information of images, technology of recognizing human posture, and technology of gesture recognition. Research on these technologies has recently been conducted using general deep learning network modeling of artificial intelligence technology, and excellent research results have been included in this edition