Affect Analysis and Membership Recognition in Group Settings

PhD ThesisEmotions play an important role in our day-to-day life in various ways, including, but not limited to, how we humans communicate and behave. Machines can interact with humans more naturally and intelligently if they are able to recognise and understand humans’ emotions and express their own emotions. To achieve this goal, in the past two decades, researchers have been paying a lot of attention to the analysis of affective states, which has been studied extensively across various fields, such as neuroscience, psychology, cognitive science, and computer science. Most of the existing works focus on affect analysis in individual settings, where there is one person in an image or in a video. However, in the real world, people are very often with others, or interact in group settings. In this thesis, we will focus on affect analysis in group settings. Affect analysis in group settings is different from that in individual settings and provides more challenges due to dynamic interactions between the group members, various occlusions among people in the scene, and the complex context, e.g., who people are with, where people are staying and the mutual influences among people in the group. Because of these challenges, there are still a number of open issues that need further investigation in order to advance the state of the art, and explore the methodologies for affect analysis in group settings. These open topics include but are not limited to (1) is it possible to transfer the methods used for the affect recognition of a person in individual settings to the affect recognition of each individual in group settings? (2) is it possible to recognise the affect of one individual using the expressed behaviours of another member in the same group (i.e., cross-subject affect recognition)? (3) can non-verbal behaviours be used for the recognition of contextual information in group settings? In this thesis, we investigate the affect analysis in group settings and propose methods to explore the aforementioned research questions step by step. Firstly, we propose a method for individual affect recognition in both individual and group videos, which is also used for social context prediction, i.e., whether a person is alone or within a group. Secondly, we introduce a novel framework for cross-subject affect analysis in group videos. Specifically, we analyse the correlation of the affect among group members and investigate the automatic recognition of the affect of one subject using the behaviours expressed by another subject in the same group or in a different group. Furthermore, we propose methods for contextual information prediction in group settings, i.e., group membership recognition - to recognise which group of the person belongs. Comprehensive experiments are conducted using two datasets that one contains individual videos and one contains group videos. The experimental results show that (1) the methods used for affect recognition of a person in individual settings can be transferred to group settings; (2) the affect of one subject in a group can be better predicted using the expressive behaviours of another subject within the same group than using that of a subject from a different group; and (3) contextual information (i.e., whether a person is staying alone or within a group, and group membership) can be predicted successfully using non-verbal behaviours

Face Alignment Assisted by Head Pose Estimation

In this paper we propose a supervised initialization scheme for cascaded face alignment based on explicit head pose estimation. We first investigate the failure cases of most state of the art face alignment approaches and observe that these failures often share one common global property, i.e. the head pose variation is usually large. Inspired by this, we propose a deep convolutional network model for reliable and accurate head pose estimation. Instead of using a mean face shape, or randomly selected shapes for cascaded face alignment initialisation, we propose two schemes for generating initialisation: the first one relies on projecting a mean 3D face shape (represented by 3D facial landmarks) onto 2D image under the estimated head pose; the second one searches nearest neighbour shapes from the training set according to head pose distance. By doing so, the initialisation gets closer to the actual shape, which enhances the possibility of convergence and in turn improves the face alignment performance. We demonstrate the proposed method on the benchmark 300W dataset and show very competitive performance in both head pose estimation and face alignment.Comment: Accepted by BMVC201

Novel Tactile-SIFT Descriptor for Object Shape Recognition

Using a tactile array sensor to recognize an object often requires multiple touches at different positions. This process is prone to move or rotate the object, which inevitably increases difficulty in object recognition. To cope with the unknown object movement, this paper proposes a new tactile-SIFT descriptor to extract features in view of gradients in the tactile image to represent objects, to allow the features being invariant to object translation and rotation. The tactile-SIFT segments a tactile image into overlapping subpatches, each of which is represented using a dn-dimensional gradient vector, similar to the classic SIFT descriptor. Tactile-SIFT descriptors obtained from multiple touches form a dictionary of k words, and the bag-of-words method is then used to identify objects. The proposed method has been validated by classifying 18 real objects with data from an off-the-shelf tactile sensor. The parameters of the tactile-SIFT descriptor, including the dimension size dn and the number of subpatches sp, are studied. It is found that the optimal performance is obtained using an 8-D descriptor with three subpatches, taking both the classification accuracy and time efficiency into consideration. By employing tactile-SIFT, a recognition rate of 91.33% has been achieved with a dictionary size of 50 clusters using only 15 touches

iCLAP: Shape Recognition by Combining Proprioception and Touch Sensing

The work presented in this paper was partially supported by the Engineering and Physical Sciences Council (EPSRC) Grant (Ref: EP/N020421/1) and the King’s-China Scholarship Council Ph.D. scholarship

Zwitterionic polymers-armored amyloid-like protein surface combats thrombosis and biofouling

Proteins, cells and bacteria adhering to the surface of medical devices can lead to thrombosis and infection, resulting in significant clinical mortality. Here, we report a zwitterionic polymers-armored amyloid-like protein surface engineering strategy we called as “armored-tank” strategy for dual functionalization of medical devices. The “armored-tank” strategy is realized by decoration of partially conformational transformed LZM (PCTL) assembly through oxidant-mediated process, followed by armoring with super-hydrophilic poly-2-methacryloyloxyethyl phosphorylcholine (pMPC). The outer armor of the “armored-tank” shows potent and durable zone defense against fibrinogen, platelet and bacteria adhesion, leading to long-term antithrombogenic properties over 14 days in vivo without anticoagulation. Additionally, the “fired” PCTL from “armored-tank” actively and effectively kills both Gram-positive and Gram-negative bacterial over 30 days as a supplement to the lacking bactericidal functions of passive outer armor. Overall, this “armored-tank” surface engineering strategy serves as a promising solution for preventing biofouling and thrombotic occlusion of medical devices

Association between the nasopharyngeal microbiome and metabolome in patients with COVID-19

SARS-CoV-2, the causative agent for COVID-19, infect human mainly via respiratory tract, which is heavily inhabited by local microbiota. However, the interaction between SARS-CoV-2 and nasopharyngeal microbiota, and the association with metabolome has not been well characterized. Here, metabolomic analysis of blood, urine, and nasopharyngeal swabs from a group of COVID-19 and non-COVID-19 patients, and metagenomic analysis of pharyngeal samples were used to identify the key features of COVID-19. Results showed lactic acid, l-proline, and chlorogenic acid methyl ester (CME) were significantly reduced in the sera of COVID-19 patients compared with non-COVID-19 ones. Nasopharyngeal commensal bacteria including Gemella morbillorum, Gemella haemolysans and Leptotrichia hofstadii were notably depleted in the pharynges of COVID-19 patients, while Prevotella histicola, Streptococcus sanguinis, and Veillonella dispar were relatively increased. The abundance of G. haemolysans and L. hofstadii were significantly positively associated with serum CME, which might be an anti-SARS-CoV-2 bacterial metabolite. This study provides important information to explore the linkage between nasopharyngeal microbiota and disease susceptibility. The findings were based on a very limited number of patients enrolled in this study; a larger size of cohort will be appreciated for further investigation