Feature selection of facial displays for detection of non verbal communication in natural conversation

Recognition of human communication has previously focused on deliberately acted emotions or in structured or artificial social contexts. This makes the result hard to apply to realistic social situations. This paper describes the recording of spontaneous human communication in a specific and common social situation: conversation between two people. The clips are then annotated by multiple observers to reduce individual variations in interpretation of social signals. Temporal and static features are generated from tracking using heuristic and algorithmic methods. Optimal features for classifying examples of spontaneous communication signals are then extracted by AdaBoost. The performance of the boosted classifier is comparable to human performance for some communication signals, even on this challenging and realistic data set

Sign Language Recognition

This chapter covers the key aspects of sign-language recognition (SLR), starting with a brief introduction to the motivations and requirements, followed by a précis of sign linguistics and their impact on the field. The types of data available and the relative merits are explored allowing examination of the features which can be extracted. Classifying the manual aspects of sign (similar to gestures) is then discussed from a tracking and non-tracking viewpoint before summarising some of the approaches to the non-manual aspects of sign languages. Methods for combining the sign classification results into full SLR are given showing the progression towards speech recognition techniques and the further adaptations required for the sign specific case. Finally the current frontiers are discussed and the recent research presented. This covers the task of continuous sign recognition, the work towards true signer independence, how to effectively combine the different modalities of sign, making use of the current linguistic research and adapting to larger more noisy data set

On the automatic recognition of facial non-verbal communication meaning in informal, spontaneous conversation

Non-Verbal Communication (NVC) comprises all forms of inter-personal communication, apart from those that are based on words. NVC is essential to understand communicated meaning in common social situations, such as informal conversation. The expression and perception of NVC depends on many factors, including social and cultural context. The development of methods to automatically recognise NVC enables new, intuitive computer interfaces for novel applications, particularly when combined with emotion or natural speech recognition. This thesis addresses two questions: how can facial NVC signals be automatically recognised, given cultural differences in NVC perception? and, what do automatic recognition methods tell us about facial behaviour during informal conversations? A new data set was created based on recordings of people engaged in informal conversation. Minimal constraints were applied during the recording of the participants to ensure that the conversations were spontaneous. These conversations were annotated by volunteer observers, as well as paid workers via the Internet. This resulted in three sets of culturally specific annotations based on the geographical location of the annotator (Great Britain, India, Kenya). The cultures differed in the average label that the culture's annotators assigned to each video clip. Annotations were based on four NVC signals: agreement , thinking, questioning and understanding, all of which commonly occur in conversations. An automatic NVC recognition system was trained based on culturally specific annotation data and was able to make predictions that reflected cultural differences in annotation. Various visual feature extraction methods and classifiers were compared to find an effective recognition approach. The problem was also considered from the perspective of regression of dimensional, continuous valued annotation labels, using Support Vector Regression (SVR), which enables the prediction of labels which have richer information content than discrete classes. The use of Sequential Backward Elimination (SBE) feature selection was shown to greatly increase recognition performance. With a method for extracting the relevant facial features, it becomes possible to investigate human behaviour in informal conversation using computer tools. Firstly, the areas of the face used by the automatic recognition system can be identified and visualised. The involvement of gaze in thinking is confirmed, and a. new association between gestures and NVC are identified, i.e. brow lowering (AU4) during questioning. These findings provide clues as to the way humans perceive NVC. Secondly, the existence of coupling in human expression is quantified and visua1ised. Patterns exist in both mutual head pose and in the mouth area, some of which may relate to mutual smiling. This coupling effect is used in an automatic NVC recognition system based on backchannel signals.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

On the Automatic Recognition of Facial Non-Verbal Communication Meaning in Informal, Spontaneous Conversation.

Non-Verbal Communication (NVC) comprises all forms of inter-personal communication, apart from those that are based on words. NVC is essential to understand communicated meaning in common social situations, such as informal conversation. The expression and perception of NVC depends on many factors, including social and cultural context. The development of methods to automatically recognise NVC enables new, intuitive computer interfaces for novel applications, particularly when combined with emotion or natural speech recognition. This thesis addresses two questions: how can facial NVC signals be automatically recognised, given cultural differences in NVC perception? and, what do automatic recognition methods tell us about facial behaviour during informal conversations? A new data set was created based on recordings of people engaged in informal conversation. Minimal constraints were applied during the recording of the participants to ensure that the conversations were spontaneous. These conversations were annotated by volunteer observers, as well as paid workers via the Internet. This resulted in three sets of culturally specific annotations based on the geographical location of the annotator (Great Britain, India, Kenya). The cultures differed in the average label that the culture’s annotators assigned to each video clip. Annotations were based on four NVC signals: agreement, thinking, questioning and understanding, all of which commonly occur in conversations. An automatic NVC recognition system was trained based on culturally specific annotation data and was able to make predictions that reflected cultural differences in annotation. Various visual feature extraction methods and classifiers were compared to find an effective recognition approach. The problem was also considered from the perspective of regression of dimensional, continuous valued annotation labels, using Support Vector Regression (SVR), which enables the prediction of labels which have richer information content than discrete classes. The use of Sequential Backward Elimination (SBE) feature selection was shown to greatly increase recognition performance. With a method for extracting the relevant facial features, it becomes possible to investigate human behaviour in informal conversation using computer tools. Firstly, the areas of the face used by the automatic recognition system can be identified and visualised. The involvement of gaze in thinking is confirmed, and a new association between gestures and NVC are identified, i.e. brow lowering (AU4) during questioning. These findings provide clues as to the way humans perceive NVC. Secondly, the existence of coupling in human expression is quantified and visualised. Patterns exist in both mutual head pose and in the mouth area, some of which may relate to mutual smiling. This coupling effect is used in an automatic NVC recognition system based on backchannel signals

Design and operation of small hydropower projects in Scotland

In recent years, government incentives have led to the development of numerous small hydropower projects in the Scottish Highlands, four of which are described in this paper. The paper describes the control systems and the choices made to optimize the projects, which all have complex control systems and are operated remotely. It reviews construction problems encountered, including those attributable to working in a harsh environment, and experience gained from operating the completed schemes

Online learning of robust facial feature trackers

This paper presents a head pose and facial feature estimation technique that works over a wide range of pose variations without a priori knowledge of the appearance of the face. Using simple LK trackers, head pose is estimated by Levenberg-Marquardt (LM) pose estimation using the feature tracking as constraints. Factored sampling and RANSAC are employed to both provide a robust pose estimate and identify tracker drift by constraining outliers in the estimation process. The system provides both a head pose estimate and the position of facial features and is capable of tracking over a wide range of head poses

Non-linear Predictors for Facial Feature Tracking Across Pose and Expression

Abstract—This paper proposes a non-linear predictor for estimating the displacement of tracked feature points on faces that exhibit significant variations across pose and expression. Existing methods such as linear predictors, ASMs or AAMs are limited to a narrow range in pose. In order to track across a large pose range, separate pose-specific models are required that are then coupled via a pose-estimator. In our approach, we neither require a set of pose-specific models nor a poseestimator. Using just a single tracking model, we are able to robustly and accurately track across a wide range of expression on poses. This is achieved by gradient boosting of regression trees for predicting the displacement vectors of tracked points. Additionally, we propose a novel algorithm for simultaneously configuring this hierarchical set of trackers for optimal tracking results. Experiments were carried out on sequences of naturalistic conversation and sequences with large pose and expression changes. The results show that the proposed method is superior to state of the art methods, in being able to robustly track a set of facial points whilst gracefully recovering from tracking failures. I

Dams for small hydropower in Scotland

Over recent years government incentives have led to the development of numerous small hydropower schemes in the Scottish Highlands of which five are described in this paper. The schemes have Francis or Pelton turbines with installed capacities between 0.7 and 2.0 MW with gross heads of up to 220 m. The dams are between 4.7and 13.6 m high and of various types. The dams are all “smart dams” in that they have sophisticated control systems and are operated remotely. The paper describes the control systems and the choices made to optimise the schemes. It reviews construction problems encountered, including those attributable to working in a harsh environment, and experience gained from operating the completed schemes

Non-linear predictors for facial feature tracking across pose and expression

This paper proposes a non-linear predictor for estimating the displacement of tracked feature points on faces that exhibit significant variations across pose and expression. Existing methods such as linear predictors, ASMs or AAMs are limited to a narrow range in pose. In order to track across a large pose range, separate pose-specific models are required that are then coupled via a pose-estimator. In our approach, we neither require a set of pose-specific models nor a pose-estimator. Using just a single tracking model, we are able to robustly and accurately track across a wide range of expression on poses. This is achieved by gradient boosting of regression trees for predicting the displacement vectors of tracked points. Additionally, we propose a novel algorithm for simultaneously configuring this hierarchical set of trackers for optimal tracking results. Experiments were carried out on sequences of naturalistic conversation and sequences with large pose and expression changes. The results show that the proposed method is superior to state of the art methods, in being able to robustly track a set of facial points whilst gracefully recovering from tracking failures. © 2013 IEEE

Cultural factors in the regression of non-verbal communication perception

Recognition of non-verbal communication (NVC) is important for understanding human communication and designing user centric user interfaces. Cultural differences affect the expression and perception of NVC but no previous automatic system considers these cultural differences. Annotation data for the LILiR TwoTalk corpus, containing dyadic (two person) conversations, was gathered using Internet crowdsourcing, with a significant quantity collected from India, Kenya and the United Kingdom (UK). Many studies have investigated cultural differences based on human observations but this has not been addressed in the context of automatic emotion or NVC recognition. Perhaps not surprisingly, testing an automatic system on data that is not culturally representative of the training data is seen to result in low performance. We address this problem by training and testing our system on a specific culture to enable better modeling of the cultural differences in NVC perception. The system uses linear predictor tracking, with features generated based on distances between pairs of trackers. The annotations indicated the strength of the NVC which enables the use of v-SVR to perform the regression