Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace

Affective computing has been largely limited in terms of available data resources. The need to collect and annotate diverse in-the-wild datasets has become apparent with the rise of deep learning models, as the default approach to address any computer vision task. Some in-the-wild databases have been recently proposed. However: i) their size is small, ii) they are not audiovisual, iii) only a small part is manually annotated, iv) they contain a small number of subjects, or v) they are not annotated for all main behavior tasks (valence-arousal estimation, action unit detection and basic expression classification). To address these, we substantially extend the largest available in-the-wild database (Aff-Wild) to study continuous emotions such as valence and arousal. Furthermore, we annotate parts of the database with basic expressions and action units. As a consequence, for the first time, this allows the joint study of all three types of behavior states. We call this database Aff-Wild2. We conduct extensive experiments with CNN and CNN-RNN architectures that use visual and audio modalities; these networks are trained on Aff-Wild2 and their performance is then evaluated on 10 publicly available emotion databases. We show that the networks achieve state-of-the-art performance for the emotion recognition tasks. Additionally, we adapt the ArcFace loss function in the emotion recognition context and use it for training two new networks on Aff-Wild2 and then re-train them in a variety of diverse expression recognition databases. The networks are shown to improve the existing state-of-the-art. The database, emotion recognition models and source code are available at http://ibug.doc.ic.ac.uk/resources/aff-wild2

Detecção multi-label de action units em múltiplas poses da cabeça com regiões dinâmicas de aprendizado, redes neurais convolucionais e redes neurais recorrentes

Orientadora: Prof. Dra. Olga R. P. BellonDissertação (mestrado) - Universidade Federal do Paraná, Setor de Ciências Exatas, Programa de Pós-Graduação em Informática. Defesa : Curitiba, 11/07/2018Inclui referências: p.39-41Área de concentração: Ciência da ComputaçãoResumo: Este trabalho apresenta a análise de expressões faciais através da detecção multi-label de Action Units (AUs) em múltiplas poses da cabeça. A análise de expressões faciais em múltiplas poses da cabeça é um problema que detectores robustos de AUs devem lidar, pois é incomum uma pessoa manter sempre a mesma pose ao realizar expressões faciais. Para isto, este trabalho propõe uma abordagem de region learning que cria regiões dinâmicas dentro de uma rede neural convolucional (CNN) usando pontos fiduciais faciais. As regiões dinâmicas de aprendizado (DRL) garantem que cada AU esteja no centro da região, assim como siga o movimento da pose da cabeça. A dynamic region learning foi implementada no final da rede neural convolucional VGG-Face, utilizado transfer-learning para iniciar o treinamento. Além disso, para melhorar as detecções, este trabalho explora informações temporais através de uma rede neural recorrente. Para tal, foi treinada uma rede Long-Short Term Memory (LSTM) utilizando características previamente extraídas pela DRL. Os experimentos foram conduzidos na base de dados Facial Expression Recognition and Analysis Challenge (FERA 2017), que contém nove poses diferentes, e mostram que o trabalho proposto foi capaz de se adaptar às noves poses, superando o estado da arte. Palavras-chave: aprendizado profundo, detecção de unidades de ação, análise de expressões faciais, múltiplas poses, regiões dinâmicas de aprendizado, redes neurais convolucionais, redes neurais recorrentes.Abstract: This work presents a facial expression analysis through multi-label detection of Action Units (AUs) on multiple head poses. The facial expression analysis on multiple head poses is an issue that robust AU detectors must deal with, as it is uncommon for a person to keep the same pose while performing facial expressions. To this end, this work proposes a region learning approach that creates dynamic regions of interest inside a convolutional neural network (CNN) using facial landmark points. The dynamic region learning (DRL) ensures that each AU is in the center of the region, and also follows the head pose movement. The DRL was implemented in the final part of the VGG-Face convolutional neural network, using transfer-learning to start the training. Also, to improve the detection, this work explores temporal information through a recurrent neural network. For this, a Long-Short Term Memory (LSTM) network was trained using features previously extracted by the DRL. The experiments were conducted on the Facial Expression Recognition (FERA 2017) database, which contains nine di_erent head poses, and shows that the proposed approach was able to adapt to all the head poses, surpassing the state-of-the-art. Keywords: Deep learning, action units detection, facial expression analysis, multiple head poses, dynamic regions learning, convolutional neural networks, recurrent neural networks

Automatic analysis of facial actions: a survey

As one of the most comprehensive and objective ways to describe facial expressions, the Facial Action Coding System (FACS) has recently received significant attention. Over the past 30 years, extensive research has been conducted by psychologists and neuroscientists on various aspects of facial expression analysis using FACS. Automating FACS coding would make this research faster and more widely applicable, opening up new avenues to understanding how we communicate through facial expressions. Such an automated process can also potentially increase the reliability, precision and temporal resolution of coding. This paper provides a comprehensive survey of research into machine analysis of facial actions. We systematically review all components of such systems: pre-processing, feature extraction and machine coding of facial actions. In addition, the existing FACS-coded facial expression databases are summarised. Finally, challenges that have to be addressed to make automatic facial action analysis applicable in real-life situations are extensively discussed. There are two underlying motivations for us to write this survey paper: the first is to provide an up-to-date review of the existing literature, and the second is to offer some insights into the future of machine recognition of facial actions: what are the challenges and opportunities that researchers in the field face

ALGRNet: Multi-relational adaptive facial action unit modelling for face representation and relevant recognitions

Facial action units (AUs) represent the fundamental activities of a group of muscles, exhibiting subtle changes that are useful for various face analysis tasks. One practical application in real-life situations is the automatic estimation of facial paralysis. This involves analyzing the delicate changes in facial muscle regions and skin textures. It seems logical to assess the severity of facial paralysis by combining well-defined muscle regions (similar to AUs) symmetrically, thus creating a comprehensive facial representation. To this end, we have developed a new model to estimate the severity of facial paralysis automatically and is inspired by the facial action units (FAU) recognition that deals with rich, detailed facial appearance information, such as texture, muscle status, etc. Specifically, a novel Adaptive Local-Global Relational Network (ALGRNet) is designed to adaptively mine the context of well-defined facial muscles and enhance the visual details of facial appearance and texture, which can be flexibly adapted to facial-based tasks, e.g., FAU recognition and facial paralysis estimation. ALGRNet consists of three key structures: (i) an adaptive region learning module that identifies high-potential muscle response regions, (ii) a skip-BiLSTM that models the latent relationships among local regions, enabling better correlation between multiple regional lesion muscles and texture changes, and (iii) a feature fusion&refining module that explores the complementarity between the local and global aspects of the face. We have extensively evaluated ALGRNet to demonstrate its effectiveness using two widely recognized AU benchmarks, BP4D and DISFA. Furthermore, to assess the efficacy of FAUs in subsequent applications, we have investigated their application in the identification of facial paralysis. Experimental findings obtained from a facial paralysis benchmark, meticulously gathered and annotated by medical experts, underscore the potential of utilizing identified AU attributes to estimate the severity of facial paralysis

Facial Expression Analysis under Partial Occlusion: A Survey

Automatic machine-based Facial Expression Analysis (FEA) has made substantial progress in the past few decades driven by its importance for applications in psychology, security, health, entertainment and human computer interaction. The vast majority of completed FEA studies are based on non-occluded faces collected in a controlled laboratory environment. Automatic expression recognition tolerant to partial occlusion remains less understood, particularly in real-world scenarios. In recent years, efforts investigating techniques to handle partial occlusion for FEA have seen an increase. The context is right for a comprehensive perspective of these developments and the state of the art from this perspective. This survey provides such a comprehensive review of recent advances in dataset creation, algorithm development, and investigations of the effects of occlusion critical for robust performance in FEA systems. It outlines existing challenges in overcoming partial occlusion and discusses possible opportunities in advancing the technology. To the best of our knowledge, it is the first FEA survey dedicated to occlusion and aimed at promoting better informed and benchmarked future work.Comment: Authors pre-print of the article accepted for publication in ACM Computing Surveys (accepted on 02-Nov-2017

Redes neurais convolucionais para análise de expressões faciais

Orientador: Luciano SilvaCoorientadora: Olga R. P. BellonDissertação (mestrado) - Universidade Federal do Paraná, Setor de Ciências Exatas, Programa de Pós-Graduação em Informática. Defesa : Curitiba, 31/08/2018Inclui referências: p.40-43Área de concentração: Ciência da ComputaçãoResumo: Este trabalho propõe uma rede neural convolucional (CNN) para efetuar a detecção e estimativa de intensidade de Action Units (AUs), de forma simultânea, em imagens de faces em poses arbitrárias. Na literatura existem vários métodos para detectar e estimar intensidades de AUs, entretanto, poucos lidam com as variações na pose e levam em consideração a correlação entre os AUs e as intensidades. Ainda, ao considerar a inferência conjunta surge o problema de desequilíbrio entre a quantidade de anotações para cada classe, o que dificulta o processo de otimização e generalização. Porém, é necessário lidar com essas restrições para que esses métodos possam ser utilizados em ambientes não controlados. Outro detalhe que dificulta a generalização para esses ambientes é a falta de bases de imagens anotadas. Nesse caso, é possível estender bases com modelos 3D para gerar poses arbitrárias de forma sintética assim como feito no Facial Expression Analysis and Recognition Challenge (FERA) 2017. Portanto, utilizando uma base de poses sintéticas, este trabalho propõe um modelo baseado em uma CNN, chamado AUMPNet, e aprendizado multi-tarefa para detectar e estimar a intensidade de AUs. Além do modelo para inferência conjunta, também é demonstrada uma abordagem para diminuir o desequilíbrio entre as intensidades dos AUs durante a otimização. O desempenho do modelo proposto, utilizando as bases do FERA 2015 e FERA 2017, é similar ao estado-da-arte, sendo superior para algumas AUs individualmente. Palavras-chave: análise de expressões faciais, visão computacional, redes neurais convolucionais.Abstract: This work presents a convolutional neural network (CNN) for joint Action Unit (AU) detection and intensity estimation on images of face in arbitrary head poses. There are a variety of approaches for AU detection and intensity estimation, however, few of them take into account head pose variations and the correlations among AUs and their intensities. Still, the problem of class imbalance appears when considering the joint inference of AUs, making optimization and generalization harder. Though, it is required to cope with these constraints in order to apply these methods in unconstrained environments. Another difficulty is the lack of labelled images in these conditions. In this case, it is possible to extend existing databases of 3D models to produce synthetic images in arbitrary head poses as in Facial Expression Recognition and Analysis Challenge (FERA) 2017. Thus, by using this database of synthetic head poses this work proposes a multi-task CNN based model, called AUMPNet, to detect AUs and estimate their intensity. Moreover, an approach to handle class imbalance among AUs during optimization is shown. The proposed model, when applied on the FERA 2015 and FERA 2017 databases, achieves average results comparable to the state-of-the-art, and surpasses them for some AUs individually. Keywords: facial expression analysis, computer vision, convolutional neural networks

Reconhecimento de expressões faciais na língua de sinais brasileira por meio do sistema de códigos de ação facial

Orientadores: Paula Dornhofer Paro Costa, Kate Mamhy Oliveira KumadaTese (doutorado) - Universidade Estadual de Campinas, Faculdade de Engenharia Elétrica e de ComputaçãoResumo: Surdos ao redor do mundo usam a língua de sinais para se comunicarem, porém, apesar da ampla disseminação dessas línguas, os surdos ou indivíduos com deficiência auditiva ainda enfrentam dificuldades na comunicação com ouvintes, na ausência de um intérprete. Tais dificuldades impactam negativamente o acesso dos surdos à educação, ao mercado de trabalho e aos serviços públicos em geral. As tecnologias assistivas, como o Reconhecimento Automático de Língua de Sinais, do inglês Automatic Sign Language Recognition (ASLR), visam superar esses obstáculos de comunicação. No entanto, o desenvolvimento de sistemas ASLR confiáveis apresenta vários desafios devido à complexidade linguística das línguas de sinais. As línguas de sinais (LSs) são sistemas linguísticos visuoespaciais que, como qualquer outra língua humana, apresentam variações linguísticas globais e regionais, além de um sistema gramatical. Além disso, as línguas de sinais não se baseiam apenas em gestos manuais, mas também em marcadores não-manuais, como expressões faciais. Nas línguas de sinais, as expressões faciais podem diferenciar itens lexicais, participar da construção sintática e contribuir para processos de intensificação, entre outras funções gramaticais e afetivas. Associado aos modelos de reconhecimento de gestos, o reconhecimento da expressões faciais é um componente essencial da tecnologia ASLR. Neste trabalho, propomos um sistema automático de reconhecimento de expressões faciais para Libras, a língua brasileira de sinais. A partir de uma pesquisa bibliográfica, apresentamos um estudo da linguagem e uma taxonomia diferente para expressões faciais de Libras associadas ao sistema de codificação de ações faciais. Além disso, um conjunto de dados de expressões faciais em Libras foi criado. Com base em experimentos, a decisão sobre a construção do nosso sistema foi através de pré-processamento e modelos de reconhecimento. Os recursos obtidos para a classificação das ações faciais são resultado da aplicação combinada de uma região de interesse, e informações geométricas da face dado embasamento teórico e a obtenção de desempenho melhor do que outras etapas testadas. Quanto aos classificadores, o SqueezeNet apresentou melhores taxas de precisão. Com isso, o potencial do modelo proposto vem da análise de 77% da acurácia média de reconhecimento das expressões faciais de Libras. Este trabalho contribui para o crescimento dos estudos que envolvem a visão computacional e os aspectos de reconhecimento da estrutura das expressões faciais da língua de sinais, e tem como foco principal a importância da anotação da ação facial de forma automatizadaAbstract: Deaf people around the world use sign languages to communicate but, despite the wide dissemination of such languages, deaf or hard of hearing individuals still face difficulties in communicating with hearing individuals, in the absence of an interpreter. Such difficulties negatively impact the access of deaf individuals to education, to the job market, and to public services in general. Assistive technologies, such as Automatic Sign Language Recognition (ASLR), aim at overcoming such communication obstacles. However, the development of reliable ASLR systems imposes numerous challenges due the linguistic complexity of sign languages. Sign languages (SLs) are visuospatial linguistic systems that, like any other human language, present global and regional linguistic variations, and a grammatical system. Also, sign languages do not rely only on manual gestures but also non-manual markers, such as facial expressions. In SL, facial expressions may differentiate lexical items, participate in syntactic construction, and contribute to change the intensity of a sentence, among other grammatical and affective functions. Associated with the gesture recognition models, facial expression recognition (FER) is an essential component of ASLR technology. In this work, we propose an automatic facial expression recognition (FER) system for Brazilian Sign Language (Libras). Derived from a literature survey, we present a language study and a different taxonomy for facial expressions of Libras associated with the Facial Action Coding System (FACS). Also, a dataset of facial expressions in Libras was created. An experimental setting was done for the construction of our framework for a preprocessing stage and recognizer model. The features for the classification of the facial actions resulted from the application of a combined region of interest and geometric information given a theoretical basis and better performance than other tested steps. As for classifiers, SqueezeNet returned better accuracy rates. With this, the potential of the proposed model comes from the analysis of 77% of the average accuracy of recognition of Libras' facial expressions. This work contributes to the growth of studies that involve the computational vision and recognition aspects of the structure of sign language facial expressions, and its main focus is the importance of facial action annotation in an automated wayDoutoradoEngenharia de ComputaçãoDoutora em Engenharia Elétrica001CAPE