Facial emotion recognition using min-max similarity classifier

Recognition of human emotions from the imaging templates is useful in a wide variety of human-computer interaction and intelligent systems applications. However, the automatic recognition of facial expressions using image template matching techniques suffer from the natural variability with facial features and recording conditions. In spite of the progress achieved in facial emotion recognition in recent years, the effective and computationally simple feature selection and classification technique for emotion recognition is still an open problem. In this paper, we propose an efficient and straightforward facial emotion recognition algorithm to reduce the problem of inter-class pixel mismatch during classification. The proposed method includes the application of pixel normalization to remove intensity offsets followed-up with a Min-Max metric in a nearest neighbor classifier that is capable of suppressing feature outliers. The results indicate an improvement of recognition performance from 92.85% to 98.57% for the proposed Min-Max classification method when tested on JAFFE database. The proposed emotion recognition technique outperforms the existing template matching methods

Facial Expression Analysis under Partial Occlusion: A Survey

Automatic machine-based Facial Expression Analysis (FEA) has made substantial progress in the past few decades driven by its importance for applications in psychology, security, health, entertainment and human computer interaction. The vast majority of completed FEA studies are based on non-occluded faces collected in a controlled laboratory environment. Automatic expression recognition tolerant to partial occlusion remains less understood, particularly in real-world scenarios. In recent years, efforts investigating techniques to handle partial occlusion for FEA have seen an increase. The context is right for a comprehensive perspective of these developments and the state of the art from this perspective. This survey provides such a comprehensive review of recent advances in dataset creation, algorithm development, and investigations of the effects of occlusion critical for robust performance in FEA systems. It outlines existing challenges in overcoming partial occlusion and discusses possible opportunities in advancing the technology. To the best of our knowledge, it is the first FEA survey dedicated to occlusion and aimed at promoting better informed and benchmarked future work.Comment: Authors pre-print of the article accepted for publication in ACM Computing Surveys (accepted on 02-Nov-2017

4D (3D Dynamic) statistical models of conversational expressions and the synthesis of highly-realistic 4D facial expression sequences

In this thesis, a novel approach for modelling 4D (3D Dynamic) conversational interactions and synthesising highly-realistic expression sequences is described. To achieve these goals, a fully-automatic, fast, and robust pre-processing pipeline was developed, along with an approach for tracking and inter-subject registering 3D sequences (4D data). A method for modelling and representing sequences as single entities is also introduced. These sequences can be manipulated and used for synthesising new expression sequences. Classification experiments and perceptual studies were performed to validate the methods and models developed in this work. To achieve the goals described above, a 4D database of natural, synced, dyadic conversations was captured. This database is the first of its kind in the world. Another contribution of this thesis is the development of a novel method for modelling conversational interactions. Our approach takes into account the time-sequential nature of the interactions, and encompasses the characteristics of each expression in an interaction, as well as information about the interaction itself. Classification experiments were performed to evaluate the quality of our tracking, inter-subject registration, and modelling methods. To evaluate our ability to model, manipulate, and synthesise new expression sequences, we conducted perceptual experiments. For these perceptual studies, we manipulated modelled sequences by modifying their amplitudes, and had human observers evaluate the level of expression realism and image quality. To evaluate our coupled modelling approach for conversational facial expression interactions, we performed a classification experiment that differentiated predicted frontchannel and backchannel sequences, using the original sequences in the training set. We also used the predicted backchannel sequences in a perceptual study in which human observers rated the level of similarity of the predicted and original sequences. The results of these experiments help support our methods and our claim of our ability to produce 4D, highly-realistic expression sequences that compete with state-of-the-art methods

3D Face Modelling, Analysis and Synthesis

Human faces have always been of a special interest to researchers in the computer vision and graphics areas. There has been an explosion in the number of studies around accurately modelling, analysing and synthesising realistic faces for various applications. The importance of human faces emerges from the fact that they are invaluable means of effective communication, recognition, behaviour analysis, conveying emotions, etc. Therefore, addressing the automatic visual perception of human faces efficiently could open up many influential applications in various domains, e.g. virtual/augmented reality, computer-aided surgeries, security and surveillance, entertainment, and many more. However, the vast variability associated with the geometry and appearance of human faces captured in unconstrained videos and images renders their automatic analysis and understanding very challenging even today. The primary objective of this thesis is to develop novel methodologies of 3D computer vision for human faces that go beyond the state of the art and achieve unprecedented quality and robustness. In more detail, this thesis advances the state of the art in 3D facial shape reconstruction and tracking, fine-grained 3D facial motion estimation, expression recognition and facial synthesis with the aid of 3D face modelling. We give a special attention to the case where the input comes from monocular imagery data captured under uncontrolled settings, a.k.a. \textit{in-the-wild} data. This kind of data are available in abundance nowadays on the internet. Analysing these data pushes the boundaries of currently available computer vision algorithms and opens up many new crucial applications in the industry. We define the four targeted vision problems (3D facial reconstruction $\&$ tracking, fine-grained 3D facial motion estimation, expression recognition, facial synthesis) in this thesis as the four 3D-based essential systems for the automatic facial behaviour understanding and show how they rely on each other. Finally, to aid the research conducted in this thesis, we collect and annotate a large-scale videos dataset of monocular facial performances. All of our proposed methods demonstarte very promising quantitative and qualitative results when compared to the state-of-the-art methods

Artificial Intelligence Tools for Facial Expression Analysis.

Inner emotions show visibly upon the human face and are understood as a basic guide to an individual’s inner world. It is, therefore, possible to determine a person’s attitudes and the effects of others’ behaviour on their deeper feelings through examining facial expressions. In real world applications, machines that interact with people need strong facial expression recognition. This recognition is seen to hold advantages for varied applications in affective computing, advanced human-computer interaction, security, stress and depression analysis, robotic systems, and machine learning. This thesis starts by proposing a benchmark of dynamic versus static methods for facial Action Unit (AU) detection. AU activation is a set of local individual facial muscle parts that occur in unison constituting a natural facial expression event. Detecting AUs automatically can provide explicit benefits since it considers both static and dynamic facial features. For this research, AU occurrence activation detection was conducted by extracting features (static and dynamic) of both nominal hand-crafted and deep learning representation from each static image of a video. This confirmed the superior ability of a pretrained model that leaps in performance. Next, temporal modelling was investigated to detect the underlying temporal variation phases using supervised and unsupervised methods from dynamic sequences. During these processes, the importance of stacking dynamic on top of static was discovered in encoding deep features for learning temporal information when combining the spatial and temporal schemes simultaneously. Also, this study found that fusing both temporal and temporal features will give more long term temporal pattern information. Moreover, we hypothesised that using an unsupervised method would enable the leaching of invariant information from dynamic textures. Recently, fresh cutting-edge developments have been created by approaches based on Generative Adversarial Networks (GANs). In the second section of this thesis, we propose a model based on the adoption of an unsupervised DCGAN for the facial features’ extraction and classification to achieve the following: the creation of facial expression images under different arbitrary poses (frontal, multi-view, and in the wild), and the recognition of emotion categories and AUs, in an attempt to resolve the problem of recognising the static seven classes of emotion in the wild. Thorough experimentation with the proposed cross-database performance demonstrates that this approach can improve the generalization results. Additionally, we showed that the features learnt by the DCGAN process are poorly suited to encoding facial expressions when observed under multiple views, or when trained from a limited number of positive examples. Finally, this research focuses on disentangling identity from expression for facial expression recognition. A novel technique was implemented for emotion recognition from a single monocular image. A large-scale dataset (Face vid) was created from facial image videos which were rich in variations and distribution of facial dynamics, appearance, identities, expressions, and 3D poses. This dataset was used to train a DCNN (ResNet) to regress the expression parameters from a 3D Morphable Model jointly with a back-end classifier

Diretrizes para desenvolvimento de aplicativos de realidade aumentada para crianças com TEA na perspectiva do desenho universal

Orientador: Prof. Dr. Márcio Fontana CatapanDissertação (mestrado) - Universidade Federal do Paraná, Setor de Artes, Comunicação e Design, Programa de Pós-Graduação em Design. Defesa : Curitiba, 30/09/2021Inclui referências: p. 100-108Resumo: O Transtorno do Espectro Autista (TEA) afeta o neurodesenvolvimento de crianças, prejudicando a capacidade de interação social, comunicação e gerando comportamentos repetitivos. O papel do designer nesse contexto é incorporar os princípios de Usabilidade e acessibilidade às terapias específicas realizadas com esse público, desenvolvendo artefatos e estabelecendo diretrizes para tais ferramentas. A Realidade Aumentada (RA) vem colaborando como complemento aos métodos já existentes, praticados por psicólogos/terapeutas. Desta forma, a presente pesquisa se divide em três fases, e utiliza o método de pesquisa Design Science Research, além de técnicas de revisão bibliográfica de literatura para estabelecer lacunas e realizar levantamento do contexto do usuário. O objetivo principal foi o desenvolvimento de diretrizes para criação de aplicativos, baseados no Desenho Universal para aprendizagem de crianças com TEA por meio da Realidade Aumentada, que resultou em 16 tópicos de recomendações conceituadas e detalhadas, reunidas em um material de divulgação. Além disso, foi desenvolvido um aplicativo chamado "Todo dia eu", que conta com dois personagens, um menino e uma menina, com detalhamento das etapas de modelagem 3D, animação, criação e disponibilização do aplicativo para Android. As tarefas dos personagens foram selecionadas conforme protocolo de avaliação do Manual para Intervenção Comportamental Intensiva quanto ao Ensino de Habilidades Básicas para Pessoas com Autismo. O recorte da pesquisa definiu as seguintes ações: "Tocar o nariz", "sentar" e "beber no copo", a fim de treinar habilidades de atenção e de imitação. O aplicativo foi desenvolvido em duas versões, devido à retroalimentação do método aplicado, sendo assim, a pesquisa atingiu o resultado esperado em termos de aplicação das diretrizes, sendo essas o produto final da Design Science Research.Abstract: Autism Spectrum Disorder (ASD) affects the neurodevelopment of children, impairing the ability to social interaction, communication and generating repetitive behaviors. The designer's role in this context is to incorporate the principles of Usability and accessibility to specific therapies performed with this audience, developing artifacts and establishing guidelines for such tools. Augmented Reality (AR) has been collaborating as a complement to existing methods practiced by psychologists/therapists. In this way, the present research is divided into three phases, and uses the Design Science Research research method, in addition to literature review techniques to establish gaps and carry out a survey of the user's context. The main objective was the development of guidelines for creating applications, based on Universal Design for learning by children with ASD through Augmented Reality, which resulted in 16 topics of conceptualized and detailed recommendations, gathered in a dissemination material. In addition, an application called "Todo dia eu" (Every day I) was developed, which has two characters, a boy and a girl, with details of the steps of 3D modeling, animation, creation and availability of the application for Android. The characters' tasks were selected according to the evaluation protocol of the Manual for Intensive Behavioral Intervention regarding the Teaching of Basic Skills for People with Autism. The research cutout defined the following actions: "Touch the nose", "sit" and "drink from a glass", to train attention and imitation skills. The application was developed in two versions, due to the feedback of the applied method, therefore, the research reached the expected result in terms of application of the guidelines, which are the final product of Design Science Research