Recognition of human activities and expressions in video sequences using shape context descriptor

The recognition of objects and classes of objects is of importance in the field of computer vision due to its applicability in areas such as video surveillance, medical imaging and retrieval of images and videos from large databases on the Internet. Effective recognition of object classes is still a challenge in vision; hence, there is much interest to improve the rate of recognition in order to keep up with the rising demands of the fields where these techniques are being applied. This thesis investigates the recognition of activities and expressions in video sequences using a new descriptor called the spatiotemporal shape context. The shape context is a well-known algorithm that describes the shape of an object based upon the mutual distribution of points in the contour of the object; however, it falls short when the distinctive property of an object is not just its shape but also its movement across frames in a video sequence. Since actions and expressions tend to have a motion component that enhances the capability of distinguishing them, the shape based information from the shape context proves insufficient. This thesis proposes new 3D and 4D spatiotemporal shape context descriptors that incorporate into the original shape context changes in motion across frames. Results of classification of actions and expressions demonstrate that the spatiotemporal shape context is better than the original shape context at enhancing recognition of classes in the activity and expression domains

Facial Expression Recognition System

A key requirement for developing any innovative system in a computing environment is to integrate a sufficiently friendly interface with the average end user. Accurate design of such a user-centered interface, however, means more than just the ergonomics of the panels and displays. It also requires that designers precisely define what information to use and how, where, and when to use it. Facial expression as a natural, non-intrusive and efficient way of communication has been considered as one of the potential inputs of such interfaces. The work of this thesis aims at designing a robust Facial Expression Recognition (FER) system by combining various techniques from computer vision and pattern recognition. Expression recognition is closely related to face recognition where a lot of research has been done and a vast array of algorithms have been introduced. FER can also be considered as a special case of a pattern recognition problem and many techniques are available. In the designing of an FER system, we can take advantage of these resources and use existing algorithms as building blocks of our system. So a major part of this work is to determine the optimal combination of algorithms. To do this, we first divide the system into 3 modules, i.e. Preprocessing, Feature Extraction and Classification, then for each of them some candidate methods are implemented, and eventually the optimal configuration is found by comparing the performance of different combinations. Another issue that is of great interest to facial expression recognition systems designers is the classifier which is the core of the system. Conventional classification algorithms assume the image is a single variable function of a underlying class label. However this is not true in face recognition area where the appearance of the face is influenced by multiple factors: identity, expression, illumination and so on. To solve this problem, in this thesis we propose two new algorithms, namely Higher Order Canonical Correlation Analysis and Simple Multifactor Analysis which model the image as a multivariable function. The addressed issues are challenging problems and are substantial for developing a facial expression recognition system

Robust recognition of facial expressions on noise degraded facial images

Magister Scientiae - MScWe investigate the use of noise degraded facial images in the application of facial expression recognition. In particular, we trained Gabor+SVMclassifiers to recognize facial expressions images with various types of noise. We applied Gaussian noise, Poisson noise, varying levels of salt and pepper noise, and speckle noise to noiseless facial images. Classifiers were trained with images without noise and then tested on the images with noise. Next, the classifiers were trained using images with noise, and then on tested both images that had noise, and images that were noiseless. Finally, classifiers were tested on images while increasing the levels of salt and pepper in the test set. Our results reflected distinct degradation of recognition accuracy. We also discovered that certain types of noise, particularly Gaussian and Poisson noise, boost recognition rates to levels greater than would be achieved by normal, noiseless images. We attribute this effect to the Gaussian envelope component of Gabor filters being sympathetic to Gaussian-like noise, which is similar in variance to that of the Gabor filters. Finally, using linear regression, we mapped a mathematical model to this degradation and used it to suggest how recognition rates would degrade further should more noise be added to the images.South Afric

Reconhecimento de espécies florestais através de imagens macroscópicas

Resumo: A identificação de espécies e uma necessidade primordial para as atividades de comércio e preservacao de florestas. Entretanto, devido a escassez de dados e bases de imagens florestais, os estudos computacionais relacionados a esse tema sao raros e recentes. Outros fatores que influenciam a raridade desses estudos estao relacionados a falta de tecnicas computacionais comprovadamente eficazes para essa tarefa e ao custo para a aquisicão de imagens para a construcao das bases e modelos computacionais, uma vez que equipamentos sofisticados e caros sao utilizados. Tendo em vista esse contexto e com objetivo de minimizar os custos relacionados ao processo de identificaçao de especies florestais, e proposta uma nova abordagem para essa tarefa, com a qual a identificaçao podera ser realizada em campo e com equipamentos de baixo valor, agregando maior mobilidade e agilidade à execucao dessa tarefa. Para avaliar e validar essa proposta, foram construídas duas bases de imagens macroscópicas a partir de amostras de madeira de especies florestais encontradas no território nacional, considerando dois metodos diferentes: abordagem tradicional em laboratório e abordagem em campo, sendo esta ultima, a proposta deste trabalho. Um protocolo modular baseado na estratégia de dividir para conquistar foi proposto, nele as imagens sao divididas em subimagens, com o intuito de que problemas locais nao afetem a classificacao geral da imagem. A partir delas, sao extraídas informacoes de cor e textura que sao utilizadas para a construcão de conjuntos de treinamento, teste e validaçao de classificadores. Para extraçao desses atributos sao avaliadas diversas tecnicas consagradas como analises de cor, GLCM, histograma de borda, Fractais, LBP, LPQ e Gabor. Apos a classificação de cada conjunto de atributos das subimagens, seus resultados passam por duas camadas de fusoes (baixo e alto nível), para se chegar a decisão final de qual especie a amostra pertence. Inicialmente, a avaliaçao experimental foi realizada com a base de imagens obtidas a partir da abordagem em campo uma vez que dessa maneira os resultados sao mais conservadores devido à presenca de ruídos nos conjuntos de dados e ao naão tratamento das amostras adquiridas. A taxa de reconhecimento obtida nessa etapa foi 95,82%. Apos a validacao do metodo proposto, os modelos de classificação foram reconstruídos e avaliados a partir da base de imagens criada com a abordagem tradicional em laboratório. Com esse novo modelo, a taxa de classificaçao foi de 99,49%. A partir da analise dos resultados, observa-se a viabilidade da abordagem proposta neste trabalho, que alem de apresentar uma excelente taxa de classificaçao, muito proxima da obtida com tecnicas mais sofisticadas e de alto custo, ainda agrega a mobilidade para a classificacão de especies em campo. Ressalta-se ainda, a construcao e disponibilizacao das bases de imagens florestais, contribuindo, desta forma, para trabalhos futuros nesta area

Affect recognition & generation in-the-wild

Affect recognition based on a subject’s facial expressions has been a topic of major research in the attempt to generate machines that can understand the way subjects feel, act and react. In the past, due to the unavailability of large amounts of data captured in real-life situations, research has mainly focused on controlled environments. However, recently, social media and platforms have been widely used. Moreover, deep learning has emerged as a means to solve visual analysis and recognition problems. This Ph.D. Thesis exploits these advances and makes significant contributions for affect analysis and recognition in-the-wild. We tackle affect analysis and recognition as a dual knowledge generation problem: i) we create new, large and rich in-the-wild databases and ii) we design and train novel deep neural architectures that are able to analyse affect over these databases and to successfully generalise their performance on other datasets. At first, we present the creation of Aff-Wild database annotated according to valence-arousal and an end-to-end CNN-RNN architecture, AffWildNet. Then we use AffWildNet as a robust prior for dimensional and categorical affect recognition and extend it by extracting low-/mid-/high-level latent information and analysing this via multiple RNNs. Additionally, we propose a novel loss function for DNN-based categorical affect recognition. Next, we generate Aff-Wild2, the first database containing annotations for all main behavior tasks: estimate Valence-Arousal; classify into Basic Expressions; detect Action Units. We develop multi-task and multi-modal extensions of AffWildNet by fusing these tasks and propose a novel holistic approach that utilises all existing databases with non-overlapping annotations and couples them through co-annotation and distribution matching. Finally, we present an approach for valence-arousal, or basic expressions’ facial affect synthesis. We generate an image with a given affect, or a sequence of images with evolving affect, by annotating a 4-D database and utilising a 3-D morphable model.Open Acces