Restoration of Partially Occluded Shapes of Faces using Neural Networks

One of the major difficulties encountered in the development of face image processing algorithms, is the possible presence of occlusions that hide part of the face images to be processed. Typical examples of facial occlusions include sunglasses, beards, hats and scarves. In our work we address the problem of restoring the overall shape of faces given only the shape presentation of a small part of the face. In the experiments described in this paper the shape of a face is defined by a series of landmarks located on the face outline and on the outline of different facial features. We describe the use of a number of methods including a method that utilizes a Hopfield neural network, a method that uses Multi-Layer Perceptron (MLP) neural network, a novel technique which combines Hopfield and MLP together, and a method based on associative search. We analyze comparative experiments in order to assess the performance of the four methods mentioned above. According to the experimental results it is possible to recover with reasonable accuracy the overall shape of faces even in the case that a substantial part of the shape of a given face is not visible. The techniques presented could form the basis for developing face image processing systems capable of dealing with occluded faces

Generative Face Completion

In this paper, we propose an effective face completion algorithm using a deep generative model. Different from well-studied background completion, the face completion task is more challenging as it often requires to generate semantically new pixels for the missing key components (e.g., eyes and mouths) that contain large appearance variations. Unlike existing nonparametric algorithms that search for patches to synthesize, our algorithm directly generates contents for missing regions based on a neural network. The model is trained with a combination of a reconstruction loss, two adversarial losses and a semantic parsing loss, which ensures pixel faithfulness and local-global contents consistency. With extensive experimental results, we demonstrate qualitatively and quantitatively that our model is able to deal with a large area of missing pixels in arbitrary shapes and generate realistic face completion results.Comment: Accepted by CVPR 201

Deep Neural Networks - A Brief History

Introduction to deep neural networks and their history.Comment: 14 pages, 14 figure

FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors

Face Super-Resolution (SR) is a domain-specific super-resolution problem. The specific facial prior knowledge could be leveraged for better super-resolving face images. We present a novel deep end-to-end trainable Face Super-Resolution Network (FSRNet), which makes full use of the geometry prior, i.e., facial landmark heatmaps and parsing maps, to super-resolve very low-resolution (LR) face images without well-aligned requirement. Specifically, we first construct a coarse SR network to recover a coarse high-resolution (HR) image. Then, the coarse HR image is sent to two branches: a fine SR encoder and a prior information estimation network, which extracts the image features, and estimates landmark heatmaps/parsing maps respectively. Both image features and prior information are sent to a fine SR decoder to recover the HR image. To further generate realistic faces, we propose the Face Super-Resolution Generative Adversarial Network (FSRGAN) to incorporate the adversarial loss into FSRNet. Moreover, we introduce two related tasks, face alignment and parsing, as the new evaluation metrics for face SR, which address the inconsistency of classic metrics w.r.t. visual perception. Extensive benchmark experiments show that FSRNet and FSRGAN significantly outperforms state of the arts for very LR face SR, both quantitatively and qualitatively. Code will be made available upon publication.Comment: Chen and Tai contributed equally to this pape

State of the Art in Face Recognition

Notwithstanding the tremendous effort to solve the face recognition problem, it is not possible yet to design a face recognition system with a potential close to human performance. New computer vision and pattern recognition approaches need to be investigated. Even new knowledge and perspectives from different fields like, psychology and neuroscience must be incorporated into the current field of face recognition to design a robust face recognition system. Indeed, many more efforts are required to end up with a human like face recognition system. This book tries to make an effort to reduce the gap between the previous face recognition research state and the future state

Masked face recognition using deep learning: a review

A large number of intelligent models for masked face recognition (MFR) has been recently presented and applied in various fields, such as masked face tracking for people safety or secure authentication. Exceptional hazards such as pandemics and frauds have noticeably accelerated the abundance of relevant algorithm creation and sharing, which has introduced new challenges. Therefore, recognizing and authenticating people wearing masks will be a long-established research area, and more efficient methods are needed for real-time MFR. Machine learning has made progress in MFR and has significantly facilitated the intelligent process of detecting and authenticating persons with occluded faces. This survey organizes and reviews the recent works developed for MFR based on deep learning techniques, providing insights and thorough discussion on the development pipeline of MFR systems. State-of-the-art techniques are introduced according to the characteristics of deep network architectures and deep feature extraction strategies. The common benchmarking datasets and evaluation metrics used in the field of MFR are also discussed. Many challenges and promising research directions are highlighted. This comprehensive study considers a wide variety of recent approaches and achievements, aiming to shape a global view of the field of MFR

Facial Expression Analysis under Partial Occlusion: A Survey

Automatic machine-based Facial Expression Analysis (FEA) has made substantial progress in the past few decades driven by its importance for applications in psychology, security, health, entertainment and human computer interaction. The vast majority of completed FEA studies are based on non-occluded faces collected in a controlled laboratory environment. Automatic expression recognition tolerant to partial occlusion remains less understood, particularly in real-world scenarios. In recent years, efforts investigating techniques to handle partial occlusion for FEA have seen an increase. The context is right for a comprehensive perspective of these developments and the state of the art from this perspective. This survey provides such a comprehensive review of recent advances in dataset creation, algorithm development, and investigations of the effects of occlusion critical for robust performance in FEA systems. It outlines existing challenges in overcoming partial occlusion and discusses possible opportunities in advancing the technology. To the best of our knowledge, it is the first FEA survey dedicated to occlusion and aimed at promoting better informed and benchmarked future work.Comment: Authors pre-print of the article accepted for publication in ACM Computing Surveys (accepted on 02-Nov-2017

Grounding semantics in robots for Visual Question Answering

In this thesis I describe an operational implementation of an object detection and description system that incorporates in an end-to-end Visual Question Answering system and evaluated it on two visual question answering datasets for compositional language and elementary visual reasoning

Facial Inpainting Methods for Robust Face Recognition

Το ανθρώπινο πρόσωπο είναι πιθανώς το πιο χαρακτηριστικό αναγνωριστικό της ταυτότητας ενός ανθρώπου σε κάθε έκφανση της ζωής του. Στη σύχρονη εποχή, η ανάπτυξη των καμερών και των ηλεκτρονικών συσκευών έχει οδηγήσει στην αδιάκοπη παραγωγή και συλλογή εικόνων με πρόσωπα, που βρίσκουν εφαρμογή σε πολλούς τομείς, όπως η εκπαίδευση, η υγεία, τα ηλεκτρονικά παιχνίδια, η ασφάλεια, η ποινική και ιατροδικαστική έρευνα. Είναι προφανές, ότι η πρόοδος σε αυτούς τους τομείς μπορεί να διευκολύνει την καθημερινή ζωή των ανθρώπων και να τους βηθήσει να ζουν σε πιο ασφαλείς κοινωνίες. Όμως, για να μπορέσουν αυτού του είδους οι εφαρμογές να λειτουργήσουν ορθά, απαιτείται η φωτογραφική λήψη προσώπων μεγάλης καθαρότητας και ευκρίνειας. Αυτή η απαίτηση είναι κάτι παραπάνω από δύσκολο να ικανοποιηθεί στις πραγματικές συνθήκες διαβίωσης. Occlusions όπως γυαλιά μυωπίας, γυαλιά ηλίου, μάσκες προσώπου, φουλάρια, χέρια κ.ά. προκαλούν σοβαρές αλλοιώσεις στις φωτογραφίες με πρόσωπα και αποδυναμώνουν την απόδοση της ταυτοποίησης προσώπου, από τις αντίστοιχες εφαρμογές. Παρόλο που ορισμένοι αλγόριθμοι μπορούν να διαχειριστούν την αναγνώριση προσώπου με occlusion, εξακολουθούν να υφίστανται μείωση στην απόδοσή τους εξαιτίας της έκτασης του occlusion. Επομένως, η αφαίρεση των occlusions από τις εικόνες με πρόσωπα είναι μια πολύ σημαντική, αλλά και απαιτητική εργασία. Η δυσκολία της οφείλεται στο γεγονός, ότι μια μέθοδος ανακατασκευής πρέπει να βρει κάποιον τρόπο, ώστε να αποκαταστήσει τα occluded μέρη του προσώπου σε μια μη occluded μορφή, στοχεύοντας στην παραγωγή ενός καθαρού προσώπου. Όπως γνωρίζουμε, τα ανθρώπινα πρόσωπα έχουν παρόμοιο σχήμα και μέγεθος σε γενικές γραμμές. Ωστόσο, ορισμένα χαρακτηριστικά μπορεί να διαφέρουν πολύ με βάση την φυλή, το γένος και την ηλικία τους. Αυτές οι λεπτομέρεις αυξάνουν ακόμα περισσότερο το βαθμό δυσκολίας της διαδικασίας αποκατάστασης του προσώπου. Ο σκοπός αυτής της Πτυχιακής Μελέτης είναι η αποκατάσταση occluded εικόνων με πρόσωπα σε μια μη occluded μορφή, ώστε να διευκολυνθεί η ταυτοποίησή τους. Για να το πετύχουμε αυτό, διερευνούμε ένα πλήθος από μοντέλα, ειδικευμένα στην ανάπλαση του προσώπου και τα αξιολογούμε με βάση την απόδοσή τους στην αναγνώριση προσώπου. Τα μοντέλα στηρίζονται σε δύο κυρίαρχες μεθοδολογίες της ανάπλασης προσώπου. Η πρώτη, επιτηρούμενη μεθοδολογία, γνωστή ως Generative Landmark Guided Face Inpainting (ή LaFIn) αξιοποιεί μερικά από τα πιο καινοτόμα και υπερσύγχρονα εργαλεία στο πεδίο της μηχανικής μάθησης, τα βαθειά νευρωνικά δίκτυα. Η αρχιτεκτονική του LaFIn επωφελείται από την ενσωμάτωση των διακριτών σημείων του προσώπου και επιτυγχάνει την επιθυμητή αποκατάστασή του. Η δεύτερη, μη επιτηρούμενη μέθοδος γνωστή ως Principal Component Pursuit using Side Information, Features and Missing Values (ή PCPSFM) είναι μια γενίκευση της διάσημης μεθόδου Robust Principal Component Analysis (RPCA). Η PCPSFM αξιοποιεί την προϋπάρχουσα γνώση και καταφέρνει να ανακτήσει έναν πίνακα L0, χαμηλού βαθμού, ο οποίος περιέχει το αναπλασμένο πρόσωπο. Ταυτόχρονα, απομονώνει τα occlusions σε έναν ξεχωριστό, αραιό πίνακα S0. Για να αξιολογήσουμε τις προτεινόμενες μεθόδους, δουλέψαμε σε ένα τμήμα του δημοφιλούς συνόλου δεδομένων CelebA, το οποίο περιέχει τις αναπαραστάσεις των προσώπων διάφορων διάσημων προσωπικοτήτων. Για τα πειράματά μας, δημιουργήσαμε occlusions διαφορετικών μεγεθών και σχημάτων, ώστε να αξιολογήσουμε τα μοντέλα υπό πολλαπλές συνθήκες. Όσον αφορά την διαδικασία αξιολόγησης, χρησιμοποιήθηκαν τρία διαφορετικά μοντέλα, που προσπαθούν να εντοπίσουν την κυρίαρχη μεθόδο ανάπλασης, με βάση το ποσοστό των επιτυχημένων ταιριασμάτων μεταξύ των αναπλασμένων και των καθαρών προσώπων όλων των διάσημων προσωπικοτήτων, που εμπεριέχονται στο σύνολο δεδομένων.Human face is probably the most characteristic identifier in every aspect of a person’s life. In modern times, the development of cameras and digital electronics, has led to a non-stop generation and collection of face images enabling applications in numerous fields, like education, health, gaming, security, criminal and forensic investigation. It’s obvious, that the progress in these fields can facilitate people’s daily life and help them live in more secure societies. In order for this kind of applications to function properly, though, faces of high clearness and sharpness are required to be captured. This request is far from easy to satisfy in real world conditions. Occlusions such as eyeglasses, sunglasses, face masks, scarves, hands and more, cause serious corruptions to the face images and weaken the identification performance of face-related applications. Although some algorithms can handle face recognition with occlusion, they still suffer from performance degradation due to occlusion’s extent. Therefore, the removal of occlusions in face images is a very important, yet challenging task. The difficulty of the task lies in the fact that, a reconstruction method has to find a way to restore the occluded face parts to a non-occluded form, aiming to the generation of a clean face. As we know, human faces have similar shapes and appearances in general. However, the feature details may differ substantially among people depending on their race, gender and age. These details are the ones that raise even more the degree of difficulty of the face restoration procedure. The objective of this thesis is the restoration of occluded face images to a nonoccluded form, in order to facilitate their identification. To achieve that, we investigate a number of inpainting models and we evaluate them on face recognition task. The models are based on two principal face inpainting methodologies. The first, supervised method, known as Generative Landmark Guided Face Inpainting (or LaFIn) exploits some of the most innovative and state-of-the-art tools, in the machine learning field, the deep neural networks. LaFIn’s architecture benefits from the integration of facial landmarks and accomplishes the desired face restoration. The second, unsupervised method known as Principal Component Pursuit using Side Information, Features and Missing Values (or PCPSFM) is a variation of the famous Robust Principal Component Analysis (RPCA) method. PCPSFM utilizes domain dependent prior knowledge and manages to recover a lowrank matrix L0, containing the inpainted face. At the same time, it isolates the occlusions in a separate, sparse matrix S0. To evaluate the proposed methods, we worked on a portion of the popular CelebA dataset, which contains face representations of numerous celebrities. For the purpose of our experiments, we created occlusions of different sizes and shapes, in order to test the models under multiple scenarios. Concerning the evaluation process, three different models were employed to detect the dominant inpainting method, based on the percentage of successful matches between the inpainted faces and the clean faces of all the celebrity identities in the dataset