The wisdom of the crowd: a case of post- to ante-mortem face matching by police super-recognisers

This case report describes novel methodology used to identify a 43-year-old post-mortem photo of a drowned male recovered from a London river in the 1970’s. Embedded in an array of foils, police super-recognisers (n = 25) possessing superior simultaneous face matching ability, and police controls (n = 139) provided confidence ratings as to the similarity of the post-mortem photo to an ante-mortem photo of a man who went missing at about the same time Indicative of a match, compared to controls, super-recognisers provided higher ratings to the target than the foils. Effects were enhanced when drawing on the combined wisdom of super-recogniser crowds, but not control crowds. These findings supported additional case evidence allowing the coroner to rule that the deceased male and missing male were likely one and the same person. A description of how similar super-recogniser wisdom of the crowd procedures could be applied to other visual image identification cases when no other method is feasible is provided

Crowd effects in unfamiliar face matching

Summary: Psychological research shows that humans can not reliably match unfamiliar faces. This presents a practical problem, because identity verification processes in a variety of occupational settings depend on people to perform these tasks reliably. In this context, it is surprising that very few studies have attempted to improve human performance. Here, we investigate whether distributing face matching tasks across groups of individuals might help to solve this problem. Across four studies, we measure the accuracy of the 'crowd' on a standard test of face matching ability and find that aggregating individual responses produces substantial gains in matching accuracy. We discuss the practical implications of this result and also suggest ways in which this approach might be used to improve our understanding of face perception more generally

Simulated AFRS as decision-aids in face matching

Automated Facial Recognition Systems (AFRS) are used by governments, law enforcement agencies and private businesses to verify the identity of individuals. While previous research has compared the performance of AFRS and humans on tasks of one-to-one face matching, little is known about how effectively human operators can use these AFRS as decision-aids. Our aim was to investigate how the prior decision from an AFRS affects human performance on a face matching task, and to establish whether human oversight of AFRS decisions can lead to collaborative performance gains for the human algorithm team. The identification decisions from our simulated AFRS were informed by the performance of a real, state-of-the-art, Deep Convolutional Neural Network (DCNN) AFRS on the same task. Across five pre-registered experiments, human operators used the decisions from highly accurate AFRS (>90%) to improve their own face matching performance compared to baseline (sensitivity gain: Cohen’s d = 0.71-1.28; overall accuracy gain: d = 0.73-1.46). Yet, despite this improvement, AFRS-aided human performance consistently failed to reach the level that the AFRS achieved alone. Even when the AFRS erred only on the face pairs with the highest human accuracy (>89%), participants often failed to correct the system’s errors, while also overruling many correct decisions, raising questions about the conditions under which human oversight might enhance AFRS operation. Overall, these data demonstrate that the human operator is a limiting factor in this simple model of human-AFRS teaming. These findings have implications for the “human-in-the-loop” approach to AFRS oversight in forensic face matching scenariosOutput Status: Forthcomin

Simulated Automated Facial Recognition Systems as Decision-Aids in Forensic Face Matching Tasks

Automated Facial Recognition Systems (AFRS) are used by governments, law enforcement agencies and private businesses to verify the identity of individuals. While previous research has compared the performance of AFRS and humans on tasks of one-to-one face matching, little is known about how effectively human operators can use these AFRS as decision-aids. Our aim was to investigate how the prior decision from an AFRS affects human performance on a face matching task, and to establish whether human oversight of AFRS decisions can lead to collaborative performance gains for the human algorithm team. The identification decisions from our simulated AFRS were informed by the performance of a real, state-of-the-art, Deep Convolutional Neural Network (DCNN) AFRS on the same task. Across five pre-registered experiments, human operators used the decisions from highly accurate AFRS (>90%) to improve their own face matching performance compared to baseline (sensitivity gain: Cohen’s d = 0.71-1.28; overall accuracy gain: d = 0.73-1.46). Yet, despite this improvement, AFRS-aided human performance consistently failed to reach the level that the AFRS achieved alone. Even when the AFRS erred only on the face pairs with the highest human accuracy (>89%), participants often failed to correct the system’s errors, while also overruling many correct decisions, raising questions about the conditions under which human oversight might enhance AFRS operation. Overall, these data demonstrate that the human operator is a limiting factor in this simple model of human-AFRS teaming. These findings have implications for the “human-in-the-loop” approach to AFRS oversight in forensic face matching scenario

Face processing in photographic identity documents

Photographic identity documents (ID) are widely used to prove the bearer’s identity. In classic laboratory experiments, face images are mostly presented in isolation against a white background, while real life photo-ID checks normally compare a face embedded in a document with the holder. Researchers have begun to ask whether this additional document context might affect face matching. Recent research shows that in face matching tasks, embedding faces into passports introduces a response bias, such that viewers are more likely to accept two pictures as showing the same person. The experiments in this thesis examine the cause of this bias, and whether it generalises to other face processing tasks. In the first three experimental chapters (Chapter 2, 3 and 4), the bias is replicated using various identity documents (passports, driving licences, and student-ID) and in different face matching conditions (e.g. varying mismatch prevalence and task difficulty). Results show that the bias does not rely on perceived authority of the ID or the isolated processing of document elements. Instead, it seems to occur only in the presence of both card background and personal information, which converge in photo-ID. The document-induced bias is specific to unfamiliar faces, and occurs in face matching tasks where the documents themselves are task-irrelevant. Chapters 5 and 6 examine the locus of the document bias, testing both encoding and decisional processes. The effect of documents on memory tasks and first impressions is also examined. The results show that the document-bias seems primarily to affect decision-stage processes. The theoretical and practical implications of these findings are discussed

Towards a Fast and Accurate Face Recognition System from Deep Representations

The key components of a machine perception algorithm are feature extraction followed by classification or regression. The features representing the input data should have the following desirable properties: 1) they should contain the discriminative information required for accurate classification, 2) they should be robust and adaptive to several variations in the input data due to illumination, translation/rotation, resolution, and input noise, 3) they should lie on a simple manifold for easy classification or regression. Over the years, researchers have come up with various hand crafted techniques to extract meaningful features. However, these features do not perform well for data collected in unconstrained settings due to large variations in appearance and other nuisance factors. Recent developments in deep convolutional neural networks (DCNNs) have shown impressive performance improvements in various machine perception tasks such as object detection and recognition. DCNNs are highly non-linear regressors because of the presence of hierarchical convolutional layers with non-linear activation. Unlike the hand crafted features, DCNNs learn the feature extraction and feature classification/regression modules from the data itself in an end-to-end fashion. This enables the DCNNs to be robust to variations present in the data and at the same time improve their discriminative ability. Ever-increasing computation power and availability of large datasets have led to significant performance gains from DCNNs. However, these developments in deep learning are not directly applicable to the face analysis tasks due to large variations in illumination, resolution, viewpoint, and attributes of faces acquired in unconstrained settings. In this dissertation, we address this issue by developing efficient DCNN architectures and loss functions for multiple face analysis tasks such as face detection, pose estimation, landmarks localization, and face recognition from unconstrained images and videos. In the first part of this dissertation, we present two face detection algorithms based on deep pyramidal features. The first face detector, called DP2MFD, utilizes the concepts of deformable parts model (DPM) in the context of deep learning. It is able to detect faces of various sizes and poses in unconstrained conditions. It reduces the gap in training and testing of DPM on deep features by adding a normalization layer to the DCNN. The second face detector, called Deep Pyramid Single Shot Face Detector (DPSSD), is fast and capable of detecting faces with large scale variations (especially tiny faces). It makes use of the inbuilt pyramidal hierarchy present in a DCNN, instead of creating an image pyramid. Extensive experiments on publicly available unconstrained face detection datasets show that both these face detectors are able to capture the meaningful structure of faces and perform significantly better than many traditional face detection algorithms. In the second part of this dissertation, we present two algorithms for simultaneous face detection, landmarks localization, pose estimation and gender recognition using DCNNs. The first method called, HyperFace, fuses the intermediate layers of a DCNN using a separate CNN followed by a multi-task learning algorithm that operates on the fused features. The second approach extends HyperFace to incorporate additional tasks of face verification, age estimation, and smile detection, in All-In-One Face. HyperFace and All-In-One Face exploit the synergy among the tasks which improves individual performances. In the third part of this dissertation, we focus on improving the task of face verification by designing a novel loss function that maximizes the inter-class distance and minimizes the intraclass distance in the feature space. We propose a new loss function, called Crystal Loss, that adds an L2-constraint to the feature descriptors which restricts them to lie on a hypersphere of a fixed radius. This module can be easily implemented using existing deep learning frameworks. We show that integrating this simple step in the training pipeline significantly boosts the performance of face verification. We additionally describe a deep learning pipeline for unconstrained face identification and verification which achieves state-of-the-art performance on several benchmark datasets. We provide the design details of the various modules involved in automatic face recognition: face detection, landmark localization and alignment, and face identification/verification. We present experimental results for end-to-end face verification and identification on IARPA Janus Benchmarks A, B and C (IJB-A, IJB-B, IJB-C), and the Janus Challenge Set 5 (CS5). Though DCNNs have surpassed human-level performance on tasks such as object classification and face verification, they can easily be fooled by adversarial attacks. These attacks add a small perturbation to the input image that causes the network to mis-classify the sample. In the final part of this dissertation, we focus on safeguarding the DCNNs and neutralizing adversarial attacks by compact feature learning. In particular, we show that learning features in a closed and bounded space improves the robustness of the network. We explore the effect of Crystal Loss, that enforces compactness in the learned features, thus resulting in enhanced robustness to adversarial perturbations. Additionally, we propose compact convolution, a novel method of convolution that when incorporated in conventional CNNs improves their robustness. Compact convolution ensures feature compactness at every layer such that they are bounded and close to each other. Extensive experiments show that Compact Convolutional Networks (CCNs) neutralize multiple types of attacks, and perform better than existing methods in defending adversarial attacks, without incurring any additional training overhead compared to CNNs

Correlatos neuroanatómicos del déficit de memoria episódica en personas mayores con deterioro cognitivo leve

Programa Oficial de Postgrado en NeurocienciasLa enfermedad de Alzheimer (EA) es la forma más común de demencia y, en su forma esporádica, afecta sobre todo a personas mayores de 65 años. En España, un 8,2% de la población sufre esta enfermedad, cifra que podría triplicarse en el año 2050. Aunque la sintomatologia se asocia con un metabolismo alterado de las proteínas beta-amiloide y tau, aún se desconocen cuáles son los factores que desencadenan tales alteraciones. Nos enfrentamos por tanto a una enfermedad sin tratamiento eficaz y de carácter terminal. Las evidencias sugieren que cuanto más temprana sea la intervención terapéutica más probabilidades habrá de ralentizar la progresión de la enfermedad, por lo que la búsqueda de biomarcadores tempranos se ha convertido en un reto para la neurociencia contemporánea. Tanto los pacientes diagnosticados con EA como las personas mayores no dementes que presentan deterioro cognitivo leve (DCL), estadio considerado como la fase prodrómica, muestran cambios anatómicos en diferentes regiones del lóbulo temporal medial (LTM), lo cual explicaría la pérdida gradual de la memoria episódica, muy especialmente de la memoria asociativa. Pero esta relación no siempre es evidente, como ocurre con las atrofias localizadas en la corteza entorrinal y en la capa CA1 del hipocampo. Inspirados por estos resultados, el presente trabajo tiene un triple objetivo. En primer lugar, determinar en personas mayores con DCL de tipo amnésico (DCLa) la magnitud del deterioro de la memoria asociativa y el grado de reversibilidad cuando se introducen aspectos que facilitan la codificación y consolidación de nuevas asociaciones, como ocurre con la congruencia semántica del contexto en el que se codifican los eventos estimulares. En segundo lugar, determinar si las alteraciones de la memoria asociativa guardan relación con la integridad anatómica de diferentes estructuras del LTM como son la corteza entorrinal, subiculum, Cornu Ammonis (CA) y giro dentado. Y por último, evaluar el impacto del genotipo ApoE4 sobre dicha relación, por ser este el principal factor de riesgo genético para desarrollar la EA. Los resultados han puesto de manifiesto que las personas con DCLa, y muy especialmente las portadoras del genotipo ApoE4, muestran una menor capacidad para establecer y/o recuperar nuevas asociaciones así como para beneficiarse de la congruencia semántica durante la codificación. Este déficit en la memoria asociativa correlaciona con cambios de volumen que afectan fundamentalmente a la corteza entorrinal, a la región CA1 del hipocampo y a la transición CA1-CA2, mientras que la incapacidad para beneficiarse del contexto semántico durante la codificación correlaciona con cambios de volumen en CA. Las diferencias de grupo en lo que a estas relaciones se refiere son independientes del genotipo ApoE. Estos resultados son congruentes con la idea de que el fenotipo cognitivo de la EA guarda una estrecha relación con la distribución topográfica de las lesiones cerebrales que anteceden al diagnóstico de la enfermedad. Además, abren nuevas perspectivas para mejorar nuestro conocimiento sobre los daños cerebrales que caracterizan a las fases prodrómicas de la EA, aspecto que podría tener implicaciones prácticas para el diagnóstico temprano de esta patología neurodegenerativa.Universidad Pablo de Olavide. Departamento de Fisiología, Anatomía y Biología Celula

Investigating face perception in humans and DCNNs

This thesis aims to compare strengths and weaknesses of AI and humans performing face identification tasks, and to use recent advances in machine-learning to develop new techniques for understanding face identity processing. By better understanding underlying processing differences between Deep Convolutional Neural Networks (DCNNs) and humans, it can help improve the ways in which AI technology is used to support human decision-making and deepen understanding of face identity processing in humans and DCNNs. In Chapter 2, I test how the accuracy of humans and DCNNs is affected by image quality and find that humans and DCNNs are affected differently. This has important applied implications, for example, when identifying faces from poor-quality imagery in police investigations, and also points to different processing strategies used by humans and DCNNs. Given these diverging processing strategies, in Chapter 3, I investigate the potential for human and DCNN decisions to be combined in face identification decisions. I find a large overall benefit of 'fusing' algorithm and human face identity judgments, and that this depends on the idiosyncratic accuracy and response patterns of the particular DCNNs and humans in question. This points to new optimal ways that individual humans and DCNNs can be aggregated to improve the accuracy of face identity decisions in applied settings. Building on my background in computer vision, in Chapters 4 and 5, I then aim to better understand face information sampling by humans using a novel combination of eye-tracking and machine-learning approaches. In chapter 4, I develop exploratory methods for studying individual differences in face information sampling strategies. This reveals differences in the way that 'super-recognisers' sample face information compared to typical viewers. I then use DCNNs to assess the computational value of the face information sampled by these two groups of human observers, finding that sampling by 'super-recognisers' contains more computationally valuable face identity information. In Chapter 5, I develop a novel approach to measuring fixations to people in unconstrained natural settings by combining wearable eye-tracking technology with face and body detection algorithms. Together, these new approaches provide novel insight into individual differences in face information sampling, both when looking at faces in lab-based tasks performed on computer monitors and when looking at faces 'in the wild'

Expertise in Applied Face Matching: Training, Forensic Examiners, Super Matchers and Algorithms

Face matching is widely used in applied settings, including policing and border control, to identify persons of interest, where the consequences of an incorrect decision can have profound consequences. It is, therefore, of paramount importance that applied face- matching systems are accurate and reliable. However, humans are generally poor at matching face of people they don’t know, with large individual differences in accuracy. The aim of this thesis was to evaluate different sources of face-matching expertise (training, forensic face examination, superior face matchers and algorithms) and provide recommendations for how to improve face-matching performance in applied settings. Study one presents a survey of face-matching training, providing insights into the diverse and inconsistent approaches organisations use to train face-matching operators. The second study evaluates a two-day professional face-matching training course, demonstrating the limitations of short courses and the risk of introducing a match bias in low performers. In study three the perceptual skill of superior face matchers and forensic face examiners were compared, showing that by combining the selection of high performers with a wisdom of crowds approach, comparable levels of performance to trained examiners can be achieved in quick-decision face matching. Study four investigated the fusion of human face-matching decisions and algorithm similarity scores for faces that were challenging to humans and to the algorithm, highlighting the effectiveness of fusion in improving face-matching performance. Study five compared the operational accuracy of individual examiners and examiner teams on a face-matching task. Teams achieved higher levels of performance than individuals, with performance improving for both groups after fusion with a facial recognition algorithm. The thesis concludes with a discussion of how different sources of face-matching expertise can be used and combined in applied face-matching systems, and highlights areas for further research that would benefit the applied face-matching community