Automatic classification of human facial features based on their appearance

[EN] Classification or typology systems used to categorize different human body parts have existed for many years. Nevertheless, there are very few taxonomies of facial features. Ergonomics, forensic anthropology, crime prevention or new human-machine interaction systems and online activities, like e-commerce, e-learning, games, dating or social networks, are fields in which classifications of facial features are useful, for example, to create digital interlocutors that optimize the interactions between human and machines. However, classifying isolated facial features is difficult for human observers. Previous works reported low inter-observer and intra-observer agreement in the evaluation of facial features. This work presents a computer-based procedure to automatically classify facial features based on their global appearance. This procedure deals with the difficulties associated with classifying features using judgements from human observers, and facilitates the development of taxonomies of facial features. Taxonomies obtained through this procedure are presented for eyes, mouths and noses.Fuentes-Hurtado, F.; Diego-Mas, JA.; Naranjo Ornedo, V.; Alcañiz Raya, ML. (2019). Automatic classification of human facial features based on their appearance. PLoS ONE. 14(1):1-20. https://doi.org/10.1371/journal.pone.0211314S120141Damasio, A. R. (1985). Prosopagnosia. Trends in Neurosciences, 8, 132-135. doi:10.1016/0166-2236(85)90051-7Bruce, V., & Young, A. (1986). Understanding face recognition. British Journal of Psychology, 77(3), 305-327. doi:10.1111/j.2044-8295.1986.tb02199.xTodorov, A. (2011). Evaluating Faces on Social Dimensions. Social Neuroscience, 54-76. doi:10.1093/acprof:oso/9780195316872.003.0004Little, A. C., Burriss, R. P., Jones, B. C., & Roberts, S. C. (2007). Facial appearance affects voting decisions. Evolution and Human Behavior, 28(1), 18-27. doi:10.1016/j.evolhumbehav.2006.09.002Porter, J. P., & Olson, K. L. (2001). Anthropometric Facial Analysis of the African American Woman. Archives of Facial Plastic Surgery, 3(3), 191-197. doi:10.1001/archfaci.3.3.191Gündüz Arslan, S., Genç, C., Odabaş, B., & Devecioğlu Kama, J. (2007). Comparison of Facial Proportions and Anthropometric Norms Among Turkish Young Adults With Different Face Types. Aesthetic Plastic Surgery, 32(2), 234-242. doi:10.1007/s00266-007-9049-yFerring, V., & Pancherz, H. (2008). Divine proportions in the growing face. American Journal of Orthodontics and Dentofacial Orthopedics, 134(4), 472-479. doi:10.1016/j.ajodo.2007.03.027Mane, D. R., Kale, A. D., Bhai, M. B., & Hallikerimath, S. (2010). Anthropometric and anthroposcopic analysis of different shapes of faces in group of Indian population: A pilot study. Journal of Forensic and Legal Medicine, 17(8), 421-425. doi:10.1016/j.jflm.2010.09.001Ritz-Timme, S., Gabriel, P., Tutkuviene, J., Poppa, P., Obertová, Z., Gibelli, D., … Cattaneo, C. (2011). Metric and morphological assessment of facial features: A study on three European populations. Forensic Science International, 207(1-3), 239.e1-239.e8. doi:10.1016/j.forsciint.2011.01.035Ritz-Timme, S., Gabriel, P., Obertovà, Z., Boguslawski, M., Mayer, F., Drabik, A., … Cattaneo, C. (2010). A new atlas for the evaluation of facial features: advantages, limits, and applicability. International Journal of Legal Medicine, 125(2), 301-306. doi:10.1007/s00414-010-0446-4Kong, S. G., Heo, J., Abidi, B. R., Paik, J., & Abidi, M. A. (2005). Recent advances in visual and infrared face recognition—a review. Computer Vision and Image Understanding, 97(1), 103-135. doi:10.1016/j.cviu.2004.04.001Tavares, G., Mourão, A., & Magalhães, J. (2016). Crowdsourcing facial expressions for affective-interaction. Computer Vision and Image Understanding, 147, 102-113. doi:10.1016/j.cviu.2016.02.001Buckingham, G., DeBruine, L. M., Little, A. C., Welling, L. L. M., Conway, C. A., Tiddeman, B. P., & Jones, B. C. (2006). Visual adaptation to masculine and feminine faces influences generalized preferences and perceptions of trustworthiness. Evolution and Human Behavior, 27(5), 381-389. doi:10.1016/j.evolhumbehav.2006.03.001Boberg M, Piippo P, Ollila E. Designing Avatars. DIMEA ‘08 Proc 3rd Int Conf Digit Interact Media Entertain Arts. ACM; 2008; 232–239. doi: https://doi.org/10.1145/1413634.1413679Rojas Q., M., Masip, D., Todorov, A., & Vitria, J. (2011). Automatic Prediction of Facial Trait Judgments: Appearance vs. Structural Models. PLoS ONE, 6(8), e23323. doi:10.1371/journal.pone.0023323Laurentini, A., & Bottino, A. (2014). Computer analysis of face beauty: A survey. Computer Vision and Image Understanding, 125, 184-199. doi:10.1016/j.cviu.2014.04.006Alemany S, Gonzalez J, Nacher B, Soriano C, Arnaiz C, Heras H. Anthropometric survey of the Spanish female population aimed at the apparel industry. Proceedings of the 2010 Intl Conference on 3D Body scanning Technologies. 2010. pp. 307–315.Vinué, G., Epifanio, I., & Alemany, S. (2015). Archetypoids: A new approach to define representative archetypal data. Computational Statistics & Data Analysis, 87, 102-115. doi:10.1016/j.csda.2015.01.018Jee, S., & Yun, M. H. (2016). An anthropometric survey of Korean hand and hand shape types. International Journal of Industrial Ergonomics, 53, 10-18. doi:10.1016/j.ergon.2015.10.004Kim, N.-S., & Do, W.-H. (2014). Classification of Elderly Women’s Foot Type. Journal of the Korean Society of Clothing and Textiles, 38(3), 305-320. doi:10.5850/jksct.2014.38.3.305Sarakon P, Charoenpong T, Charoensiriwath S. Face shape classification from 3D human data by using SVM. The 7th 2014 Biomedical Engineering International Conference. IEEE; 2014. pp. 1–5. doi: https://doi.org/10.1109/BMEiCON.2014.7017382PRESTON, T. A., & SINGH, M. (1972). Redintegrated Somatotyping. Ergonomics, 15(6), 693-700. doi:10.1080/00140137208924469Lin, Y.-L., & Lee, K.-L. (1999). Investigation of anthropometry basis grouping technique for subject classification. Ergonomics, 42(10), 1311-1316. doi:10.1080/001401399184965Malousaris, G. G., Bergeles, N. K., Barzouka, K. G., Bayios, I. A., Nassis, G. P., & Koskolou, M. D. (2008). Somatotype, size and body composition of competitive female volleyball players. Journal of Science and Medicine in Sport, 11(3), 337-344. doi:10.1016/j.jsams.2006.11.008Carvalho, P. V. R., dos Santos, I. L., Gomes, J. O., Borges, M. R. S., & Guerlain, S. (2008). Human factors approach for evaluation and redesign of human–system interfaces of a nuclear power plant simulator. Displays, 29(3), 273-284. doi:10.1016/j.displa.2007.08.010Fabri M, Moore D. The use of emotionally expressive avatars in Collaborative Virtual Environments. AISB’05 Convention:Proceedings of the Joint Symposium on Virtual Social Agents: Social Presence Cues for Virtual Humanoids Empathic Interaction with Synthetic Characters Mind Minding Agents. 2005. pp. 88–94. doi:citeulike-article-id:790934Sukhija, P., Behal, S., & Singh, P. (2016). Face Recognition System Using Genetic Algorithm. Procedia Computer Science, 85, 410-417. doi:10.1016/j.procs.2016.05.183Trescak T, Bogdanovych A, Simoff S, Rodriguez I. Generating diverse ethnic groups with genetic algorithms. Proceedings of the 18th ACM symposium on Virtual reality software and technology—VRST ‘12. New York, New York, USA: ACM Press; 2012. p. 1. doi: https://doi.org/10.1145/2407336.2407338Vanezis, P., Lu, D., Cockburn, J., Gonzalez, A., McCombe, G., Trujillo, O., & Vanezis, M. (1996). Morphological Classification of Facial Features in Adult Caucasian Males Based on an Assessment of Photographs of 50 Subjects. Journal of Forensic Sciences, 41(5), 13998J. doi:10.1520/jfs13998jTamir, A. (2011). Numerical Survey of the Different Shapes of the Human Nose. Journal of Craniofacial Surgery, 22(3), 1104-1107. doi:10.1097/scs.0b013e3182108eb3Tamir, A. (2013). Numerical Survey of the Different Shapes of Human Chin. Journal of Craniofacial Surgery, 24(5), 1657-1659. doi:10.1097/scs.0b013e3182942b77Richler, J. J., Cheung, O. S., & Gauthier, I. (2011). Holistic Processing Predicts Face Recognition. Psychological Science, 22(4), 464-471. doi:10.1177/0956797611401753Taubert, J., Apthorp, D., Aagten-Murphy, D., & Alais, D. (2011). The role of holistic processing in face perception: Evidence from the face inversion effect. Vision Research, 51(11), 1273-1278. doi:10.1016/j.visres.2011.04.002Donnelly, N., & Davidoff, J. (1999). The Mental Representations of Faces and Houses: Issues Concerning Parts and Wholes. Visual Cognition, 6(3-4), 319-343. doi:10.1080/135062899395000Davidoff, J., & Donnelly, N. (1990). Object superiority: A comparison of complete and part probes. Acta Psychologica, 73(3), 225-243. doi:10.1016/0001-6918(90)90024-aTanaka, J. W., & Farah, M. J. (1993). Parts and Wholes in Face Recognition. The Quarterly Journal of Experimental Psychology Section A, 46(2), 225-245. doi:10.1080/14640749308401045Wang, R., Li, J., Fang, H., Tian, M., & Liu, J. (2012). Individual Differences in Holistic Processing Predict Face Recognition Ability. Psychological Science, 23(2), 169-177. doi:10.1177/0956797611420575Rhodes, G., Ewing, L., Hayward, W. G., Maurer, D., Mondloch, C. J., & Tanaka, J. W. (2009). Contact and other-race effects in configural and component processing of faces. British Journal of Psychology, 100(4), 717-728. doi:10.1348/000712608x396503Miller, G. A. (1994). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 101(2), 343-352. doi:10.1037/0033-295x.101.2.343Scharff, A., Palmer, J., & Moore, C. M. (2011). Evidence of fixed capacity in visual object categorization. Psychonomic Bulletin & Review, 18(4), 713-721. doi:10.3758/s13423-011-0101-1Meyers, E., & Wolf, L. (2007). Using Biologically Inspired Features for Face Processing. International Journal of Computer Vision, 76(1), 93-104. doi:10.1007/s11263-007-0058-8Cootes, T. F., Edwards, G. J., & Taylor, C. J. (2001). Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(6), 681-685. doi:10.1109/34.927467Ahonen, T., Hadid, A., & Pietikainen, M. (2006). Face Description with Local Binary Patterns: Application to Face Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(12), 2037-2041. doi:10.1109/tpami.2006.244Belhumeur, P. N., Hespanha, J. P., & Kriegman, D. J. (1997). Eigenfaces vs. Fisherfaces: recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7), 711-720. doi:10.1109/34.598228Turk, M., & Pentland, A. (1991). Eigenfaces for Recognition. Journal of Cognitive Neuroscience, 3(1), 71-86. doi:10.1162/jocn.1991.3.1.71Klare B, Jain AK. On a taxonomy of facial features. IEEE 4th International Conference on Biometrics: Theory, Applications and Systems, BTAS 2010. IEEE; 2010. pp. 1–8. doi: https://doi.org/10.1109/BTAS.2010.5634533Chihaoui, M., Elkefi, A., Bellil, W., & Ben Amar, C. (2016). A Survey of 2D Face Recognition Techniques. Computers, 5(4), 21. doi:10.3390/computers5040021Ma, D. S., Correll, J., & Wittenbrink, B. (2015). The Chicago face database: A free stimulus set of faces and norming data. Behavior Research Methods, 47(4), 1122-1135. doi:10.3758/s13428-014-0532-5Asthana A, Zafeiriou S, Cheng S, Pantic M. Incremental face alignment in the wild. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2014. pp. 1859–1866. doi: https://doi.org/10.1109/CVPR.2014.240Bag S, Barik S, Sen P, Sanyal G. A statistical nonparametric approach of face recognition: combination of eigenface & modified k-means clustering. Proceedings Second International Conference on Information Processing. 2008. p. 198.Doukas, C., & Maglogiannis, I. (2010). A Fast Mobile Face Recognition System for Android OS Based on Eigenfaces Decomposition. Artificial Intelligence Applications and Innovations, 295-302. doi:10.1007/978-3-642-16239-8_39Huang P, Huang Y, Wang W, Wang L. Deep embedding network for clustering. Proceedings—International Conference on Pattern Recognition. 2014. pp. 1532–1537. doi: https://doi.org/10.1109/ICPR.2014.272Dizaji KG, Herandi A, Deng C, Cai W, Huang H. Deep Clustering via Joint Convolutional Autoencoder Embedding and Relative Entropy Minimization. Proceedings of the IEEE International Conference on Computer Vision. 2017. doi: https://doi.org/10.1109/ICCV.2017.612Xie J, Girshick R, Farhadi A. Unsupervised deep embedding for clustering analysis [Internet]. Proceedings of the 33rd International Conference on International Conference on Machine Learning—Volume 48. JMLR.org; 2016. pp. 478–487. Available: https://dl.acm.org/citation.cfm?id=3045442Nousi, P., & Tefas, A. (2017). Discriminatively Trained Autoencoders for Fast and Accurate Face Recognition. Communications in Computer and Information Science, 205-215. doi:10.1007/978-3-319-65172-9_18Sirovich, L., & Kirby, M. (1987). Low-dimensional procedure for the characterization of human faces. Journal of the Optical Society of America A, 4(3), 519. doi:10.1364/josaa.4.00051

Colour Constancy: Biologically-inspired Contrast Variant Pooling Mechanism

Pooling is a ubiquitous operation in image processing algorithms that allows for higher-level processes to collect relevant low-level features from a region of interest. Currently, max-pooling is one of the most commonly used operators in the computational literature. However, it can lack robustness to outliers due to the fact that it relies merely on the peak of a function. Pooling mechanisms are also present in the primate visual cortex where neurons of higher cortical areas pool signals from lower ones. The receptive fields of these neurons have been shown to vary according to the contrast by aggregating signals over a larger region in the presence of low contrast stimuli. We hypothesise that this contrast-variant-pooling mechanism can address some of the shortcomings of max-pooling. We modelled this contrast variation through a histogram clipping in which the percentage of pooled signal is inversely proportional to the local contrast of an image. We tested our hypothesis by applying it to the phenomenon of colour constancy where a number of popular algorithms utilise a max-pooling step (e.g. White-Patch, Grey-Edge and Double-Opponency). For each of these methods, we investigated the consequences of replacing their original max-pooling by the proposed contrast-variant-pooling. Our experiments on three colour constancy benchmark datasets suggest that previous results can significantly improve by adopting a contrast-variant-pooling mechanism

STV-based Video Feature Processing for Action Recognition

In comparison to still image-based processes, video features can provide rich and intuitive information about dynamic events occurred over a period of time, such as human actions, crowd behaviours, and other subject pattern changes. Although substantial progresses have been made in the last decade on image processing and seen its successful applications in face matching and object recognition, video-based event detection still remains one of the most difficult challenges in computer vision research due to its complex continuous or discrete input signals, arbitrary dynamic feature definitions, and the often ambiguous analytical methods. In this paper, a Spatio-Temporal Volume (STV) and region intersection (RI) based 3D shape-matching method has been proposed to facilitate the definition and recognition of human actions recorded in videos. The distinctive characteristics and the performance gain of the devised approach stemmed from a coefficient factor-boosted 3D region intersection and matching mechanism developed in this research. This paper also reported the investigation into techniques for efficient STV data filtering to reduce the amount of voxels (volumetric-pixels) that need to be processed in each operational cycle in the implemented system. The encouraging features and improvements on the operational performance registered in the experiments have been discussed at the end

PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume

We present a compact but effective CNN model for optical flow, called PWC-Net. PWC-Net has been designed according to simple and well-established principles: pyramidal processing, warping, and the use of a cost volume. Cast in a learnable feature pyramid, PWC-Net uses the cur- rent optical flow estimate to warp the CNN features of the second image. It then uses the warped features and features of the first image to construct a cost volume, which is processed by a CNN to estimate the optical flow. PWC-Net is 17 times smaller in size and easier to train than the recent FlowNet2 model. Moreover, it outperforms all published optical flow methods on the MPI Sintel final pass and KITTI 2015 benchmarks, running at about 35 fps on Sintel resolution (1024x436) images. Our models are available on https://github.com/NVlabs/PWC-Net.Comment: CVPR 2018 camera ready version (with github link to Caffe and PyTorch code

A factorization approach to inertial affine structure from motion

We consider the problem of reconstructing a 3-D scene from a moving camera with high frame rate using the affine projection model. This problem is traditionally known as Affine Structure from Motion (Affine SfM), and can be solved using an elegant low-rank factorization formulation. In this paper, we assume that an accelerometer and gyro are rigidly mounted with the camera, so that synchronized linear acceleration and angular velocity measurements are available together with the image measurements. We extend the standard Affine SfM algorithm to integrate these measurements through the use of image derivatives