This paper presents a detailed summary of the inconsistencies in the non-commercial releaseof the MORPH-II dataset and covers the steps and strategy taken to clean it. In addition,examples of prior research that made use of the uncleaned data are briefly introduced and thepotential implications on their results are discussed

NC DOCKS at The University of North Carolina Wilmington

Wang, Yishi

English

The University of North Carolina at Greensboro

MORPH-II: Inconsistencies and Cleaning WhitepaperParticipants: G. Bingham, B. Yip, M. Ferguson, and C. NansaloMentors: C. Chen, Y. Wang, and T. KlingNSF-REU Site at UNC Wilmington, Summer 2017AbstractThis paper presents a detailed summary of the inconsistencies in the non-commercial releaseof the MORPH-II dataset and covers the steps and strategy taken to clean it. In addition,examples of prior research that made use of the uncleaned data are briefly introduced and thepotential implications on their results are discussed.1 IntroductionSince its first release in 2006, the MORPH-II dataset [5] has been cited over 500 times. Multipleversions of MORPH-II have been released, but the 2008 non-commercial release will be used andreferred to as MORPH-II in this paper. The dataset is a collection of 55,134 mugshots taken between2003 and late 2007, and includes many images of individuals that were arrested multiple times overthis five year span. This gives the data a longitudinal aspect that has made it very useful in thefield of computer vision and pattern recognition. In particular, the MORPH-II dataset is widelyutilized in research on gender and race classification, as well as age estimation and synthesis. Ourteam was preparing to use this dataset for similar purposes when our preliminary exploration ofthe data uncovered numerous inconsistencies. According to the local police department, most ofthe data gathered for mugshots is self-reported, and technology has helped tremendously in beingable to verify the information being given. As far as we are aware, previous research efforts withMORPH-II have neglected to correct or acknowledge these errors. Accordingly, this paper aimsto provide a thorough explanation of the inconsistencies in MORPH-II and to explicitly detail ourcleaning methodology.1.1 The Original DataIn order to clearly define the inconsistencies in MORPH-II, we first present a summary of thedatabase, drawn from the whitepaper attached to this release [1].Table 1: Number of Images by Gender and Ancestry (n=55,134)Black White Asian Hispanic Other TotalMale 36,832 7,961 141 1,667 44 46,645Female 5,757 2,598 13 102 19 8,489Total 42,589 10,559 154 1,769 63 55,1341Table 2: MetadataVariable Informationid num 6-digit subject identifier (with leading zeros)picture num subject photo numberdob date of birth (mm/dd/yyyy)doa date of arrest (mm/dd/yyyy)race (B, W, A, H, O)gender (M or F)facial hair not recorded (NULL)age integer age (bdoa− dobc)age diff time since last arrest (days)glasses not recorded (NULL)photo image filenameThe gender and race distribution for MORPH-II is listed in Table 1, with metadata detailsprovided in Table 2.1.2 Discovering ErrorsBy using the 6-digit subject identifier, we found there to be 13,617 unique individuals in theMORPH-II dataset. However, when we count the number of unique males and unique females,we get the following result.Table 3: Number of Distinct Individuals by GenderDistinct IndividualsMale 11,459Female 2,159Total 13,618Table 3 shows that the total number of distinct subjects is now 13,618, suggesting that there maybe an individual listed as both male and female. Repeating the same procedure for race producessimilar results in Table 4.Table 4: Number of Distinct Individuals by Race and GenderBlack White Asian Hispanic Other TotalMale 8,838 2,070 49 517 15 11,489Female 1,494 634 6 30 5 2,169Total 10,332 2,704 55 547 20 13,658Clearly, the total number of distinct individuals in Table 4 (13,658) does not agree with the true2count (13,617), illustrating the extent of the inconsistencies in the dataset. In the next section, wediscuss the reasons for these discrepancies and investigate similar inconsistencies with date of birth.2 Inconsistencies in MORPH-IIMany subjects have multiple entries in the MORPH-II dataset. However, some of these subjectshave more than one gender, race, and/or birthdate across their database entries. This causesproblems when trying to use the MORPH-II images to build facial demographic systems. Theinconsistencies among gender, race and birthdate are summarized in Table 5.Table 5: MORPH-II Inconsistencies by AttributeAttribute Number of SubjectsGender 1Race 33Birthdate 1,779Note that for the 457 subjects with only one entry in the dataset, there is no way to checkwhether their information is correct or not. The methods used to resolve the inconsistencies inMORPH-II are detailed in the following section.3 Cleaning Process3.1 Gender InconsistencyThere is only one subject with inconsistent gender in the database, as shown in Figure 1. Since5 of the 6 images were marked as female, and this person does indeed appear to be a female, wechanged picture (b) to female.Figure 1: Gender Inconsistency(a) Female (b) Male (c) Female(d) Female (e) Female (f) FemaleFigure 2: Race Inconsistencies(1a) White (1b) Black (1c) White(2a) Asian (2b) White (2c) Black33.2 Race InconsistenciesThere are 33 subjects with 132 images in MORPH-II with inconsistent race. In order to bestdetermine race in the MORPH-II dataset, human perception was utilized. To reduce personal bias,a group of researchers were trained by the following steps: First, literature on race classification wascarefully selected and reviewed, including [4], [3], and [2]. The literature outlines the significanceof eyes and nose and the insignificance of features such as skin tone. Researchers were also madeaware of possible bias from the other-race-effect: the tendency to more easily recognize faces ofone’s own race. The most popular and effective methods of perceiving race were summarized tocreate race perception guidelines. Next, researchers were trained on human race perception withcorrectly labeled images in MORPH-II. The images of Asians and Hispanics were focused on asthese made up the majority of the misclassifications.After reviewing literature and being trained on race perception, the researchers attempted toidentify the race of the subjects in question. To start, any individual with a clear majority of imagesbelonging to one race was identified as this majority race. For example, the first subject in Figure2 has 24 images which are classified as White, while 1 image is classified as Black. Therefore thissubject is classified as white and (1b) is changed to White. In cases without a clear majority, humanrace perception was applied to perceive the race of the individual by using the information gatheredfrom the literature and the training acquired from the rest of the dataset. In this process, multipleperspectives were also considered to eliminate as much bias as possible. Finally, in the case thatthe race of the individual is unclear or not enough information is available, the subject is identifiedas the “Other” race category. For example, the second subject in Figure 2 is identified as “Other”because she does not clearly exhibit only one race.3.3 Birthdate InconsistenciesThere are 1,779 subjects in MORPH-II with inconsistent birthdates. 1,524 of the 1,779 cases wereresolved with a simple majority, much like with person 1 in Figure 2. However, the remaining 255subjects pose additional problems. For some of them, their birthdates are in a multiway tie. Forothers, there is no majority, or their birthdates differ by several years. This makes it difficult tochoose one birthdate over another.Figure 3: MORPH-II Inconsistency by Age4Figure 3 shows the proportion of subjects’ birthdates that are within a given range in days. Foreach subject whose birthdates differed by no more than one year (366 days to include leap year), wecalculated the mean birthdate and assigned this date to all images of this subject. This strategy wasapplied to 185 subjects. The remaining 70 subjects were set aside as Not For Training. In general,previous studies have used the floor age instead of the decimal age. For example, this means thata subject who is actually 20.6 years old is recorded as simply 20 years old. Thus, it is reasonableto use the mean birthdate when the range is within one year.4 Multiple Versions of Cleaned DatasetsAfter cleaning MORPH-II of gender, race, and birthdate consistencies, three cleaned datasets werecreated:• morphII cleaned v2 - same as original dataset (morph 2008 nonCommercial.csv), but withdob, race, and gender inconsistencies corrected.• morphII go for age - individuals with unidentifiable birthdates are removed from the abovedataset. This leaves all the images with consistent age information that are ready for trainingand testing age estimation models.• morphII holdout for age - images with unidentifiable birthdates (greater than 1 year differencein the inconsistent birthdates).Two new variables are created for each of the above datasets, shown below in Table 6. Thecorrected column takes a value between 0 and 8 representing what changes were made to a givenentry. The indicators and their associated meanings are explained in Table 7.Table 6: New Variables CreatedVariable Informationcorrected indicator (0-8)age dec decimal age (dob− doa)Table 7: Indicators for new variable correctedIndicator Information0 no change1 dob - majority2 dob - averaged3 dob - uncorrectable4 race - majority5 race - perception6 race - too difficult to tell, assigned to Other7 more than 1 change8 gender corrected5Figure 4: Inconsistent Birthdates SummaryFigure 5: Worst Inconsistent Birthdates(a) Age=23 (b) Age=55 (c) Age=23(c) Age=20 (d) Age=21 (e) Age=515 Research Based on Uncleaned MORPH-II DataA substantial amount of research has been done on the MORPH-II dataset. Unfortunately, whenresearchers report, for example, that the total number of subjects in the dataset is 13,618 (when it isactually 13,617) or that the number of males classified as “Other” is three (upon further inspectionone of these three has inconsistent race), this indicates that data used in such research were notproperly cleaned. Without discrediting the important contributions that have been made, suchresearch outcomes could be more accurate if the data were cleaned properly.There will not likely be an enormous impact on model performance for gender or race prediction,because the number of gender and race inconsistencies is relatively small. However, age estima-tion models may see an increased Mean Absolute Error (MAE). Figure 4 shows the summary forinconsistent birthdates. The worst case is a subject whose reported birthdates are 32 years apart(see Figure 5). In some cases, subjects’ birthdates change so that in the dataset their reported agedecreases with time. This could significantly affect models concerned with age estimation or ageprogression.6 ConclusionData validation and cleaning are critical before any research work is conducted. This not onlypreserves the accuracy of research results, but also the integrity. Many researchers base their workoff of previous results, making it even more important to ensure that one’s own work is accurate.7 AcknowledgementsThis material is based in part upon work supported by the National Science Foundation underGrant Numbers DMS-1659288. Any opinions, findings, conclusions, or recommendations expressedin this material are those of the author(s) and do not necessarily reflect the views of the NationalScience Foundation.6References[1] MORPH Non-Commercial Release Whitepaper. http://www.faceaginggroup.com.[2] S. Fu, H. He, and Z.-G. Hou. Learning race from face: A survey. IEEE transactions on patternanalysis and machine intelligence, 36(12):2483–2509, 2014.[3] G. Guo and G. Mu. Human age estimation: What is the influence across race and gender? InComputer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer SocietyConference on, pages 71–78. IEEE, 2010.[4] G. Guo and G. Mu. A study of large-scale ethnicity estimation with gender and age variations. InComputer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer SocietyConference on, pages 79–86. IEEE, 2010.[5] K. Ricanek and T. Tesafaye. Morph: A longitudinal image database of normal adult age-progression. In Automatic Face and Gesture Recognition, 2006. FGR 2006. 7th InternationalConference on, pages 341–345. IEEE, 2006.7

MORPH-II : Inconsistencies and cleaning whitepaper

http://libres.uncg.edu/ir/uncw/f/wangy2017-1.pdf

MORPH-II : Inconsistencies and cleaning whitepaper

Abstract

Similar works

Full text

Available Versions

The University of North Carolina at Greensboro