Modifying social dimensions of human faces with ModifAE

At first glance, humans extract social judgments from faces, in-cluding how trustworthy, attractive, and aggressive they look.These impressions have profound social, economic, and polit-ical consequences, as they subconsciously influence decisionslike voting and criminal sentencing. Therefore, understand-ing human perception of these judgments is important for thesocial sciences. In this work, we present a modifying autoen-coder (ModifAE, pronounced “modify”) that can model andalter these facial impressions. We assemble a face impressiondataset large enough for training a generative model by ap-plying a state-of-the-art (SOTA) impression predictor to facesfrom CelebA. Then, we apply ModifAE to learn generalizablemodifications of these continuous-valued traits in faces (e.g.,make a face look slightly more intelligent or much less aggres-sive). ModifAE can modify face images to create controlledsocial science experimental datasets, and it can reveal datasetbiases by creating direct visualizations of what makes a facesalient in social dimensions. The ModifAE architecture is alsosmaller and faster than SOTA image-to-image translation mod-els, while outperforming SOTA in quantitative evaluations

Modifying Social Dimensions of Human Faces with ModifAE

At first glance, humans extract social judgments from faces, including how trustworthy, attractive, and aggressive they look. These impressions have profound social, economic, and political consequences, as they subconsciously influence decisions like voting and criminal sentencing. Therefore, understanding human perception of these judgments is important for the social sciences. In this work, we present a modifying autoencoder (ModifAE, pronounced ``modify'') that can model and alter these facial impressions. We assemble a face impression dataset large enough for training a generative model by applying a state-of-the-art (SOTA) impression predictor to faces from CelebA. Then, we apply ModifAE to learn generalizable modifications of these continuous-valued traits in faces (e.g., make a face look slightly more intelligent or much less aggressive). ModifAE can modify face images to create controlled social science experimental datasets, and it can reveal dataset biases by creating direct visualizations of what makes a face salient in social dimensions. The ModifAE architecture is also smaller and faster than SOTA image-to-image translation models, while outperforming SOTA in quantitative evaluations