1 research outputs found
Understanding Subjectivity through the Lens of Motivational Context in Model-Generated Image Satisfaction
Image generation models are poised to become ubiquitous in a range of
applications. These models are often fine-tuned and evaluated using human
quality judgments that assume a universal standard, failing to consider the
subjectivity of such tasks. To investigate how to quantify subjectivity, and
the scale of its impact, we measure how assessments differ among human
annotators across different use cases. Simulating the effects of ordinarily
latent elements of annotators subjectivity, we contrive a set of motivations
(t-shirt graphics, presentation visuals, and phone background images) to
contextualize a set of crowdsourcing tasks. Our results show that human
evaluations of images vary within individual contexts and across combinations
of contexts. Three key factors affecting this subjectivity are image
appearance, image alignment with text, and representation of objects mentioned
in the text. Our study highlights the importance of taking individual users and
contexts into account, both when building and evaluating generative model