Large Language Models (LLMs) have been observed to encode and perpetuate
harmful associations present in the training data. We propose a theoretically
grounded framework called StereoMap to gain insights into their perceptions of
how demographic groups have been viewed by society. The framework is grounded
in the Stereotype Content Model (SCM); a well-established theory from
psychology. According to SCM, stereotypes are not all alike. Instead, the
dimensions of Warmth and Competence serve as the factors that delineate the
nature of stereotypes. Based on the SCM theory, StereoMap maps LLMs'
perceptions of social groups (defined by socio-demographic features) using the
dimensions of Warmth and Competence. Furthermore, the framework enables the
investigation of keywords and verbalizations of reasoning of LLMs' judgments to
uncover underlying factors influencing their perceptions. Our results show that
LLMs exhibit a diverse range of perceptions towards these groups, characterized
by mixed evaluations along the dimensions of Warmth and Competence.
Furthermore, analyzing the reasonings of LLMs, our findings indicate that LLMs
demonstrate an awareness of social disparities, often stating statistical data
and research findings to support their reasoning. This study contributes to the
understanding of how LLMs perceive and represent social groups, shedding light
on their potential biases and the perpetuation of harmful associations.Comment: Accepted to EMNLP 202