Magdy, Walid

Ross, Björn

Zheng, Wendy

English

Counterspeech has been proposed as a solution to the proliferation of online hate. Research has shown that natural language processing (NLP) approaches could generate such counterspeech automatically, but there are competing ideas for how NLP models might be used for this task and a variety of evaluation metrics whose relationship to one another is unclear. We test three different approaches and collect ratings of the generated counterspeech for 1,740 tweetparticipant pairs to systematically compare the counterspeech on three aspects: quality, effectiveness and user preferences. We examine which model performs best at which metric and which aspects of counterspeech predict user preferences. A free-form text generation approach using ChatGPT performs the most consistently well, though its generations are occasionally unspecific and repetitive. In our experiment, participants’ preferences for counterspeech are predicted by the quality of the counterspeech, not its perceived effectiveness. The results can help future research approach counterspeech evaluation more systematically

Edinburgh Research Explorer

What Makes Good Counterspeech? A Comparison of Generation Approaches and Evaluation Metrics

https://www.research.ed.ac.uk/files/373204757/What_Makes_Good_ZHENG_D0A24072023_AFV_CC_BY.pdf

What Makes Good Counterspeech? A Comparison of Generation Approaches and Evaluation Metrics

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer