Comment sections below online news articles enjoy growing popularity among
readers. However, the overwhelming number of comments makes it infeasible for
the average news consumer to read all of them and hinders engaging discussions.
Most platforms display comments in chronological order, which neglects that
some of them are more relevant to users and are better conversation starters.
In this paper, we systematically analyze user engagement in the form of the
upvotes and replies that a comment receives. Based on comment texts, we train a
model to distinguish comments that have either a high or low chance of
receiving many upvotes and replies. Our evaluation on user comments from
TheGuardian.com compares recurrent and convolutional neural network models, and
a traditional feature-based classifier. Further, we investigate what makes some
comments more engaging than others. To this end, we identify engagement
triggers and arrange them in a taxonomy. Explanation methods for neural
networks reveal which input words have the strongest influence on our model's
predictions. In addition, we evaluate on a dataset of product reviews, which
exhibit similar properties as user comments, such as featuring upvotes for
helpfulness.Comment: Accepted at the International Conference on Web and Social Media
(ICWSM 2020); 11 pages; code and data are available at
https://hpi.de/naumann/projects/repeatability/text-mining.htm