1 research outputs found
Monitoring stance towards vaccination in Twitter messages
We developed a system to automatically classify stance towards vaccination in
Twitter messages, with a focus on messages with a negative stance. Such a
system makes it possible to monitor the ongoing stream of messages on social
media, offering actionable insights into public hesitance with respect to
vaccination. For Dutch Twitter messages that mention vaccination-related key
terms, we annotated their stance and feeling in relation to vaccination
(provided that they referred to this topic). Subsequently, we used these coded
data to train and test different machine learning set-ups. With the aim to best
identify messages with a negative stance towards vaccination, we compared
set-ups at an increasing dataset size and decreasing reliability, at an
increasing number of categories to distinguish, and with different
classification algorithms. We found that Support Vector Machines trained on a
combination of strictly and laxly labeled data with a more fine-grained
labeling yielded the best result, at an F1-score of 0.36 and an Area under the
ROC curve of 0.66, outperforming a rule-based sentiment analysis baseline that
yielded an F1-score of 0.25 and an Area under the ROC curve of 0.57. The
outcomes of our study indicate that stance prediction by a computerized system
only is a challenging task. Our analysis of the data and behavior of our system
suggests that an approach is needed in which the use of a larger training
dataset is combined with a setting in which a human-in-the-loop provides the
system with feedback on its predictions.Comment: 16 pages, 2 figures, accepted for BMC Medical Informatics and
Decision Making journa