Data Scarcity in Event Analysis and Abusive Language Detection

Lack of data is almost always the cause of the suboptimal performance of neural networks. Even though data scarce scenarios can be simulated for any task by assuming limited access to training data, we study two problem areas where data scarcity is a practical challenge: event analysis and abusive content detection} Journalists, social scientists and political scientists need to retrieve and analyze event mentions in unstructured text to compute useful statistical information to understand society. We claim that it is hard to specify information need about events using keyword-based representation and propose a Query by Example (QBE) setting for event retrieval. In the QBE setting, we assume that there are a few example sentences mentioning the event class a user is interested in and we aim to retrieve relevant events using only the examples as a query. Traditional event detection approaches are not applicable in this setting as event detection datasets are constructed based on pre-defined schemas which limits them to a small set of event and event-argument types. Moreover, the amount of annotated data in event detection datasets is limited that only allows us to build a retrieval corpus for evaluation. Thus we assume that there are no relevance judgments to train an event retrieval model -- except for the few examples of a specific event type. We create three QBE evaluation settings from three event detection datasets: PoliceKilling, ACE, and IndiaPoliceEvents. For the PoliceKilling dataset, where a relevant sentence describes a police killing event, we show that a query model constructed from the NLP features extracted from the few given examples is effective compared to event detection baselines. For the ACE dataset, where there are thirty-three types of events, we construct a QBE setting for each type and show that a sentence embedding approach effectively transfers for event matching. Finally, we conducted a unified evaluation of all three datasets using the sentence-embedding-based model and showed that it outperforms strong baselines. We further examine the effect of data scarcity in abusive language detection. We first study a specific type of abusive language -- hate speech. Neural hate speech detection models trained from one dataset poorly generalize to another dataset from a different domain. This is because characteristics of hate speech vary based on racial and cultural aspects. Our data scarcity scenario assumes that we have a hate speech dataset from a domain and it needs to generalize to a test set from another domain using the unlabeled data from the test domain only. Thus we assume zero target domain data in this scenario. To tackle the data scarcity, we propose an unsupervised domain adaptation approach to augment labeled data for hate speech detection. We evaluate the approach with three different models (character CNNs, BiLSTMs, and BERT) on three different collections. We show our approach improves Area under the Precision/Recall curve by as much as 42% and recall by as much as 278%, with no loss (and in some cases a significant gain) in precision. Finally, we examine the cross-lingual abusive language detection problem. Abusive language is a superclass of hate speech that includes profanity, aggression, offensiveness, cyberbullying, toxicity, and hate speech itself. There is a large collection of abusive language detection datasets in English such as Jigsaw. For other languages there exist datasets for abusive language detection but with very limited data. We propose a cross-lingual transfer learning approach to learn an effective neural abusive language classifier for such low-resource languages with help from a dataset from a resource-rich language. The framework is based on a nearest-neighbor architecture and is thus interpretable by design. It is a modern instantiation of the classic k-nearest neighbor model, as we use transformer representations in all its components. Unlike prior work on neighborhood-based approaches, we encode the neighborhood information based on query-neighbor interactions. We propose two encoding schemes and show their effectiveness using both qualitative and quantitative analyses. Our evaluation results on eight languages from two different datasets for abusive language detection show sizable improvements in F1 over strong baselines

A Systematic Literature Review on Cyberbullying in Social Media: Taxonomy, Detection Approaches, Datasets, And Future Research Directions

In the area of Natural Language Processing, sentiment analysis, also called opinion mining, aims to extract human thoughts, beliefs, and perceptions from unstructured texts. In the light of social media's rapid growth and the influx of individual comments, reviews and feedback, it has evolved as an attractive, challenging research area. It is one of the most common problems in social media to find toxic textual content.  Anonymity and concealment of identity are common on the Internet for people coming from a wide range of diversity of cultures and beliefs. Having freedom of speech, anonymity, and inadequate social media regulations make cyber toxic environment and cyberbullying significant issues, which require a system of automatic detection and prevention. As far as this is concerned, diverse research is taking place based on different approaches and languages, but a comprehensive analysis to examine them from all angles is lacking. This systematic literature review is therefore conducted with the aim of surveying the research and studies done to date on classification of  cyberbullying based in textual modality by the research community. It states the definition, , taxonomy, properties, outcome of cyberbullying, roles in cyberbullying  along with other forms of bullying and different offensive behavior in social media. This article also shows the latest popular benchmark datasets on cyberbullying, along with their number of classes (Binary/Multiple), reviewing the state-of-the-art methods to detect cyberbullying and abusive content on social media and discuss the factors that drive offenders to indulge in offensive activity, preventive actions to avoid online toxicity, and various cyber laws in different countries. Finally, we identify and discuss the challenges, solutions, additionally future research directions that serve as a reference to overcome cyberbullying in social media

SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)

We present the results and main findings of SemEval-2020 Task 12 on Multilingual Offensive Language Identification in Social Media (OffensEval 2020). The task involves three subtasks corresponding to the hierarchical taxonomy of the OLID schema (Zampieri et al., 2019a) from OffensEval 2019. The task featured five languages: English, Arabic, Danish, Greek, and Turkish for Subtask A. In addition, English also featured Subtasks B and C. OffensEval 2020 was one of the most popular tasks at SemEval-2020 attracting a large number of participants across all subtasks and also across all languages. A total of 528 teams signed up to participate in the task, 145 teams submitted systems during the evaluation period, and 70 submitted system description papers.Comment: Proceedings of the International Workshop on Semantic Evaluation (SemEval-2020

Modelos de aprendizaje supervisado como apoyo a la toma de decisiones en las organizaciones basados en datos de redes sociales: Una revisión sistemática de la literatura.

Las redes sociales se han convertido en la herramienta de comunicación e interacción más utilizada entre las personas y se han diversificado para cumplir funciones importantes dentro de la organización. En consecuencia, las redes sociales se han vuelto una fuente inmensa de datos que son procesados a través de modelos de aprendizaje supervisado para producir información que sea competente para la toma de decisiones como la predicción de campañas electorales, la predicción de consumo de un producto y/o servicio, la reputación de una empresa entre otros. De manera que el presente estudio tiene como objetivo identificar los modelos de aprendizaje supervisado como apoyo a la toma de decisiones en las organizaciones basados en datos de redes sociales. Para la identificación de modelos de aprendizaje supervisado se realizó una revisión sistemática de la literatura(RSL) en bases de datos reconocidas y revistas indexadas. De un total de 1614 artículos se identificaron 32 artículos que hacen referencia a 6 modelos de aprendizaje supervisado y las funciones que cumplen como apoyo a la toma de decisiones en una organización. Se puede concluir que existen diversos modelos de aprendizaje supervisado siendo el de Support Vector Machine de mayor grado de precisión. También se han encontrado en las investigaciones modelos de: Naive Bayes, Decision Tree, Regression: Logistic y lineal, k-Nearest Neighbors, y finalmente Neural Network.Trabajo de investigaciónLIMAEscuela Profesional de Ingeniería de SistemasTecnología de información e innovación tecnológic

Detecting offensive speech in conversational code-mixed dialogue on social media: A contextual dataset and benchmark experiments

The spread of Hate Speech on online platforms is a severe issue for societies and requires the identification of offensive content by platforms. Research has modeled Hate Speech recognition as a text classification problem that predicts the class of a message based on the text of the message only. However, context plays a huge role in communication. In particular, for short messages, the text of the preceding tweets can completely change the interpretation of a message within a discourse. This work extends previous efforts to classify Hate Speech by considering the current and previous tweets jointly. In particular, we introduce a clearly defined way of extracting context. We present the development of the first dataset for conversational-based Hate Speech classification with an approach for collecting context from long conversations for code-mixed Hindi (ICHCL dataset). Overall, our benchmark experiments show that the inclusion of context can improve classification performance over a baseline. Furthermore, we develop a novel processing pipeline for processing the context. The best-performing pipeline uses a fine-tuned SentBERT paired with an LSTM as a classifier. This pipeline achieves a macro F1 score of 0.892 on the ICHCL test dataset. Another KNN, SentBERT, and ABC weighting-based pipeline yields an F1 Macro of 0.807, which gives the best results among traditional classifiers. So even a KNN model gives better results with an optimized BERT than a vanilla BERT model

Mapping (Dis-)Information Flow about the MH17 Plane Crash

Digital media enables not only fast sharing of information, but also disinformation. One prominent case of an event leading to circulation of disinformation on social media is the MH17 plane crash. Studies analysing the spread of information about this event on Twitter have focused on small, manually annotated datasets, or used proxys for data annotation. In this work, we examine to what extent text classifiers can be used to label data for subsequent content analysis, in particular we focus on predicting pro-Russian and pro-Ukrainian Twitter content related to the MH17 plane crash. Even though we find that a neural classifier improves over a hashtag based baseline, labeling pro-Russian and pro-Ukrainian content with high precision remains a challenging problem. We provide an error analysis underlining the difficulty of the task and identify factors that might help improve classification in future work. Finally, we show how the classifier can facilitate the annotation task for human annotators

Detection and Prevention of Cyberbullying on Social Media

The Internet and social media have undoubtedly improved our abilities to keep in touch with friends and loved ones. Additionally, it has opened up new avenues for journalism, activism, commerce and entertainment. The unbridled ubiquity of social media is, however, not without negative consequences and one such effect is the increased prevalence of cyberbullying and online abuse. While cyberbullying was previously restricted to electronic mail, online forums and text messages, social media has propelled it across the breadth of the Internet, establishing it as one of the main dangers associated with online interactions. Recent advances in deep learning algorithms have progressed the state of the art in natural language processing considerably, and it is now possible to develop Machine Learning (ML) models with an in-depth understanding of written language and utilise them to detect cyberbullying and online abuse. Despite these advances, there is a conspicuous lack of real-world applications for cyberbullying detection and prevention. Scalability; responsiveness; obsolescence; and acceptability are challenges that researchers must overcome to develop robust cyberbullying detection and prevention systems. This research addressed these challenges by developing a novel mobile-based application system for the detection and prevention of cyberbullying and online abuse. The application mitigates obsolescence by using different ML models in a “plug and play” manner, thus providing a mean to incorporate future classifiers. It uses ground truth provided by the enduser to create a personalised ML model for each user. A new large-scale cyberbullying dataset of over 62K tweets annotated using a taxonomy of different cyberbullying types was created to facilitate the training of the ML models. Additionally, the design incorporated facilities to initiate appropriate actions on behalf of the user when cyberbullying events are detected. To improve the app’s acceptability to the target audience, user-centred design methods were used to discover stakeholders’ requirements and collaboratively design the mobile app with young people. Overall, the research showed that (a) the cyberbullying dataset sufficiently captures different forms of online abuse to allow the detection of cyberbullying and online abuse; (b) the developed cyberbullying prevention application is highly scalable and responsive and can cope with the demands of modern social media platforms (b) the use of user-centred and participatory design approaches improved the app’s acceptability amongst the target audience