A Novel Human Computation Game for Critique Aggregation

We present a human computation game based on the popular board game - Dixit. We ask the players not only for annotations, but for a direct critique of the result of an automated system.We present the results of the initial run of the game, in which the answers of 15 players were used to profile the mistakes of an aspect-based opinion mining system. We show that the gameplay allowed us to identify the major faults of the extracted opinions. The players' actions thus helped improve the opinion extraction algorithm

A Model of Online Social Interactions based on Sentiment Analysis and Content Similarity

In this paper we create a model of human behavior in online communities, based on the network topology and on the communication content. The model contains eleven distinct hypotheses, which validate three intuitions. The rst intuition is that the network topology alone fails to clearly distinguish between the users who contribute to the community and the troublemakers. The second intuition is that the content of the messages exchanged in an online community can separate good and insightful contri- butions from the rest. The third intuition is that there is a delay until the network stabilizes and un- til standard measures, such as betweenness central- ity, can be used accurately. Taken together, these three intuitions are a solid case against indiscrimi- nately using network measures. They also underline the importance of the communication content. We show that the sentiment within the messages, espe- cially antagonism, can signicantly alter the commu- nity perception. We create a novel sentiment analysis technique to identify antagonistic behavior. We use real world data, taken from the Slashdot discussion forum to validate our model. All the find- ings are accompanied by extremely signicant t-test p-values

Direct Negative Opinions in Online Discussions

In this paper we investigate the impact of antagonism in online discussions. We define antagonism as a new class of textual opinions - direct sentiment towards the authors of previous comments. We detect the negative sentiment using aspect-based opinion mining techniques. We create a model of human behavior in online communities, based on the network topology and on the communication content. The model contains seven hypotheses, which validate two intuitions. The first intuition is that the content of the messages exchanged in an online community can separate good and insightful contributions from the rest. The second intuition is that there is a delay until the network stabilizes and until standard measures, such as betweenness centrality, can be used accurately. Taken together, these intuitions are a solid case for using the content of the communication along with network measures. We show that the sentiment within the messages, especially antagonism, can significantly alter the community perception. We use real world data, taken from the Slashdot1 discussion forum to validate our model. All the findings are accompanied by extremely significant t-test p-values

Fine-Grained Emotion Recognition in Olympic Tweets Based on Human Computation

In this paper, we detail a method for domain specific, multi-category emotion recognition, based on human computation. We create an Amazon Mechanical Turk1 task that elicits emotion labels and phrase-emotion associations from the participants. Using the proposed method, we create an emotion lexicon, compatible with the 20 emotion categories of the Geneva Emotion Wheel. GEW is the first computational resource that can be used to assign emotion labels with such a high level of granularity. Our emotion annotation method also produced a corpus of emotion labeled sports tweets. We compared the crossvalidated version of the lexicon with existing resources for both the positive/negative and multi-emotion classification problems. We show that the presented domain-targeted lexicon outperforms the existing general purpose ones in both settings. The performance gains are most pronounced for the fine-grained emotion classification, where we achieve an accuracy twice higher than the benchmark.

A :) Is Worth a Thousand Words: How People Attach Sentiment to Emoticons and Words in Tweets

Emoticons are widely used to express positive or negative sentiment on Twitter. We report on a study with live users to determine whether emoticons are used to merely emphasize the sentiment of tweets, or whether they are the main elements carrying the sentiment. We found that the sentiment of an emoticon is in substantial agreement with the sentiment of the entire tweet. Thus, emoticons are useful as predictors of tweet sentiment and should not be ignored in sentiment classification. However, the sentiment expressed by an emoticon agrees with the sentiment of the accompanying text only slightly better than random. Thus, using the text accompanying emoticons to train sentiment models is not likely to produce the best results, a fact that we show by comparing lexicons generated using emoticons with others generated using simple textual features. © 2013 IEEE

Simple Unsupervised Keyphrase Extraction using Sentence Embeddings

Keyphrase extraction is the task of automatically selecting a small set of phrases that best describe a given free text document. Keyphrases can be used for indexing, searching, aggregating and summarizing text documents, serving many automatic as well as human-facing use cases. Existing supervised systems for keyphrase extraction require large amounts of labeled training data and generalize very poorly outside the domain of the training data. At the same time, unsupervised systems found in the literature have poor accuracy, and often do not generalize well, as they require the input document to belong to a larger corpus also given as input. Furthermore, both supervised and unsupervised methods are often too slow for real-time scenarios and suffer from over-generation. Addressing these drawbacks, in this paper, we introduce an unsupervised method for keyphrase extraction from single documents that leverages sentence embeddings. By selecting phrases whose semantic embeddings are close to the embeddings of the whole document, we are able to separate the best candidate phrases from the rest. We show that our embedding-based method is not only simpler, but also more effective than graph-based state of the art systems, achieving higher F-scores on standard datasets. Simplicity is a significant advantage, especially when processing large amounts of documents from the Web, resulting in considerable speed gains. Moreover, we describe how to increase coverage and diversity among the selected keyphrases by introducing an embedding-based maximal marginal relevance (MMR) for new phrases. A user study including over 200 votes showed that, although reducing the phrase semantic overlap leads to no gains in terms of F-score, our diversity enriched selection is preferred by humans

Personalizing Product Rankings Using Collaborative Filtering on Opinion-Derived Topic Profiles

Product review sites such as TripAdvisor, Yelp or Amazon provide a single, non personalized ranking of products. The sparse review data makes personalizing recommendations difficult. Topic Profile Collaborative Filtering exploits review texts to identify user profiles as a basis for similarity. We show that careful use of the available data and separating users into classes can greatly improve the performance of such techniques. We significantly improve MAE, RMSE, and Kendall tau, compared to the previous best results. In addition, we show that personalization does not benefit all the users to the same extent. We propose switching between a personalized and a non personalized method based on the user opinion profile. We show that the user’s opinionatedness is a good indicator of whether the personalization will work or not

Semi-Supervised Method for Multi-Category Emotion Recognition in Tweets

Each tweet is limited to 140 characters. This constraint surprisingly makes Twitter a more spontaneous platform to express our emotions. Detecting emotions and correctly classifying them automatically is an increasingly important task if we want to understand how large groups of people feel about an event or relevant topic. However, constructing supervised classifiers can be a daunting task because of the high manual annotation costs. We propose constructing emotion classifiers with a minimal amount of initial knowledge (e.g. a generalpurpose emotion lexicon) and using a semi-supervised learning method to extend it to correctly detect more emotional tweets within a specific domain. Additionally, we show that our algorithm, Balanced Weighted Voting (or BWV) is able to overcome the imbalanced distribution of emotions in the initial labeled data. Our validation experiments show that BWV improves the performance of three initial classifiers, at least in the specific domain of sports. Furthermore, its comparison with other two learning strategies reveals its superiority in terms of macro F1-score, as well as more stable performance among different emotion categories