A Novel Human Computation Game for Critique Aggregation

We present a human computation game based on the popular board game - Dixit. We ask the players not only for annotations, but for a direct critique of the result of an automated system.We present the results of the initial run of the game, in which the answers of 15 players were used to profile the mistakes of an aspect-based opinion mining system. We show that the gameplay allowed us to identify the major faults of the extracted opinions. The players' actions thus helped improve the opinion extraction algorithm

Constructing Context-Aware Sentiment Lexicons with an Asynchronous Game with a Purpose

One of the reasons sentiment lexicons do not reach human-level performance is that they lack the contexts that define the polarities of words. While obtaining this knowledge through machine learning would require huge amounts of data, context is commonsense knowledge for people, so human computation is a better choice. We identify context using a game with a purpose that increases the workers' engagement in this complex task. With the contextual knowledge we obtain from only a small set of answers, we already halve the sentiment lexicons' performance gap relative to human performance