Comparing the utility of different classification schemes for emotive language analysis

In this paper we investigated the utility of different classification schemes for emotive language analysis with the aim of providing experimental justification for the choice of scheme for classifying emotions in free text. We compared six schemes: (1) Ekman's six basic emotions, (2) Plutchik's wheel of emotion, (3) Watson and Tellegen's Circumplex theory of affect, (4) the Emotion Annotation Representation Language (EARL), (5) WordNet–Affect, and (6) free text. To measure their utility, we investigated their ease of use by human annotators as well as the performance of supervised machine learning. We assembled a corpus of 500 emotionally charged text documents. The corpus was annotated manually using an online crowdsourcing platform with five independent annotators per document. Assuming that classification schemes with a better balance between completeness and complexity are easier to interpret and use, we expect such schemes to be associated with higher inter–annotator agreement. We used Krippendorff's alpha coefficient to measure inter–annotator agreement according to which the six classification schemes were ranked as follows: (1) six basic emotions (a = 0.483), (2) wheel of emotion (a = 0.410), (3) Circumplex (a = 0.312), EARL (a = 0.286), (5) free text (a = 0.205), and (6) WordNet–Affect (a = 0.202). However, correspondence analysis of annotations across the schemes highlighted that basic emotions are oversimplified representations of complex phenomena and as such likely to lead to invalid interpretations, which are not necessarily reflected by high inter-annotator agreement. To complement the result of the quantitative analysis, we used semi–structured interviews to gain a qualitative insight into how annotators interacted with and interpreted the chosen schemes. The size of the classification scheme was highlighted as a significant factor affecting annotation. In particular, the scheme of six basic emotions was perceived as having insufficient coverage of the emotion space forcing annotators to often resort to inferior alternatives, e.g. using happiness as a surrogate for love. On the opposite end of the spectrum, large schemes such as WordNet–Affect were linked to choice fatigue, which incurred significant cognitive effort in choosing the best annotation. In the second part of the study, we used the annotated corpus to create six training datasets, one for each scheme. The training data were used in cross–validation experiments to evaluate classification performance in relation to different schemes. According to the F-measure, the classification schemes were ranked as follows: (1) six basic emotions (F = 0.410), (2) Circumplex (F = 0.341), (3) wheel of emotion (F = 0.293), (4) EARL (F = 0.254), (5) free text (F = 0.159) and (6) WordNet–Affect (F = 0.158). Not surprisingly, the smallest scheme was ranked the highest in both criteria. Therefore, out of the six schemes studied here, six basic emotions are best suited for emotive language analysis. However, both quantitative and qualitative analysis highlighted its major shortcoming – oversimplification of positive emotions, which are all conflated into happiness. Further investigation is needed into ways of better balancing positive and negative emotions. Keywords: annotation, crowdsourcing, text classification, sentiment analysis, supervised machine learnin

Academic desertion and emotions in e-learning environments

En los espacios académicos de nivel superior, existe preocupación en relación a la deserción y abandono de los estudiantes. Por otra parte, existen investigaciones que asocian esta situación académica con el estado afectivo de los estudiantes. En este sentido, es importante analizar y comprender el estado anímico de los estudiantes universitarios, tratando de detectar qué emociones se manifiestan durante el cursado. Las emociones pueden expresarse y detectarse a través de diversos medios, tales como la expresión oral, expresiones faciales, gestos y expresión escrita. En este trabajo, se realizó un recorrido por algunas plataformas educativas consideradas afectivas, analizando los métodos y recursos que se utilizan en cada una de ellas, para la detección de emociones. Adicionalmente, se consultó a 20 docentes de nivel superior participantes de un curso de posgrado vinculado a la temática, acerca de su percepción respecto a las emociones que influyen en la deserción y abandono en los cursos que siguen la modalidad e-learning. En los resultados se evidenció que la emoción frustración, es considerada como aquella que influye en forma más negativa en el aprendizaje, y puede ser causante de abandono o deserción de los estudiantes, entre otros motivos.In university education, there is concern in relation to the dropout and abandonment of students. On the other hand, there is research that associates this academic situation with the affective state of students. In this sense, it is important to analyze and understand the state of mind of university students, trying to detect what emotions are manifested during the course. Emotions can be expressed and detected through various means, such as oral expression, facial expressions, gestures, and written expression. A tour was made of some educational platforms considered affective, analyzing the methods and resources used in each of them, for the detection of emotions. In addition, 20 higher-level teachers, who participated in a postgraduate course linked to the subject, were consulted about their perception of the emotions that influence dropout in courses that follow the e-learning modality. The results showed that emotional frustration is considered the one that most negatively influences learning, being the cause, among other reasons, of abandonment or desertion of students.Facultad de Informátic

A multi-disciplinary co-design approach to social media sensemaking with text mining

This thesis presents the development of a bespoke social media analytics platform called Sentinel using an event driven co-design approach. The performance and outputs of this system, along with its integration into the routine research methodology of its users, were used to evaluate how the application of an event driven co-design approach to system design improves the degree to which Social Web data can be converted into actionable intelligence, with respect to robustness, agility, and usability. The thesis includes a systematic review into the state-of-the-art technology that can support real-time text analysis of social media data, used to position the text analysis elements of the Sentinel Pipeline. This is followed by research chapters that focus on combinations of robustness, agility, and usability as themes, covering the iterative developments of the system through the event driven co-design lifecycle. Robustness and agility are covered during initial infrastructure design and early prototyping of bottom-up and top-down semantic enrichment. Robustness and usability are then considered during the development of the Semantic Search component of the Sentinel Platform, which exploits the semantic enrichment developed in the prototype, alpha, and beta systems. Finally, agility and usability are used whilst building upon the Semantic Search functionality to produce a data download functionality for rapidly collecting corpora for further qualitative research. These iterations are evaluated using a number of case studies that were undertaken in conjunction with a wider research programme, within the field of crime and security, that the Sentinel platform was designed to support. The findings from these case studies are used in the co-design process to inform how developments should evolve. As part of this research programme the Sentinel platform has supported the production of a number of research papers authored by stakeholders, highlighting the impact the system has had in the field of crime and security researc