Automatically detecting open academic review praise and criticism

This is an accepted manuscript of an article published by Emerald in Online Information Review on 15 June 2020. The accepted version of the publication may differ from the final published version, accessible at https://doi.org/10.1108/OIR-11-2019-0347.Purpose: Peer reviewer evaluations of academic papers are known to be variable in content and overall judgements but are important academic publishing safeguards. This article introduces a sentiment analysis program, PeerJudge, to detect praise and criticism in peer evaluations. It is designed to support editorial management decisions and reviewers in the scholarly publishing process and for grant funding decision workflows. The initial version of PeerJudge is tailored for reviews from F1000Research’s open peer review publishing platform. Design/methodology/approach: PeerJudge uses a lexical sentiment analysis approach with a human-coded initial sentiment lexicon and machine learning adjustments and additions. It was built with an F1000Research development corpus and evaluated on a different F1000Research test corpus using reviewer ratings. Findings: PeerJudge can predict F1000Research judgements from negative evaluations in reviewers’ comments more accurately than baseline approaches, although not from positive reviewer comments, which seem to be largely unrelated to reviewer decisions. Within the F1000Research mode of post-publication peer review, the absence of any detected negative comments is a reliable indicator that an article will be ‘approved’, but the presence of moderately negative comments could lead to either an approved or approved with reservations decision. Originality/value: PeerJudge is the first transparent AI approach to peer review sentiment detection. It may be used to identify anomalous reviews with text potentially not matching judgements for individual checks or systematic bias assessments

Active learning in annotating micro-blogs dealing with e-reputation

Elections unleash strong political views on Twitter, but what do people really think about politics? Opinion and trend mining on micro blogs dealing with politics has recently attracted researchers in several fields including Information Retrieval and Machine Learning (ML). Since the performance of ML and Natural Language Processing (NLP) approaches are limited by the amount and quality of data available, one promising alternative for some tasks is the automatic propagation of expert annotations. This paper intends to develop a so-called active learning process for automatically annotating French language tweets that deal with the image (i.e., representation, web reputation) of politicians. Our main focus is on the methodology followed to build an original annotated dataset expressing opinion from two French politicians over time. We therefore review state of the art NLP-based ML algorithms to automatically annotate tweets using a manual initiation step as bootstrap. This paper focuses on key issues about active learning while building a large annotated data set from noise. This will be introduced by human annotators, abundance of data and the label distribution across data and entities. In turn, we show that Twitter characteristics such as the author's name or hashtags can be considered as the bearing point to not only improve automatic systems for Opinion Mining (OM) and Topic Classification but also to reduce noise in human annotations. However, a later thorough analysis shows that reducing noise might induce the loss of crucial information.Comment: Journal of Interdisciplinary Methodologies and Issues in Science - Vol 3 - Contextualisation digitale - 201

A study on text-score disagreement in online reviews

In this paper, we focus on online reviews and employ artificial intelligence tools, taken from the cognitive computing field, to help understanding the relationships between the textual part of the review and the assigned numerical score. We move from the intuitions that 1) a set of textual reviews expressing different sentiments may feature the same score (and vice-versa); and 2) detecting and analyzing the mismatches between the review content and the actual score may benefit both service providers and consumers, by highlighting specific factors of satisfaction (and dissatisfaction) in texts. To prove the intuitions, we adopt sentiment analysis techniques and we concentrate on hotel reviews, to find polarity mismatches therein. In particular, we first train a text classifier with a set of annotated hotel reviews, taken from the Booking website. Then, we analyze a large dataset, with around 160k hotel reviews collected from Tripadvisor, with the aim of detecting a polarity mismatch, indicating if the textual content of the review is in line, or not, with the associated score. Using well established artificial intelligence techniques and analyzing in depth the reviews featuring a mismatch between the text polarity and the score, we find that -on a scale of five stars- those reviews ranked with middle scores include a mixture of positive and negative aspects. The approach proposed here, beside acting as a polarity detector, provides an effective selection of reviews -on an initial very large dataset- that may allow both consumers and providers to focus directly on the review subset featuring a text/score disagreement, which conveniently convey to the user a summary of positive and negative features of the review target.Comment: This is the accepted version of the paper. The final version will be published in the Journal of Cognitive Computation, available at Springer via http://dx.doi.org/10.1007/s12559-017-9496-

The Business Impact of Social Media - Sentiment Analysis Approach -

이 연구의 목적은 소셜 미디어에서 추출된 7개의 감성 도메인이 자동차 시장 점유율 예측에 대한 감성 분석 실험을 위한 데이터로서 적합한 지에 대한 신뢰성을 확인하고 고객들의 의견이 기업의 성과에 어떻게 영향을 미치는 지에 대하여 확인하기 위한 것이다. 본 연구는 총3단계에 걸쳐서 진행되었습니다. 첫 번째 단계는 감성사전 구축의 단계로서 2013년 1월 1일부터 2015년 12월 31일까지 미국 내 26개의 자동차 제조 회사의 고객의 소리 (VOC: Voice of the Customer) 총 45,447개를 자동차 커뮤니티로부터 크롤링 (crawling)하여 POS (Part-of-Speech) 즉 품사정보를 추출하는 태깅 (tagging)과정을 거쳐 부정적, 긍정적 감성의 빈도수를 측정하여 감성사전을 구축하였고, 이에 대한 극성을 측정하여 7개의 감성도메인을 만들었습니다. 두 번째 단계는 데이터에 대한 신뢰성 분석의 단계로서 자기상관관계분석 (Auto-correlation Analysis)과 주성분분석 (PCA: Principal Component Analysis)을 통해 데이터가 실험에 적합한지를 검증하였다. 세 번째 단계에서는 2개의 선형회귀분석 모델로 7개의 감성영역이 미국내 자동차 제조 회사 중 GM, 포드, FCA, 폭스바겐 등 총 4개의 자동차 생산 기업을 선정하여 이들 기업의 성과 즉, 자동차 시장점유율에 어떤 영향을 미치고 있는 지 실험하였다. 그 결과, 우리는 4,815개의 부정적인 어휘들과 2,021개의 긍정적인 감성어휘들을 추출하여 감성사전을 구축하였으며, 구축된 감성사전을 바탕으로, 추출되고 분류된 부정적이고 긍정적인 어휘들을 자동차 산업에 관련된 어휘들과 조합하였고, 자기상관분석과 PCA (주성분 분석)를 통해 감성의 특성을 조사하였다. 실험 결과에 따르면, 자기상관분석에 의해서 감성 데이터에 어떤 일정한 패턴이 존재한다는 것이 발견되었고, 각각의 감성 영역의 감성이 자기상관성이 있으며, 감성의 시계열성 또한 관찰되었다. PCA에 의한 결과로서, 7개 감성영역이 부정성, 긍정성, 중립성을 주성분으로 연결되어 있음을 확인할 수 있었다. 자기상관분석과 PCA를 통한 VOC 감성 데이터에 대한 신뢰성을 바탕으로 2개의 선형회귀분석 모델을 구축하여 실험을 진행하였다. 첫 번째 모델은 주성분 분석에서 부정적 감성의 Sadness, Anger, Fear와 긍정적 감성도메인인 Delight, Satisfaction을 독립변수로 선정하고, 시장점유율을 종속변수로 선정하여 실행하였고 두 번째 모델은 첫 번째 모델에 주성분이 중립성으로 결과가 나온 Shame, Frustration을 독립변수에 추가하여 중립성을 띠고 있는 감성이 시장 점유율에 유의미한 영향을 미치고 있는 지를 확인하였다. 분석 결과, 각 기업 마다 시장점유율에 유의미한 영향을 미치는 감성들이 존재하고 모델 1과, 모델 2에서의 감성 영향력이 차이가 있음을 발견하였다. 본 연구를 통해, 데이터 상에 나타난 정보를 가진 감성이 과거 값에 기초하여 자동차 시장에서 변화를 수반할 수 있다는 것을 나타내고 있음을 확인하였다. 또한, 우리가 시장 데이터의 가용성을 적용하려고 할 때, 자동차 시장 관련 정보나 감성의 자기상관성을 잘 활용할 수 있다면, 감정 분석에 대한 연구에 큰 기여를 할 수 있을 뿐만 아니라, 실제 시장에서의 비지니스 성과에도 다양한 방법으로 기여할 수 있을 것으로 기대된다.List of Tables iv List of Figures v Abstract 1 1. Introduction 1.1 Back Ground 3 1.2 Necessity of Study 6 1.3 Purpose & Questions 8 1.4 Structure 9 2. Literature Reviews of VOC Analysis 2.1 Importance of VOC 11 2.2 Data Mining 15 2.2.1 Concept & Functionalities 15 2.2.2 Methodologies of Data mining 20 2.3 Text Mining 24 2.4 Sentiment Analysis 26 2.5 Research Trend in Korea 30 3. Methodology 3.1 Research Flow 32 3.2 Proposed Methodologies 34 3.2.1 Sentiment Analysis 34 3.2.2 Auto-correlation Analysis 37 3.2.3 Principal Component Analysis (PCA) 38 3.2.4 Linear Regression 40 4. Experiment & Analysis 4.1 Phase I: Constructing Sentiment Lexicon & 7 Sentiment Domains 43 4.1.1 The Subject of Analysis & Crawling Data 43 4.1.2 Extracting POS Information 44 4.1.3 Review Extracting POS Information 46 4.2 Phase II : Reliability Analysis 49 4.2.1 Auto-correlation Analysis of Sentiment 51 4.2.2 Principal Component Analysis of Sentiment 55 4.3 Phase III : Influence on Automotive Market Share 58 4.3.1 Linear Regression Model 58 4.3.2 Definition of Variables 60 4.3.3 The Result of Linear Regression Analysis 62 5. Conclusion 5.1 Summary of Study 73 5.2 Managerial Implication and Limitation 75 5.3 Future Study 77 References 79Docto