The advent of the real-time web is proving both challeng-

ing and at the same time disruptive for a number of areas of research,

notably information retrieval and web data mining. As an area of research reaching maturity, sentiment analysis oers a promising direction for modelling the text content available in real-time streams. This paper reviews the real-time web as a new area of focus for sentiment analysis

and discusses the motivations and challenges behind such a direction

Bermingham, Adam

Smeaton, Alan F.

English

The advent of the real-time web is proving both challeng-
ing and at the same time disruptive for a number of areas of research,
notably information retrieval and web data mining. As an area of research reaching maturity, sentiment analysis oers a promising direction for modelling the text content available in real-time streams. This paper reviews the real-time web as a new area of focus for sentiment analysis
and discusses the motivations and challenges behind such a direction

Irish Universities

Crowdsourced real-world sensing: sentiment analysis and the real-time web

DCU Online Research Access Service

Crowdsourced Real-world Sensing:Sentiment Analysis and the Real-Time WebAdam Bermingham and Alan F. SmeatonCLARITY: Centre for Sensor Web Technologies,School of Computing,Dublin City University{abermingham, asmeaton}@computing.dcu.ieAbstract. The advent of the real-time web is proving both challeng-ing and at the same time disruptive for a number of areas of research,notably information retrieval and web data mining. As an area of re-search reaching maturity, sentiment analysis offers a promising directionfor modelling the text content available in real-time streams. This paperreviews the real-time web as a new area of focus for sentiment analysisand discusses the motivations and challenges behind such a direction.Keywords: sentiment analysis, real-time web, microblog1 IntroductionIn the last 10 years, user-generated content has come to dominate a large por-tion of the web. Reviews, blogs, social networks, discussion forums and wikis areall familiar concepts to the average Internet user. User-generated content hasnow earned respect as a credible source for exploring both factual and subjec-tive information. This has inspired research in the area of automatic sentimentanalysis: methods for automatic detection of negative and positive emotions,opinions and other evaluations in text.The real-time web refers to the portion of the web where information isavailable shortly after it is created and where it is connected in some way withevents that are happening in the real world either at, or close to that time. Interms of user-generated content, the information takes the form of blog posts,microblog posts, news feeds and social network content amongst others. Thiscontent is often reactionary in nature, disseminating news of real-world eventsin real-time and expressing associated opinion and commentary. Just as eventsin the real-world can happen at specific times and are scheduled, or can beunpredicted and occur spontaneously, so too does user-generated content havea prominent time component. Examples of scheduled real-world events wouldbe sporting contests and TV programs and spontanous real-word events wouldinclude riots and civil disturbances.The microblogging service, Twitter, is a good example of information makingup the real-time web. Twitter allows users to publish short text messages andthese messages then appear in their followers’ feeds and may appear in searches.Twitter users write about a wide variety of topics including both scheduledand spontaneous real-world and real-time events. The diversity of content, thelarge volume and the availability of data mean that Twitter provides us with aunique opportunity to mine sentiment in real-time in a way not possible before.Throughout this paper we use Twitter as a case study for sentiment in thereal-time web.The task of applying sentiment analysis to the real-time web however has anumber of challenges. Due to the dynamic nature of the real-time web, topicsof interest are constantly evolving. There are also infrastructural challenges asstatic indexes and latency in computation are no longer acceptable. This alsoposes difficulties for experimental methodology as methods for evaluating tech-niques for real-time knowledge discovery are not well established.This paper offers a review of sentiment analysis in the real-time web as afuture direction for research. The rest of this paper is laid out as follows: inSection 2, background to sentiment analysis is presented followed by a motivationfor pursuing this line of research in Section 3. Challenges are discussed in Sections4 and we conclude in Section 5.PUBLICPRIVATEARCHIVAL REAL-TIMEBooksInstant MessagingBreaking News BlogsEditorial News MicroblogsEmailSocial NetworksResearch PapersREAL-TIME WEBDiscussion ForumsSMSReviewsCompany DocumentsPersonal ArchivesFig. 1. A conceptualization of the the real-time web in terms of digital content.2 BackgroundMuch sentiment analysis research concerns the study of static data. A number ofpublicly available collections have been widely used in research such as Pang andLee’s movie review corpus [6], Wilson at al.’s MPQA corpus [10], Blogs06 fromthe TREC Blog Track [4] and the data provided as part of the NTCIR opinionfinding task [8] to name a few of the more common examples. These corporamay all be thought of as static corpora: all of the topics are predefined and thelabels are independent of any temporal aspect. This has many advantages, thereplicability of experiments being one. However, in the real-time web, documents,topics and sentiment all have an inherent temporal component.Recently there have been research works based on Twitter. Bollen et al. [1]modelled trends in mood on Twitter in the 6 dimensions defined by the psy-chometric test, the Profile of Mood States (POMS): tension, depression, fatigue,vigour, anger and confusion. They developed a term-based emotional rating sys-tem by extending the 65 adjectives defined in POMS to a lexicon of 793 terms.They then use the variance of these terms in Twitter posts over a series of 153days to model the trends in mood along the 6 dimensions. They found thatsentiment on Twitter correlated with real-world values such as stock prices andcoincides with events. They conclude:“events in the social, political, cultural and economical sphere dohave a significant, immediate and highly specific effect on the variousdimensions of public mood.”Similarly, Diakapolous and Shamma analysed sentiment in Twitter poststo characterise the reaction to various issues in a US Presidential election de-bate in 2008 [2]. Rather than use an automated sentiment analysis approach,they crowdsourced the annotations using Amazon Mechanical Turk (AMT). TheAMT annotators were asked to annotate documents which discuss the debate aspositive, negative, mixed or other towards each of the presidentatial candidates.Diakopolous notes significant shift in sentiment as the debate moves betweenspeakers and topics. The also suggest that by correlating the positive and neg-ative sentiment they can identify controversial issues, though they note thatthis requires further investigation. They conclude by offering two caveats sur-rounding modelling sentiment in real-time streams: (i) the relationship betweensentiment and a real-world occurence is inferred based on document timestampand relevant terms and is not necessarily accurate in all cases and (ii) the au-thors of the documents in the stream are not necessarily representative of thewider population.These works support the assumption that the sentiment in the documentstream is indicative of people’s reaction to real-world events. This is encouragingas it demonstrates that real-time social content, such as Twitter, is a valid datasource for gauging public sentiment.Another promising avenue of research is in market research. One such workis Jansen et al. who studied the Word-Of-Mouth effect on Twitter, focusing onhow and why positive and negative sentiment towards brands spreads on Twitter[3]. They use supervised learning to classify twitter posts which mention brandsfor sentiment towards that brand using n-gram features. Interestingly, they findautomated sentiment analysis accuracy comparable to manual classification. Ofthe documents they analyse, 19% contain a reference to a product, company orservice and of these, 20% contained sentiment towards that product, companyor service. They also observe large temporal swings in sentiment and suggestthat marketing companies are required to continually monitor their streams ofdocuments over time. Again the results demonstrate that Twitter data providesa means to sense the collective sentiment towards topics of interest.These exploratory works hint at the potential of researching automatic tech-niques to model the sentiment in the real-time web. There remains much workto be done to explore fully the possibilities presented by the real-time web.3 Motivation and ApplicationsIn applying sentiment analysis to the real-time web, and in specific user-generatedcontent, we are in essence crowd-sourcing our sensing of the real world in real-time. The online conversation becomes a sea of data from which we can infersentiment and extrapolate information about the real-world around us. This isnot something that has been possible until now in any meaningful way and sowe are presented with a unique avenue for research.For some time there have been methods of near-instantaneous computer-mediated communication. Instant messaging (IM) and text messaging on mobilephones (SMS) are two such examples. Each of these types of communicationhowever are intrinsically private and obtaining and publishing datasets based onthe private correspondence of users is problematic at best. The public nature ofthe Internet means that no such privacy restriction exists in terms of mining theinformation in online content, real-time or otherwise. The standardised way inwhich this content is made available not only encourages developers and users tobetter use the content, but also us as researchers to efficiently construct datasetsand data streams to be used for study.The recent growth in the volume in the real-time web, specifically on Twitter,is staggering. At least one website has recently measured the rate of Twitter postsbeing published is 2 billion per month, or 64 million per day, and increasing1.This is undoubtedly a large volume of information to analyse, even given theshort length of twitter posts. But what portion of this deluge are relevant toa given topic interest? In the recent Soccer World Cup in South Africa, eventhe early matches saw activity in the region of hundreds of thousands of tweetsper match. Similar activity was seen during the NBA play-offs. High levels ofactivity are also seen in relation to unfolding news stories and live television.Thus the need for automated analytical and aggregation techniques is clear.Search on Twitter2 is dominated by inverse chronologically ordered results, fil-1 http://royal.pingdom.com/2010/06/08/twitter-now-2-billion-tweets-per-month/2 http://search.twitter.comtered by keyword. In this model, the assumption is that recency is the singlemost important measure of relevance. With many relevant documents beingproduced, there will be many more before a user has time to finish reading thesearch results. This simple model does not scale well. The problem of searchin the real-time web is still an unsolved problem. Perhaps real-time streams ofuser generated content are destined to be passively observed rather than ac-tively searched. The problem definition and methodologies are still in flux. Byenriching the documents with sentiment information, the opportunity is thereto employ more sophisticated methods to help users find useful information. Forexample, ensuring a level of diversity and representativeness of sentiment in theresults list.As well as helping users find documents of interest, there is also value in beingable to determine the aggregate sentiment in a real-time stream of documents.Being able to quantify sentiment for a given topic over time permits us to usesentiment as we would a stream of any other source of data: stock prices, sensordata. A real-time sentiment trend then allows us to find events that triggerdeviations in sentiment and to integrate this with other data feeds both from theonline and oﬄine world. This type of aggregation and high performance analyticsis of obvious benefit to many areas of industry, government and research.Sentiment analysis is an area of research reaching maturity (see [7] for a de-tailed history of the field). There are now established methodologies, in particularfor machine learning techniques, for obtaining accuracies comparable with thetraditionally easier task of topical classification. It is the intersection between (i)the abundance and availability of data, (ii) the maturity and of sentiment anal-ysis as an area of research and (iii) the dearth of research into sentiment-basedstrategies for real-time information analysis that motivate this area of research.4 ChallengesThe primary challenge in the real-time web is understanding the informationneeds and interaction patterns. We have already seen how real-time servicessuch as Twitter are both a disruptive and a challenging and opportunistic tech-nology. As of now it is unclear what are the common interaction patterns andperhaps more importantly, which ones will prove to be beneficial as the technol-ogy matures to a significant degree of productivity. Again taking Twitter as anexample, perhaps search will prove to be the most valuable way of interactingwith Twitter, as it has been for the traditional web. On the other hand, perhapsthe real value is being able to navigate the people you personally follow, a moresocial network oriented perspective. Perhaps focus will shift from journal stylecontent to more topical content, as has arguably happened in the blogosphere.Perhaps the real-time web in the context of an event behaves quite differently tothat at another arbitrary time. Or most likely, perhaps the overall picture is onewhich is more complex and which warrants careful thought and consideration.From a sentiment analysis point of view, the real-time web means pushingsentiment analysis beyond review classification. Review classification serves as agood constrained experiment to evaluate sentiment analysis techniques but is lessrelevant in a real-world ad-hoc domain. In review classification often the topicsare homogenous and there is little or no topic drift in the documents. Twitter byit’s nature is dynamic and unpredictable, even the sentiment topics of interestmay themselves may only become apparent as a real-world event unfolds and notbe conceived beforehand. These types of ad-hoc scenarios can be troublesome.Add to this the variability in topic nature and the temporal dependency oftraining data used and there are a lot of challenges in approaching the accuracyenjoyed in classifying reviews.Computational efficiency is also a consideration. Some of the higher perform-ing sentiment analysis systems (for example [5]) have relied on computationallyexpensive feature extraction techniques such as parsing and dependency extrac-tion to achieve their results. Without extensive computing resources, this wouldlikely be unfeasible for the forseeable future with a required throughput of manydocuments per second. These problems can be mitigated by sampling strategiesor, for aggregate sentiment, by allowing for a latency or less granular sentimentreading.As an informal communication platform, Twitter exhibits characteristics ofnoisy text. Twitter posts often contain spelling errors, grammatical contrac-tions, non-standard punctuation, emoticons etc. Tagliamonte analysed Englishlanguage use in Instant Messenger (IM) by teenagers and adolescents and foundthat although the text exhibited features of noisy text, these patterns were notprevalent in his older participants and that this type of text was not as com-mon as construed in the media [9]. In any case, with sufficient training datan-gram models (or extensions thereof) should robustly handle arbitrary tokens.This could degrade performance of approaches who rely on parsing to extractfeatures from the text. Such approaches may benefit from a step of languagenormalisation where the text is amended, either heuristically or using a machinetranslation approach, to a more standard form. However, non-standard languageusage may even prove to be beneficial to learning approaches. Often an authormay do this to express themselves in a more concise way where they may beconstrained by length of by the modality they are using for input. For example,the following punctuation sequences all add tone to the content of a document:“...”, “:-)”, “?!”, “!!!!”.Evaluating methods for sentiment analysis in real-time also poses a signifi-cant challenge for research methodologies. It is likely that a true measure of theeffectiveness of improving information discovery using sentiment analysis is notpossible to determine outside of real-time. In real-time, perhaps our only optionis the expensive task of evaluating users’ interactions with systems by inferringeffectiveness from their use of the system. Another option is to prompt users forreal-time system feedback. This type of evaluation can borrow from the field ofuser interface evaluation where contrasting methods of interaction are evaluated.In either case, this is not as scalable or as reproducible as some common eval-uation methodologies such as multiple fold cross validation in machine learningor the Cranfield model for evaluating information retrieval systems.These challenges collectively are not insurmountable and form a number ofinteresting research questions for the field of sentiment analysis to pursue.5 ConclusionThe real-time web has disrupted and challenged our methods for managing in-formation retrieval and knowledge discovery on the web. Our research method-ologies for analysing static corpora, independent of time, need to be rethoughtand we need to adapt our infrastructures for dealing with such time dependentinformation. At its core this is an exciting time for user-generated content. In ashort space of time, the Internet has a become a place where information andthoughts about a vast array of topics can be gathered nearly instantaneously.Without a well-designed method for organising this information and allowingusers to access and leverage it, the data becomes redundant, and we are not ableto take advantage of the opportunities presented by the advent of the real-timeweb.AcknowledgmentsThis work is supported by Science Foundation Ireland under grant 07/CE/I1147References1. J. Bollen, A. Pepe, and H. Mao. Modeling public mood and emotion: Twittersentiment and socio-economic phenomena. CoRR, abs/0911.1583, 2009.2. N. A. Diakopoulos and D. A. Shamma. Characterizing debate performance viaaggregated Twitter sentiment. In Conference on Human Factors in ComputingSystems (CHI 2010), 2010.3. B. Jansen, M. Zhang, K. Sobel, and A. Chowdury. Twitter power: Tweets aselectronic word of mouth. Journal of the American Society for Information Scienceand Technology, 2009.4. C. Macdonald and I. Ounis. The TREC Blogs06 collection : Creating and analysinga blog test collection. Technical report, University of Glasgow, Department ofComputing Science, 2006.5. S. Matsumoto, H. Takamura, and M. Okumura. Sentiment classification using wordsub-sequences and dependency sub-trees. In Proceedings of PAKDD’05, the 9thPacific-Asia Conference on Advances in Knowledge Discovery and Data Mining,2005.6. B. Pang and L. Lee. A sentimental education: sentiment analysis using subjectivitysummarization based on minimum cuts. In ACL ’04: Proceedings of the 42nd An-nual Meeting on Association for Computational Linguistics, page 271, Morristown,NJ, USA, 2004. Association for Computational Linguistics.7. B. Pang and L. Lee. Opinion mining and sentiment analysis. Foundation andTrends in Information Retrieval, 2(1-2):1–135, 2008.8. Y. Seki, D. K. Evans, L. Ku, L. Sun, H. Chen, and N. Kando. Overview of multi-lingual opinion analysis task at NTCIR-7. 2008.9. S. A. Tagliamonte and D. Denis. LINGUISTIC RUIN? LOL! INSTANT MESSAG-ING AND TEEN LANGUAGE. American Speech, 83(1):3–34, 2008.10. T. Wilson, J. Wiebe, and P. Hoffmann. Recognizing contextual polarity in phrase-level sentiment analysis. Proceedings of the 2005 Conference on Empirical Methodsin Natural Language Processing (EMNLP), pages 347–354, 2005.

A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts.

Characterizing debate performance via aggregated Twitter sentiment.

Modeling public mood and emotion: Twitter sentiment and socio-economic phenomena.

Opinion mining and sentiment analysis. Foundation and Trends in Information Retrieval,

Overview of multilingual opinion analysis task at NTCIR-7.

Recognizing contextual polarity in phraselevel sentiment analysis.

Sentiment classi using word sub-sequences and dependency sub-trees.

The TREC Blogs06 collection : Creating and analysing a blog test collection.

Twitter power: Tweets as electronic word of mouth.

The advent of the real-time web is proving both challeng-\ud
ing and at the same time disruptive for a number of areas of research,\ud
notably information retrieval and web data mining. As an area of research reaching maturity, sentiment analysis oers a promising direction for modelling the text content available in real-time streams. This paper reviews the real-time web as a new area of focus for sentiment analysis\ud
and discusses the motivations and challenges behind such a direction

Name not available

Crowdsourced real-world sensing: sentiment analysis and the real-time web

Abstract

Similar works

Full text

Available Versions

Irish Universities

DCU Online Research Access Service

Name not available