Weakly-supervised appraisal analysis

This article is concerned with the computational treatment of Appraisal, a Systemic Functional Linguistic theory of the types of language employed to communicate opinion in English. The theory considers aspects such as Attitude (how writers communicate their point of view), Engagement (how writers align themselves with respect to the opinions of others) and Graduation (how writers amplify or diminish their attitudes and engagements). To analyse text according to the theory we employ a weakly-supervised approach to text classification, which involves comparing the similarity of words with prototypical examples of classes. We evaluate the method's performance using a collection of book reviews annotated according to the Appraisal theory

The good, the bad and the implicit: a comprehensive approach to annotating explicit and implicit sentiment

We present a fine-grained scheme for the annotation of polar sentiment in text, that accounts for explicit sentiment (so-called private states), as well as implicit expressions of sentiment (polar facts). Polar expressions are annotated below sentence level and classified according to their subjectivity status. Additionally, they are linked to one or more targets with a specific polar orientation and intensity. Other components of the annotation scheme include source attribution and the identification and classification of expressions that modify polarity. In previous research, little attention has been given to implicit sentiment, which represents a substantial amount of the polar expressions encountered in our data. An English and Dutch corpus of financial newswire, consisting of over 45,000 words each, was annotated using our scheme. A subset of this corpus was used to conduct an inter-annotator agreement study, which demonstrated that the proposed scheme can be used to reliably annotate explicit and implicit sentiment in real-world textual data, making the created corpora a useful resource for sentiment analysis

Sentiment and behaviour annotation in a corpus of dialogue summaries

This paper proposes a scheme for sentiment annotation. We show how the task can be made tractable by focusing on one of the many aspects of sentiment: sentiment as it is recorded in behaviour reports of people and their interactions. Together with a number of measures for supporting the reliable application of the scheme, this allows us to obtain sufficient to good agreement scores (in terms of Krippendorf's alpha) on three key dimensions: polarity, evaluated party and type of clause. Evaluation of the scheme is carried out through the annotation of an existing corpus of dialogue summaries (in English and Portuguese) by nine annotators. Our contribution to the field is twofold: (i) a reliable multi-dimensional annotation scheme for sentiment in behaviour reports; and (ii) an annotated corpus that was used for testing the reliability of the scheme and which is made available to the research community

Va tuje darvɑzeǃ Di Maria has been a pest all night: Evaluative language in Persian and English live football commentary

This paper explores the expression of evaluative language in live football commentary in Persian and English. The main focus of this study was to explore differences in the use of evaluation in three different modes of football live commentary provided in the UEFA Champions League (UCL) 2014 final match between Real Madrid and Atlético Madrid: live radio commentary (LRC), live TV commentary (LTVC), and live text commentary (LTC). The expressions of evaluative language were analyzed regarding Attitude. Attitude is one of three central components of the appraisal theory (Martin and White, 2005) in language, which is concerned with the use of evaluative language. The study showed that attitudinal resources were prevalent and varied in the extracts analyzed. They were mainly Judgment oriented and negative. The case study was an attempt to contribute to this growing area of research by exploring the live football commentary genre. The mode of live commentary had a crucial role in determining the number of words spoken during the commentary. Also, the commentator’s biased opinion was undeniable, especially in the polarity of the evaluative expressions they used. In each commentary, by nature, there was a predominantly focus on product or process. In LTC, since the commentator is watching the finished action, the focus is entirely product-oriented. LTC also has more frequent use of Affect resources due to the fact that Affect in general deals with evaluating objects and products or how products and performances are valued. In the other two modes of commentaries, given that the commentators are reporting the events happening in the game in real-time and in the spur of moment, the focus is mostly on the process

Combining corpus and experimental methods to gain new insights into APPRAISAL in spoken discourse

The mainobjective of this paper is to challenge the treatment of first-person epistemicand evidential complement-taking predicate (CTP) constructions in Martin andWhite’s (2005) appraisal theory,and to offer suggestions for improving the model. Based on empirical evidencefrom a corpus-based analysis and a laboratory experiment, we demonstrate that CTPconstructions do not only serve to expand the dialogic context in which theyoccur, but also to put a lid on alternative views. The paper contributes to the refinement of appraisal as a corpus annotation tool,and provides a practical illustration of the usefulness of combining corpus andexperimental methodsto gain new and robust insights into linguistic phenomena. In appraisal, first-person CTPs such as I think and I believe are classified as instances of dialogic expansion in that they make dialogic space for possiblesubsequent arguments (Martin & White, 2005: 98). They are set in contrast toexpressions of dialogic contraction (e.g.obviously, however, but), which areused in discourse to challenge, resist or reject alternative value positions. Inspoken discourse, however, CTP constructions do not appear to always perform anexpansive function. In (1), for example, Ithink is prosodically unaccented, serving as a starting point for theopinion expressed in the complement clause, which contains an evidential markerwith a falling accent – obviously –signalling a high degree of commitment (Cruttenden, 1997; Kärkkäinen, 2003). (1) B: I think he was \obviously trying to steer us in that direction and sort of A: yes B: dropping hints Martinand White (2005: 103) recognize that the function of engagement expressions "may vary systematically under theinfluence of differentco-textual conditions, and across registers, genres and discourse domains."These conditions, however, are not discussed in detail by the authors, nor havethey been systematically investigated in the literature. In this study, we combinecorpus-based and experimental methods to test the effect of different contextualfactors on the dialogic function of CTP constructions. The study is conducted in twophases. First, an exploratory analysis of CTP constructions in the London-LundCorpus (LLC) of spoken British English is carried out to identify factors thatmay play a role in determining the dialogic force of the constructions. The annotationof CTP constructions in the corpus is performed following Fuoli’s (forthcoming) step-wise method for annotatingappraisal, and is validatedthrough an inter-rater reliability test. Hypotheses derived from the corpusfindings are then tested in a controlled experimental setting. The resultsindicate that CTP constructions not only serve to expand the dialogic contextin which they occur, but may also function to inhibit dialogue. Interlocutorstatus, prosodic marking and the co-occurrence of a contractive marker are shownto have a significant effect on the function of CTP constructions

Assessing the potential of LLM-assisted annotation for corpus-based pragmatics and discourse analysis:The case of apologies

Certain forms of linguistic annotation, like part of speech and semantic tagging, can be automated with high accuracy. However, manual annotation is still necessary for complex pragmatic and discursive features that lack a direct mapping to lexical forms. This manual process is time-consuming and error-prone, limiting the scalability of function-to-form approaches in corpus linguistics. To address this, our study explores the possibility of using large language models (LLMs) to automate pragma-discursive corpus annotation. We compare GPT-3.5 (the model behind the free-to-use version of ChatGPT), GPT-4 (the model underpinning the precise mode of Bing chatbot), and a human coder in annotating apology components in English based on the local grammar framework. We find that GPT-4 outperformed GPT-3.5, with accuracy approaching that of a human coder. These results suggest that LLMs can be successfully deployed to aid pragma-discursive corpus annotation, making the process more efficient, scalable and accessible

Assessing the potential of LLM-assisted annotation for corpus-based pragmatics and discourse analysis:The case of apologies

Certain forms of linguistic annotation, like part of speech and semantic tagging, can be automated with high accuracy. However, manual annotation is still necessary for complex pragmatic and discursive features that lack a direct mapping to lexical forms. This manual process is time-consuming and error-prone, limiting the scalability of function-to-form approaches in corpus linguistics. To address this, our study explores the possibility of using large language models (LLMs) to automate pragma-discursive corpus annotation. We compare GPT-3.5 (the model behind the free-to-use version of ChatGPT), GPT-4 (the model underpinning the precise mode of Bing chatbot), and a human coder in annotating apology components in English based on the local grammar framework. We find that GPT-4 outperformed GPT-3.5, with accuracy approaching that of a human coder. These results suggest that LLMs can be successfully deployed to aid pragma-discursive corpus annotation, making the process more efficient, scalable and accessible

Discourse Relations and Evaluation

We examine the role of discourse relations (relations between propositions) in the interpretation of evaluative or opinion words. Through a combination of Rhetorical Structure Theory or RST (Mann & Thompson, 1988) and Appraisal Theory (Martin & White, 2005), we analyze how different discourse relations modify the evaluative content of opinion words, and what impact the nucleus-satellite structure in RST has on the evaluation. We conduct a corpus study, examining and annotating over 3,000 evaluative words in 50 movie reviews in the SFU Review Corpus (Taboada, 2008) with respect to five parameters: word category (nouns, verbs, adjectives or adverbs), prior polarity (positive, negative or neutral), RST structure (both nucleus-satellite status and relation type) and change of polarity as a result of being part of a discourse relation (Intensify, Downtone, Reversal or No Change). Results show that relations such as Concession, Elaboration, Evaluation, Evidence and Restatement most frequently intensify the polarity of the opinion words, although the majority of evaluative words (about 70%) do not undergo changes in their polarity because of the relations they are a part of. We also find that most opinion words (about 70%) are positioned in the nucleus, confirming a hypothesis in the literature, that nuclei are the most important units when extracting evaluation automatically