Inter-Coder Agreement for Computational Linguistics

This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff's alpha as well as Scott's pi and Cohen's kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in computational linguistics, may be more appropriate for many corpus annotation tasks—but that their use makes the interpretation of the value of the coefficient even harder. </jats:p

An empirical study of the “prototype walkthrough”: a studio-based activity for HCI education

For over a century, studio-based instruction has served as an effective pedagogical model in architecture and fine arts education. Because of its design orientation, human-computer interaction (HCI) education is an excellent venue for studio-based instruction. In an HCI course, we have been exploring a studio-based learning activity called the prototype walkthrough, in which a student project team simulates its evolving user interface prototype while a student audience member acts as a test user. The audience is encouraged to ask questions and provide feedback. We have observed that prototype walkthroughs create excellent conditions for learning about user interface design. In order to better understand the educational value of the activity, we performed a content analysis of a video corpus of 16 prototype walkthroughs held in two HCI courses. We found that the prototype walkthrough discussions were dominated by relevant design issues. Moreover, mirroring the justification behavior of the expert instructor, students justified over 80 percent of their design statements and critiques, with nearly one-quarter of those justifications having a theoretical or empirical basis. Our findings suggest that PWs provide valuable opportunities for students to actively learn HCI design by participating in authentic practice, and provide insight into how such opportunities can be best promoted

Age group differences in performance using diverse input modalities: insertion task evaluation

Novel input modalities such as touch, tangibles or gestures try to exploit human's innate skills rather than imposing new learning processes. However, no work has been reported that systematically evaluates how these interfaces influence users' performance, that is, assesses if one interface can be more or less appropriate for interaction regarding: (1) different age groups; and (2) different basic tasks, as content insertion or manipulation. This work presents itself as an exploratory evaluation about whether or not the users' efficiency is indeed influenced by different input modalities and age. We conducted a usability evaluation with 60 subjects to understand how different interfaces may influence the speed and accuracy of three specific age groups (children, young adults and older-adults) when dealing with a basic content insertion task. Four input modalities were considered to perform the task (keyboard, touch, tangibles and gestures) and the methodology was based on usability testing (speed, accuracy and user preference). Overall, results show that there is a statistically significant difference in speed of task completion between the age groups, and there may be indications that the type of interface that is used can indeed influence efficiency in insertion tasks, and not so much other factors like age. Also, the study raises new issues regarding the "old" mouse input versus the "new" input modalities.FCT – Fundação para a Ciência e a Tecnologia (SFRH/BD/81541/2011)COMPETE: POCI-01-0145- FEDER-007043 and FCT – Fundação para a Ciência e Tecnologia within the Project Scope: UID/CEC/00319/201

IN THIS ISSUE: What Are Characteristics of Significant Research?

Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/67303/2/10.1177_108056999505800214.pd

The United Kingdom 2017 election:polarisation in a split issue space

After decades in which party competition was fought in the centre ground, the 2017 UK General Election witnessed a return to more conflictual politics. This article assesses public support for the electoral strategies of the main parties and examines the extent to which the issues the parties campaigned on resonated with their own supporters, as well as with the wider public. Drawing on the issue-yield framework, the article shows that the Conservative campaign\u2013generally considered to be badly run\u2013did not focus on issues that would fully exploit the opportunities for expanding support that were open to the party. Labour, by contrast, played a much better hand. While taking a clear left-wing stance on many policies that were popular with its constituency, the party also skilfully emphasised valence issues that Labour is often seen as more credible on, such as healthcare and education

Measuring fidelity to behavioural support delivery for smoking cessation and its association with outcomes

BACKGROUND AND AIMS: Behavioural support increases smoking cessation in clinical settings, but effect sizes differ among providers, due possibly to variations in delivery. This study evaluates a measure ('fidelity index') intended to capture fidelity to delivery of content- and interaction-based items of a behavioural support (BS) for smoking cessation and the association of fidelity with quit rates. METHODS: A fidelity index for scoring the adherence and quality domains of a specific BS intervention, '5As for quit', was developed by classifying the intervention components using the taxonomy of behaviour change techniques. The index was applied to code 154 BS sessions audiotaped among 18 chest clinics in Pakistan to assess their fidelity and explore reliability of coding. The association between intervention fidelity and successful quit achieved by the same providers in a previous study was explored using regression analysis. RESULTS: The index represented two domains: adherence to delivery of content-based activities of 5As (37 items) and quality of interaction-based activities (eight items). The intercoder reliability was good for content-based (average Krippendorff's α = 0.80) and moderate for interaction-based (average Krippendorff's α = 0.66) items. Approximately 70% (intraclass correlation coefficient: adherence scores = 0.72, quality scores = 0.71) of variation in BS delivery was contributed by providers, which increased to 97% (g-coefficient: adherence scores = 0.973, quality scores = 0.974) after accounting for other sources of variation. Higher quit rates were positively associated with average quality scores [risk ratio = 2.15; 95% confidence interval (CI) = 1.43-3.24], but negatively associated with average adherence scores (risk ratio = 0.55; 95% CI = 0.40-0.77) within services. CONCLUSIONS: The fidelity index is a reliable measure for quantifying intervention fidelity of delivering smoking cessation behavioural support. Recommended revisions of the fidelity index include incorporation of additional interaction-based items, such as the relational techniques used in motivational interviewing

The good, the bad and the implicit: a comprehensive approach to annotating explicit and implicit sentiment

We present a fine-grained scheme for the annotation of polar sentiment in text, that accounts for explicit sentiment (so-called private states), as well as implicit expressions of sentiment (polar facts). Polar expressions are annotated below sentence level and classified according to their subjectivity status. Additionally, they are linked to one or more targets with a specific polar orientation and intensity. Other components of the annotation scheme include source attribution and the identification and classification of expressions that modify polarity. In previous research, little attention has been given to implicit sentiment, which represents a substantial amount of the polar expressions encountered in our data. An English and Dutch corpus of financial newswire, consisting of over 45,000 words each, was annotated using our scheme. A subset of this corpus was used to conduct an inter-annotator agreement study, which demonstrated that the proposed scheme can be used to reliably annotate explicit and implicit sentiment in real-world textual data, making the created corpora a useful resource for sentiment analysis

Exploring the use of new school buildings through post-occupancy evaluation and participatory action research

This paper presents the results of the development and testing of an integrated post-occupancy evaluation (POE) approach for teachers, staff, pupils and community members using newly constructed school buildings. It focusses on three cases of UK secondary schools, demonstrating how users can be inspired to engage with the problems of school design and energy use awareness. The cases provided new insights into the engagement of school teachers, staff and young people regarding issues of sustainability, management, functional performance and comfort. The integrative approach adopted in these cases provided a more holistic understanding of these buildings’ performance than could have been achieved by either observational or more traditional questionnaire-based methods. Moreover, the whole-school approach, involving children in POE, provided researchers with highly contextualised information about how a school is used, how to improve the quality of school experiences (both socially and educationally) and how the school community is contributing to the building's energy performance. These POE methods also provided unique opportunities for children to examine the social and cultural factors impeding the adoption of energy-conscious and sustainable behaviours