Rethinking the Agreement in Human Evaluation Tasks

Human evaluations are broadly thought to be more valuable the higher the inter-annotator agreement. In this paper we examine this idea. We will describe our experiments and analysis within the area of Automatic Question Generation. Our experiments show how annotators diverge in language annotation tasks due to a range of ineliminable factors. For this reason, we believe that annotation schemes for natural language generation tasks that are aimed at evaluating language quality need to be treated with great care. In particular, an unchecked focus on reduction of disagreement among annotators runs the danger of creating generation goals that reward output that is more distant from, rather than closer to, natural human-like language. We conclude the paper by suggesting a new approach to the use of the agreement metrics in natural language generation evaluation tasks

Evaluation methodologies in Automatic Question Generation 2013-2018

In the last few years Automatic Question Generation (AQG) has attracted increasing interest. In this paper we survey the evaluation methodologies used in AQG. Based on a sample of 37 papers, our research shows that the systems’ development has not been accompanied by similar developments in the methodologies used for the systems’ evaluation. Indeed, in the papers we examine here, we find a wide variety of both intrinsic and extrinsic evaluation methodologies. Such diverse evaluation practices make it difficult to reliably compare the quality of different generation systems. Our study suggests that, given the rapidly increasing level of research in the area, a common framework is urgently needed to compare the performance of AQG systems and NLG systems more generally

Rethinking university assessment

Developments in globalisation and new technologies are making significant impacts in higher education. Universities in a global market are increasingly concerned to reorient their degree programmes to meet the vocational needs of the Knowledge Economy. A growing adoption of technology enhanced learning, through blended and networked learning, has the potential to transform higher education practice – but assessment methods have been slow to change. This paper argues the case for universities to align assessment methods to meet the needs of 21st Century knowledge workers. It identifies skills and dispositions associated with graduate occupations in the Knowledge Economy, informing a new conceptual model for assessment. Radical recommendations are made to faculty staff and university policymakers: instead of centring assessment on the personal, academic achievements of individuals at the end of a degree course, the focus should instead be on the quality of the collective, applied achievements of students operating in project teams

Reflections on Mira : interactive evaluation in information retrieval

Evaluation in information retrieval (IR) has focussed largely on noninteractive evaluation of text retrieval systems. This is increasingly at odds with how people use modern IR systems: in highly interactive settings to access linked, multimedia information. Furthermore, this approach ignores potential improvements through better interface design. In 1996 the Commission of the European Union Information Technologies Programme, funded a three year working group, Mira, to discuss and advance research in the area of evaluation frameworks for interactive and multimedia IR applications. Led by Keith van Rijsbergen, Steve Draper and myself from Glasgow University, this working group brought together many of the leading researchers in the evaluation domain from both the IR and human computer interaction (HCI) communities. This paper presents my personal view of the main lines of discussion that took place throughout Mira: importing and adapting evaluation techniques from HCI, evaluating at different levels as appropriate, evaluating against different types of relevance and the new challenges that drive the need for rethinking the old evaluation approaches. The paper concludes that we need to consider more varied forms of evaluation to complement engine evaluation

A geographical issue: the contribution of Citizenship Education to the building of a European citizenship. The case of the VOICEs Comenius network

Citizenship Education is currently a consolidated issue within several European curricula. It has been integrated in national educational laws in different ways: as cross-curricular education (UK, Italy), as a subject (France, Spain) or as a skill (Ireland). Despite these differences, there is a common agreement on the ethical value of Citizenship Education and on its main aim: to foster students’ sense of local, national and European citizenship. In some ways this goal has been inspired by Morin’s path to a “plural” education and a planetary citizenship (Morin, 2000). Social sciences, and in particular Geography and History, keep the function of giving tools able to show how a dialogue among the different scales is possible. Nevertheless European citizenship is undergoing a constant redefinition due to the European enlargement process, the role of Europe inside national jurisdictions and to the changes in national curricula. This evolution directly affects the guiding function conferred to school in terms of skills, aims and themes; therefore competences and methods adopted by teachers may have to be reconsidered. This essay presents the first results of the updating of the state of the art of this issue that has been carried out by the Citizenship Education Research Group of the VOICEs Comenius network (The Voice of European Teachers). The main aim of this international research group is to face the challenge of building a European citizenship by developing a comparative analysis of teachers’ practices and strategies in different local, regional and national contexts, aiming to contribute, with renewed ideas, to the debate on this promising field of research

Rethinking International Subsidy Rules. Bertelsmann Working Paper 28/02/2020

Geo-economic tensions and global collective action problems call for international cooperation to revise and de-velop rules to guide both the use of domestic subsidies and responses by governments to cross-border competition spillover effects. Current WTO rules that divide all subsidies into either prohibited or actionable cate-gories are no longer fit for purpose. Piecemeal efforts in preferential trade agreements and bi- or trilateral configurations offer a basis on which to build, but are too narrow in scope and focus. Addressing the spillover ef-fects of subsidies could start with launching a work program at the 12th Ministerial Conference of the WTO to mobilize an epistemic community concerned with subsidy policies, tasked with building a more solid evidence base on the magnitude, purpose and effects of subsidy policies