Evaluation methodologies in Automatic Question Generation 2013-2018

In the last few years Automatic Question Generation (AQG) has attracted increasing interest. In this paper we survey the evaluation methodologies used in AQG. Based on a sample of 37 papers, our research shows that the systems’ development has not been accompanied by similar developments in the methodologies used for the systems’ evaluation. Indeed, in the papers we examine here, we find a wide variety of both intrinsic and extrinsic evaluation methodologies. Such diverse evaluation practices make it difficult to reliably compare the quality of different generation systems. Our study suggests that, given the rapidly increasing level of research in the area, a common framework is urgently needed to compare the performance of AQG systems and NLG systems more generally

Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review

Educational technology innovations leveraging large language models (LLMs) have shown the potential to automate the laborious process of generating and analysing textual content. While various innovations have been developed to automate a range of educational tasks (e.g., question generation, feedback provision, and essay grading), there are concerns regarding the practicality and ethicality of these innovations. Such concerns may hinder future research and the adoption of LLMs-based innovations in authentic educational contexts. To address this, we conducted a systematic scoping review of 118 peer-reviewed papers published since 2017 to pinpoint the current state of research on using LLMs to automate and support educational tasks. The findings revealed 53 use cases for LLMs in automating education tasks, categorised into nine main categories: profiling/labelling, detection, grading, teaching support, prediction, knowledge representation, feedback, content generation, and recommendation. Additionally, we also identified several practical and ethical challenges, including low technological readiness, lack of replicability and transparency, and insufficient privacy and beneficence considerations. The findings were summarised into three recommendations for future studies, including updating existing innovations with state-of-the-art models (e.g., GPT-3/4), embracing the initiative of open-sourcing models/systems, and adopting a human-centred approach throughout the developmental process. As the intersection of AI and education is continuously evolving, the findings of this study can serve as an essential reference point for researchers, allowing them to leverage the strengths, learn from the limitations, and uncover potential research opportunities enabled by ChatGPT and other generative AI models

W(h)ither Practitioner Research?

The purpose of this paper is to understand better the possibilities for practitioner research as a mode of educational inquiry that is yet to be legitimated within the academy. The paper maps the current state of play, and then moves on to consider what might yet be done to optimise its potential to contribute to rigorous new thinking about educational practice. Its exploration proceeds in 3 parts: first, it seeks to account for the ambivalent status of practitioner research in the larger context of the modern university; second, it moves on from this account to argue both the value and the limitations of practitioner research as a contemporary mode of knowledge production in education; and finally, it suggests ways that practitioner research might be less de-limited in terms of its capacities to produce knowledge that is useful to a wider range of stakeholders

Instructional strategies and tactics for the design of introductory computer programming courses in high school

This article offers an examination of instructional strategies and tactics for the design of introductory computer programming courses in high school. We distinguish the Expert, Spiral and Reading approach as groups of instructional strategies that mainly differ in their general design plan to control students' processing load. In order, they emphasize topdown program design, incremental learning, and program modification and amplification. In contrast, tactics are specific design plans that prescribe methods to reach desired learning outcomes under given circumstances. Based on ACT* (Anderson, 1983) and relevant research, we distinguish between declarative and procedural instruction and present six tactics which can be used both to design courses and to evaluate strategies. Three tactics for declarative instruction involve concrete computer models, programming plans and design diagrams; three tactics for procedural instruction involve worked-out examples, practice of basic cognitive skills and task variation. In our evaluation of groups of instructional strategies, the Reading approach has been found to be superior to the Expert and Spiral approaches

Leading the evaluation of institutional online learning environments for quality enhancement in times of change

This paper reports on findings from a nationally funded project which aims to design and implement a quality management framework for online learning environments (OLEs). Evaluation is a key component of any quality management system and it is this aspect of the framework that is the focus of this paper. In developing the framework initial focus groups were conducted at the five participating institutions. These revealed that, although regarded as important, there did not appear to be a shared understanding of the nature and purpose of evaluation. A second series of focus groups revealed there were multiple perspectives arising from those with a vested interest in online learning. These perspectives will be outlined. Overall, how evaluation was undertaken was highly variable within and across the five institutions reflecting where they were at in relation to the development of their OLE

Bridging the gap between digital libraries and e-learning

Digital Libraries (DL) are offering access to a vast amount of digital content, relevant to practically all domains of human knowledge, which makes it suitable to enhance teaching and learning. Based on a systematic literature review, this article provides an overview and a gap analysis of educational use of DLs.The research work presented in this paper is partially supported by the FP7 Grant 316087 AComIn ”Advanced Computing for Innovation”, funded by the European Commission in the FP7 Capacity Programme in 2012-2016.peer-reviewe

Syn-QG: Syntactic and Shallow Semantic Rules for Question Generation

Question Generation (QG) is fundamentally a simple syntactic transformation; however, many aspects of semantics influence what questions are good to form. We implement this observation by developing Syn-QG, a set of transparent syntactic rules leveraging universal dependencies, shallow semantic parsing, lexical resources, and custom rules which transform declarative sentences into question-answer pairs. We utilize PropBank argument descriptions and VerbNet state predicates to incorporate shallow semantic content, which helps generate questions of a descriptive nature and produce inferential and semantically richer questions than existing systems. In order to improve syntactic fluency and eliminate grammatically incorrect questions, we employ back-translation over the output of these syntactic rules. A set of crowd-sourced evaluations shows that our system can generate a larger number of highly grammatical and relevant questions than previous QG systems and that back-translation drastically improves grammaticality at a slight cost of generating irrelevant questions.Comment: Some of the results in the paper were incorrec

Easing the transition from paper to screen: an evaluatory framework for CAA migration

Computer assisted assessment is becoming more and more common through further and higher education. There is some debate about how easy it will be to migrate current assessment practice to a computer enhanced format and how items which are currently re-used for formative purposes may be adapted to be presented online. This paper proposes an evaluatory framework to assess and enhance the practicability of large-scale CAA migration for existing items and assessments. The framework can also be used as a tool for exposing compromises between delivery mechanism and validity-exposing the limits of validity of modified paper based assessments and highlighting the crucial areas for transformative assessments

Investigation of gas circulator response to load transients in nuclear power plant operation

Gas circulator units are a critical component of the Advanced Gas-cooled Reactor (AGR), one of the nuclear power plant (NPP) designs in current use within the UK. The condition monitoring of these assets is central to the safe and economic operation of the AGRs and is achieved through analysis of vibration data. Due to the dynamic nature of reactor operation, each plant item is subject to a variety of system transients of which engineers are required to identify and reason about with regards to asset health. The AGR design enables low power refueling (LPR) which results in a change in operational state for the gas circulators, with the vibration profile of each unit reacting accordingly. The changing conditions subject to these items during LPR and other such events may impact on the assets. From these assumptions, it is proposed that useful information on gas circulator condition can be determined from the analysis of vibration response to the LPR event. This paper presents an investigation into asset vibration during an LPR. A machine learning classification approach is used in order to define each transient instance and its behavioral features statistically. Classification and reasoning about the regular transients such as the LPR represents the primary stage in modeling higher complexity events for advanced event driven diagnostics, which may provide an enhancement to the current methodology, which uses alarm boundary limits