The challenging task of summary evaluation: an overview

Evaluation is crucial in the research and development of automatic summarization applications, in order to determine the appropriateness of a summary based on different criteria, such as the content it contains, and the way it is presented. To perform an adequate evaluation is of great relevance to ensure that automatic summaries can be useful for the context and/or application they are generated for. To this end, researchers must be aware of the evaluation metrics, approaches, and datasets that are available, in order to decide which of them would be the most suitable to use, or to be able to propose new ones, overcoming the possible limitations that existing methods may present. In this article, a critical and historical analysis of evaluation metrics, methods, and datasets for automatic summarization systems is presented, where the strengths and weaknesses of evaluation efforts are discussed and the major challenges to solve are identified. Therefore, a clear up-to-date overview of the evolution and progress of summarization evaluation is provided, giving the reader useful insights into the past, present and latest trends in the automatic evaluation of summaries.This research is partially funded by the European Commission under the Seventh (FP7 - 2007- 2013) Framework Programme for Research and Technological Development through the SAM (FP7-611312) project; by the Spanish Government through the projects VoxPopuli (TIN2013-47090-C3-1-P) and Vemodalen (TIN2015-71785-R), the Generalitat Valenciana through project DIIM2.0 (PROMETEOII/2014/001), and the Universidad Nacional de Educación a Distancia through the project “Modelado y síntesis automática de opiniones de usuario en redes sociales” (2014-001-UNED-PROY)

A study on digital-based argumentative writing in English of South Korean university students

In higher education in South Korea, English proficiency has been specifically emphasised by the government (Kang, 2015; Kang, 2018; Kim, 2017; Shim & Park, 2008; Williams, 2015). However, writing skills have had little attention in education settings, including higher education institutions (Kim, 2018; Shin, 2018; Park, 2020; Shin & Hyun, 2020; Yu, 2019), despite a series of educational reforms. Students in South Korean higher education are now facing practical and specific needs for argumentative writing in English (Shim, 2016; Shin, 2018). However, the overall context of English education does not fully reflect their real needs (Kim, 2018; Kwon, 2012; Kwak, 2017; Shim, 2016). South Korean universities require their students to reach a specific level at one of the English proficiency tests (Kim, 2018; Ma, 2018; Shim, 2016), most of which include at least one argumentative writing task. Additionally, the certificate of English proficiency test is widely used as the basic skills reference for their career (Kim, 2018). In the meantime, writing proficiency has increasingly gained its own weight in English language tests (Kim, 2018; Ma, 2018; Shin, 2018), adding to the burden on students to develop their writing proficiency (Kim, 2018; Ma, 2018). Despite students’ need for improvement in English writing proficiency, including English argumentative writing, writing courses given by South Korean higher education institutions are still rare (Kim, 2018; Ma, 2018; Shin, 2018; Yu, 2019) and often allow little room for reviewing tasks (Kim, 2018; Ma, 2018; Shim, 2018), even though they commonly use a process-based approach. Furthermore, in immediate response to their needs, higher education institutions in both the public and private sectors have maintained narrow academic attention, focusing on test specific writing skills (Kim, 2018; Shin, 2018). All these situations have resulted in a lack of educational opportunities for students to receive theoretically and systematically well designed instruction in developing their argumentation skills (Shin, 2018). For South Korean students learning English as a foreign language (EFL), argumentative writing in English includes acquiring an understanding of and the skills for both critical thinking and English-specific conventions for the target genre of writing (Ahn & Park, 2019; Choi, 2008; Shim, 2016). To promote a fast and concrete understanding of argumentation in English, representative organisational structures are often used in instructional practices. While many of the courses for English argumentative writing in South Korea are limited to the delivery of instructions, or creating a rough claim-evidence link in a paragraph, this simple formula-based approach may have a limited influence on the level of argumentation that university students in South Korea are able to develop (Choi, 2008). To enhance students’ in-depth knowledge of and skills for making arguments in English, a systematic and effective instructional model is necessary, targeting argumentation development and investigated by rigorous research. However, with a traditionally narrow focus on writing in English education, studies on English writing itself, including argumentative writing, have been limited, despite the importance of this area. As a way of introducing systematically presented models into instruction in English argumentative writing, the Toulmin model can be an effective option. It suggests a detailed, sequenced, intensively explained process for the logical framework for writing in English. In this sense, it is necessary to explore how to modify and apply Toulmin’s components into the courses for English argumentative writing in South Korean higher education. In addition to the practical applicability of the Toulmin model, it is necessary to consider the common context in which writing courses in South Korean higher education institutions provide some phases for drafting and revision/editing, which are broadly anchored in the process-based writing approach. Considering the practical challenge caused by a lack of time for drafting in writing courses (Kim, 2018; Ma, 2018; Shim, 2018), online based classes can be a better option, enabling more flexibility in time and space. Even before the Covid-19 outbreak, diverse synchronous and asynchronous digital writing environments have been utilised in the field of higher education in South Korea to enhance students’ writing performance and also increase the connectivity between learners and teachers. However, the digital environments for English writing in South Korea are still based on a lack of rigour in terms of research evidence, which signals the need for more research into how best to develop digital writing platforms and incorporate necessary support for users. With the two main areas of English argumentative writing and digital learning environments for writing combined, this study explores the effectiveness of a digital-based argumentative writing course in South Korea, as well as the pedagogical implications. To investigate the effects of digital course development for instruction in English argumentative writing for university students in South Korea and derive insights in digital course design for English argumentative writing for university students in South Korea, this study used a sequential mixed-methods design: quantitative phase followed by qualitative phase for collection and analysis of data sets. The English argumentative writing course in this study applies the Toulmin model (1958; 2003) as a specific teaching strategy, with a cycle of drafting and exchanging feedback using the process-based writing approach. To provide the online group with a digital-based collaborative writing1 environment for feedback exchanges, the writing platform, Scholar, was used. In this study, 43 undergraduate students in South Korea participated in a writing course for one semester, 22 participants in a control group (offline course) and 21 participants in an intervention group (online course). They participated in pre- and post-writing tests, two sessions of interviews, and narrative writing for reflection. Also, ten university teaching staff and e-developers took part in one individual interview session each, to provide professional views on the online instructional design that is implemented in the English argumentative writing of this study. In terms of the effectiveness of the online writing course for developing argumentation skills in English, the findings from the quantitative analysis show both online and offline courses had a positive impact on improvement and retention. Although the statistical results present no indication that the online class had higher learning gains than the offline group by any significant difference, this result is supported by the findings from the qualitative analysis, which indicates that the online group performed better in terms of the quality and the quantity of peer feedback. In addition, the findings from the qualitative analysis suggest that the writing course in this study helped students to develop their knowledge and sensitivity in argumentation in English, and the online course facilitated enhanced engagement in feedback tasks. Moreover, despite recognising the value of face-to-face interaction for English argumentative writing, the qualitative findings suggest that the anonymity and convenience of the online writing course in this study encouraged participation in feedback. Finally, the findings from teaching staff and e-developer interviews reveal generally positive perceptions of and evaluations of the usefulness and applicability of the Toulmin model for English argumentation development, and the collaborative writing environment of Scholar. ( 1. In this study, the term, ‘collaborative writing,’ means individual student’s essay writing supported by external feedback, including peer and teacher feedback, not co-authorship in writing one shared essay together.

Language Learning Tasks and Automatic Analysis of Learner Language: Connecting FLTL and NLP design of ICALL materials supporting use in real-life instruction

This thesis studies the application of Natural Language Processing to Foreign Language Teaching and Learning, within the research area of Intelligent Computer- Assisted Language Learning (ICALL). In particular, we investigate the design, the implementation, and the use of ICALL materials to provide learners of foreign languages, particularly English, with automated feedback. We argue that the successful integration of ICALL materials demands a design process considering both pedagogical and computational requirements as equally important. Our investigation pursues two goals. The first one is to integrate into task design insights from Second Language Acquisition and Foreign Language Teaching and Learning with insights from computational linguistic modelling. The second goal is to facilitate the integration of ICALL materials in real-world instruction settings, as opposed to research or lab-oriented instruction settings, by empowering teachers with the methodology and the technology to autonomously author such materials. To achieve the first goal, we propose an ICALL material design process that combines basic principles of Task-Based Language Instruction and Task-Based Test Design with the specification requirements of Natural Language Processing. The relation between pedagogical and computational requirements is elucidated by exploring (i) the formal features of foreign language learning activities, (ii) the complexity and variability of learner language, and (iii) the feasibility of applying computational techniques for the automatic analysis and evaluation of learner responses. To achieve the second goal, we propose an automatic feedback generation strategy that enables teachers to customise the computational resources required to automatically correct ICALL activities without the need for programming skills. This proposal is instantiated and evaluated in real world-instruction settings involving teachers and learners in secondary education. Our work contributes methodologically and empirically to the ICALL field, with a novel approach to the design of materials that highlights the cross-disciplinary and iterative nature of the task. Our findings reveal the strength of characterising tasks both from the perspective of Foreign Language Teaching and Learning and from the perspective of Computational Linguistics as a means to clarify the nature of learning activities. Such a characterisation allows us to identify ICALL materials which are both pedagogically meaningful and computationally feasible. Our results show that teachers can characterise, author and employ ICALL mate- rials as part of their instruction programme, and that the underlying computational machinery can provide the required automatic processing with sufficient efficiency. The authoring tool and the accompanying methodology become a crucial instrument for ICALL research and practice: Teachers are able to design activities for their students to carry out without relying on an expert in Natural Language Processing. Last but not least, our results show that teachers are value the experience very positively as means to engage in technology integration, but also as a means to better apprehend the nature of their instruction task. Moreover, our results show that learners are motivated by the opportunity of using a technology that enhances their learning experience

On the Mono- and Cross-Language Detection of Text Re-Use and Plagiarism

Barrón Cedeño, LA. (2012). On the Mono- and Cross-Language Detection of Text Re-Use and Plagiarism [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/16012Palanci

Time- and value-continuous explainable affect estimation in-the-wild

Today, the relevance of Affective Computing, i.e., of making computers recognise and simulate human emotions, cannot be overstated. All technology giants (from manufacturers of laptops to mobile phones to smart speakers) are in a fierce competition to make their devices understand not only what is being said, but also how it is being said to recognise user’s emotions. The goals have evolved from predicting the basic emotions (e.g., happy, sad) to now the more nuanced affective states (e.g., relaxed, bored) real-time. The databases used in such research too have evolved, from earlier featuring the acted behaviours to now spontaneous behaviours. There is a more powerful shift lately, called in-the-wild affect recognition, i.e., taking the research out of the laboratory, into the uncontrolled real-world. This thesis discusses, for the very first time, affect recognition for two unique in-the-wild audiovisual databases, GRAS2 and SEWA. The GRAS2 is the only database till date with time- and value-continuous affect annotations for Labov effect-free affective behaviours, i.e., without the participant’s awareness of being recorded (which otherwise is known to affect the naturalness of one’s affective behaviour). The SEWA features participants from six different cultural backgrounds, conversing using a video-calling platform. Thus, SEWA features in-the-wild recordings further corrupted by unpredictable artifacts, such as the network-induced delays, frame-freezing and echoes. The two databases present a unique opportunity to study time- and value-continuous affect estimation that is truly in-the-wild. A novel ‘Evaluator Weighted Estimation’ formulation is proposed to generate a gold standard sequence from several annotations. An illustration is presented demonstrating that the moving bag-of-words (BoW) representation better preserves the temporal context of the features, yet remaining more robust against the outliers compared to other statistical summaries, e.g., moving average. A novel, data-independent randomised codebook is proposed for the BoW representation; especially useful for cross-corpus model generalisation testing when the feature-spaces of the databases differ drastically. Various deep learning models and support vector regressors are used to predict affect dimensions time- and value-continuously. Better generalisability of the models trained on GRAS2 , despite the smaller training size, makes a strong case for the collection and use of Labov effect-free data. A further foundational contribution is the discovery of the missing many-to-many mapping between the mean square error (MSE) and the concordance correlation coefficient (CCC), i.e., between two of the most popular utility functions till date. The newly invented cost function |MSE_{XY}/σ_{XY}| has been evaluated in the experiments aimed at demystifying the inner workings of a well-performing, simple, low-cost neural network effectively utilising the BoW text features. Also proposed herein is the shallowest-possible convolutional neural network (CNN) that uses the facial action unit (FAU) features. The CNN exploits sequential context, but unlike RNNs, also inherently allows data- and process-parallelism. Interestingly, for the most part, these white-box AI models have shown to utilise the provided features consistent with the human perception of emotion expression

Promotion of critical thinking in school physical science.

Thesis (Ph.D.)-University of KwaZulu-Natal, Durban, 2008.This dissertation describes an action research study aimed at promoting critical thinking in learners while learning physical science within the South African national curriculum. The data were primarily qualitative in nature, and were collected primarily through participant observation, composed of audio- and video- recorded lessons, interviews, questionnaires, journal entries and written material. Data collection, analysis and interpretation were done in the inductive, cyclic manner of action research. This process was guided by research questions about task characteristics, their position in the teaching sequence, the role of the learning environment, and the need to adjust tasks to fit the needs of different learners, so as to effectively promote critical thinking. A pragmatic approach was used. It was found that it is possible, using particular strategies and tasks, to promote critical thinking while meeting the curriculum outcomes, although the intense syllabus pressure of the curriculum makes this challenging. Task design characteristics and positioning in the teaching sequence, and conditions of the learning environment, were found to affect a task’s effectiveness at promoting critical thinking. Various teaching strategies can improve attainability by a wider range of learners. An instructional model, The Ladder Approach, emerged as being most likely to promote success. This was found to be successful when evaluated against criteria of active engagement and interest by learners, attainability with effort, display of critical thinking traits, and compatibility with the South African curriculum. In this model, an interesting problem is posed at the start of a section, after which direct instruction and learner engagement with the problem run parallel to one another, linked by scaffolding tools which are engaged in individually and collaboratively

Developing Learning Analytics for Epistemic Commitments in a Collaborative Information Seeking Environment

Learning analytics sits at the confluence of learning, information, and computer sciences. Using a distinctive account of learning analytics as a form of assessment, I first argue for its potential in pedagogically motivated learning design, suggesting a particular construct – epistemic cognition in literacy contexts – to probe using learning analytics. I argue for a recasting of epistemic cognition as ‘epistemic commitments’ in collaborative information tasks drawing a novel alignment between information seeking and multiple document processing (MDP) models, with empirical and theoretical grounding given for a focus on collaboration and dialogue in such activities. Thus, epistemic commitments are seen in the ways students seek, select, and integrate claims from multiple sources, and the ways in which their collaborative dialogue is brought to bear in this activity. Accordingly, the empirical element of the thesis develops two pedagogically grounded literacy based tasks: a MDP task, in which pre-selected documents were provided to students; and a collaborative information seeking task (CIS), in which students could search the web. These tasks were deployed at scale (n > 500) and involved writing an evaluative review, followed by a pedagogically supported peer assessment task. Assessment outcomes were analysed in the context of a new epistemic commitments-oriented set of trace data, and psychometric data regarding the participants’ epistemic cognition. Demonstrating the value of the methodological and conceptual approach taken, qualitative analyses indicate clear epistemic activity, and stark differences in behaviour between groups, the complexity of which is challenging to model computationally. Despite this complexity, quantitative analyses indicate that up to 30% of variance in output scores can be modelled using behavioural indicators. The explanatory potential of behaviourally-oriented models of epistemic commitments grounded in tool-interaction and collaborative dialogue is demonstrated. The thesis provides an exemplification of theoretically positioned analytic development, drawing on interdisciplinary literatures in addressing complex learning contexts