Assessing Critical Thinking in Open-ended Answers: An Automatic Approach

The role of Higher Education (HE) is growingly acknowledged for the promotion of Critical Thinking (CT). Constructed-response tasks (CRT) are recognized to be necessary for the CT assessment, though they present problems related to scoring quality and cost (Ku, 2009). Researchers (Liu, Frankel, Roohr, 2014) have proposed using automated scoring to address the above concerns. The present work is aimed at comparing the features of different Natural Language Processing (NLP) techniques adopted to improve the reliability of a prototype designed to automatically assess six sub-skills of CT in CRT: use of language, argumentation, relevance, importance, critical evaluation and novelty (Poce, 2017). We will present the first (1.0) and the second (2.0) version of the CT prototype and their respective reliability results. Our research question is the following: Which level of reliability are shown respectively by the 1.0 and 2.0 automatic CT assessment prototype compared to expert human evaluation? Data collection is realized in two moments, to measure respectively the CT prototype 1.0 and 2.0 reliability from a total of 264 participants and 592 open-ended answers. Two human assessors rated all of these responses on each of the subskills on a scale of 1-5. Similarly, NLP approaches are adopted to compute a feature on each dimension. Quadratic Weighted Kappa and Pearson product-moment correlation were used to evaluate the between-human agreement and human-NLP agreement. Preliminary findings based on the first data set suggest adequate level of between-human rating agreement and a lower level human-NLP agreement (r .43 for the subscales of Relevance and Importance). We are continuing the analysis of the data collected in the 2nd step and expect to complete them in June 2020

A Qualitative Analysis of the Persuasive Properties of Argumentation Schemes

Argumentation schemes are generalised patterns that provide a way to (partially) dissociate the content from the reasoning structure of the argument. On the other hand, Cialdini’s principles of persuasion provide a generic model to analyse the persuasive properties of human interaction (e.g., natural language). Establishing the relationship between principles of persuasion and argumentation schemes can contribute to the improvement of the argument-based human-computer interaction paradigm. In this work, we perform a qualitative analysis of the persuasive properties of argumentation schemes. For that purpose, we present a new study conducted on a population of over one hundred participants, where twelve different argumentation schemes are instanced into four different topics of discussion considering both stances (i.e., in favour and against). Participants are asked to relate these argumentation schemes with the perceived Cialdini’s principles of persuasion. From the results of our study, it is possible to conclude that some of the most commonly used patterns of reasoning in human communication have an underlying persuasive focus, regardless of how they are instanced in natural language argumentation (i.e., their stance, the domain, or their content)

A Qualitative Analysis of the Persuasive Properties of Argumentation Schemes

Argumentation schemes are generalised patterns that provide a way to (partially) dissociate the content from the reasoning structure of the argument. On the other hand, Cialdini’s principles of persuasion provide a generic model to analyse the persuasive properties of human interaction (e.g., natural language). Establishing the relationship between principles of persuasion and argumentation schemes can contribute to the improvement of the argument-based human-computer interaction paradigm. In this work, we perform a qualitative analysis of the persuasive properties of argumentation schemes. For that purpose, we present a new study conducted on a population of over one hundred participants, where twelve different argumentation schemes are instanced into four different topics of discussion considering both stances (i.e., in favour and against). Participants are asked to relate these argumentation schemes with the perceived Cialdini’s principles of persuasion. From the results of our study, it is possible to conclude that some of the most commonly used patterns of reasoning in human communication have an underlying persuasive focus, regardless of how they are instanced in natural language argumentation (i.e., their stance, the domain, or their content)

Parsing Argumentation Structures in Persuasive Essays

In this article, we present a novel approach for parsing argumentation structures. We identify argument components using sequence labeling at the token level and apply a new joint model for detecting argumentation structures. The proposed model globally optimizes argument component types and argumentative relations using integer linear programming. We show that our model considerably improves the performance of base classifiers and significantly outperforms challenging heuristic baselines. Moreover, we introduce a novel corpus of persuasive essays annotated with argumentation structures. We show that our annotation scheme and annotation guidelines successfully guide human annotators to substantial agreement. This corpus and the annotation guidelines are freely available for ensuring reproducibility and to encourage future research in computational argumentation.Comment: Under review in Computational Linguistics. First submission: 26 October 2015. Revised submission: 15 July 201