Defining and Assessing Critical Thinking: toward an automatic analysis of HiEd students’ written texts

L'obiettivo principale di questa tesi di dottorato è testare, attraverso due studi empirici, l'affidabilità di un metodo volto a valutare automaticamente le manifestazioni del Pensiero Critico (CT) nei testi scritti da studenti universitari. Gli studi empirici si sono basati su una review critica della letteratura volta a proporre una nuova classificazione per sistematizzare le diverse definizioni di CT e i relativi approcci teorici. La review esamina anche la relazione tra le diverse definizioni di CT e i relativi metodi di valutazione. Dai risultati emerge la necessità di concentrarsi su misure aperte per la valutazione del CT e di sviluppare strumenti automatici basati su tecniche di elaborazione del linguaggio naturale (NLP) per superare i limiti attuali delle misure aperte, come l’attendibilità e i costi di scoring. Sulla base di una rubrica sviluppata e implementata dal gruppo di ricerca del Centro di Didattica Museale – Università di Roma Tre (CDM) per la valutazione e l'analisi dei livelli di CT all'interno di risposte aperte (Poce, 2017), è stato progettato un prototipo per la misurazione automatica di alcuni indicatori di CT. Il primo studio empirico condotto su un gruppo di 66 docenti universitari mostra livelli di affidabilità soddisfacenti della rubrica di valutazione, mentre la valutazione effettuata dal prototipo non era sufficientemente attendibile. I risultati di questa sperimentazione sono stati utilizzati per capire come e in quali condizioni il modello funziona meglio. La seconda indagine empirica era volta a capire quali indicatori del linguaggio naturale sono maggiormente associati a sei sottodimensioni del CT, valutate da esperti in saggi scritti in lingua italiana. Lo studio ha utilizzato un corpus di 103 saggi pre-post di studenti universitari di laurea magistrale che hanno frequentato il corso di "Pedagogia sperimentale e valutazione scolastica". All'interno del corso, sono state proposte due attività per stimolare il CT degli studenti: la valutazione delle risorse educative aperte (OER) (obbligatoria e online) e la progettazione delle OER (facoltativa e in modalità blended). I saggi sono stati valutati sia da valutatori esperti, considerando sei sotto-dimensioni del CT, sia da un algoritmo che misura automaticamente diversi tipi di indicatori del linguaggio naturale. Abbiamo riscontrato un'affidabilità interna positiva e un accordo tra valutatori medio-alto. I livelli di CT degli studenti sono migliorati in modo significativo nel post-test. Tre indicatori del linguaggio naturale sono 5 correlati in modo significativo con il punteggio totale di CT: la lunghezza del corpus, la complessità della sintassi e la funzione di peso tf-idf (term frequency–inverse document frequency). I risultati raccolti durante questo dottorato hanno implicazioni sia teoriche che pratiche per la ricerca e la valutazione del CT. Da un punto di vista teorico, questa tesi mostra sovrapposizioni inesplorate tra diverse tradizioni, prospettive e metodi di studio del CT. Questi punti di contatto potrebbero costituire la base per un approccio interdisciplinare e la costruzione di una comprensione condivisa di CT. I metodi di valutazione automatica possono supportare l’uso di misure aperte per la valutazione del CT, specialmente nell'insegnamento online. Possono infatti facilitare i docenti e i ricercatori nell'affrontare la crescente presenza di dati linguistici prodotti all'interno di piattaforme educative (es. Learning Management Systems). A tal fine, è fondamentale sviluppare metodi automatici per la valutazione di grandi quantità di dati che sarebbe impossibile analizzare manualmente, fornendo agli insegnanti e ai valutatori un supporto per il monitoraggio e la valutazione delle competenze dimostrate online dagli studenti.The main goal of this PhD thesis is to test, through two empirical studies, the reliability of a method aimed at automatically assessing Critical Thinking (CT) manifestations in Higher Education students’ written texts. The empirical studies were based on a critical review aimed at proposing a new classification for systematising different CT definitions and their related theoretical approaches. The review also investigates the relationship between the different adopted CT definitions and CT assessment methods. The review highlights the need to focus on open-ended measures for CT assessment and to develop automatic tools based on Natural Language Processing (NLP) technique to overcome current limitations of open-ended measures, such as reliability and costs. Based on a rubric developed and implemented by the Center for Museum Studies – Roma Tre University (CDM) research group for the evaluation and analysis of CT levels within open-ended answers (Poce, 2017), a NLP prototype for the automatic measurement of CT indicators was designed. The first empirical study was carried out on a group of 66 university teachers. The study showed satisfactory reliability levels of the CT evaluation rubric, while the evaluation carried out by the prototype was not yet sufficiently reliable. The results were used to understand how and under what conditions the model works better. The second empirical investigation was aimed at understanding which NLP features are more associated with six CT sub-dimensions as assessed by human raters in essays written in the Italian language. The study used a corpus of 103 students’ pre-post essays who attended a Master's Degree module in “Experimental Education and School Assessment” to assess students' CT levels. Within the module, we proposed two activities to stimulate students' CT: Open Educational Resources (OERs) assessment (mandatory and online) and OERs design (optional and blended). The essays were assessed both by expert evaluators, considering six CT sub-dimensions, and by an algorithm that automatically calculates different kinds of NLP features. The study shows a positive internal reliability and a medium to high inter-coder agreement in expert evaluation. Students' CT levels improved significantly in the post-test. Three NLP indicators significantly correlate with CT total score: the Corpus Length, the Syntax Complexity, and an adapted measure of Term Frequency- Inverse Document Frequency. The results collected during this PhD have both theoretical and practical implications for CT research and assessment. From a theoretical perspective, this thesis shows unexplored similarities among different CT traditions, perspectives, and study methods. These similarities could be exploited to open up an interdisciplinary dialogue among experts and build up a shared understanding of CT. Automatic assessment methods can enhance the use of open-ended measures for CT assessment, especially in online teaching. Indeed, they can support teachers and researchers to deal with the growing presence of linguistic data produced within educational 4 platforms. To this end, it is pivotal to develop automatic methods for the evaluation of large amounts of data which would be impossible to analyse manually, providing teachers an

Building intersubjectivity in blended problem-solving tasks

This paper aims to shed light on the process of building intersubjectivity between student-student dyads in a blended educational context. Three girls and five boys, 17 to 18 years old, participated in two types of problem-solving tasks. They formed four dyads and were required to negotiate aloud what to post in a web-forum. Dyads were video recorded, with eight sessions in total. The same pairs participated in both tasks. We are interested in understanding how the intersubjective processes were affected by the tasks and by the dyads. The two tasks differ concerning the structure of the problems. The first task was based on two short papers – one pro and other con – referring to a problem close to students' real life: the use of digital devices in class. The second problem was based on perspective-taking: dyads were required to imagine “How would the school of the future look in 20 years.” Data were analysed through a purpose-built codebook, comprising five macro-categories and 21 subcategories. Altogether, our results indicate an effect of both the type of task and of dyads' specific style of interaction. Nevertheless, a five-step process featuring intersubjectivity was found. Practical implications for teachers and educators are highlighted

Proceedings ICPW'07: 2nd International Conference on the Pragmatic Web, 22-23 Oct. 2007, Tilburg: NL

Proceedings ICPW'07: 2nd International Conference on the Pragmatic Web, 22-23 Oct. 2007, Tilburg: N

Inservice teachers\u27 implementation of the Science Writing Heuristic as a tool for professional growth

According to the National Science Standards, science educators have been calling for the inclusion of inquiry-based approaches in science classrooms as a reform for science instruction. Teachers\u27 reluctance to implement inquiry-based approaches may be attributed to the different interpretations of the underpinning theory. Any efforts to help teachers implement effective teaching approaches, therefore, should not target merely additive skills; rather, such efforts should foster fundamental changes of beliefs, knowledge, and habits of practice that teachers deeply hold.;This study is a part of a bigger project known as the Science Writing Heuristic (SWH) Partnership Professional Development Project, conducted at Iowa State University in association with the Iowa Department of Education to help improve science teaching. The aim of the project is to help in service science teachers understand and apply a student-oriented instructional approach, using the SWH. The framework of the SWH emphasizes the role of classroom dialogical discussion in students\u27 learning. The role of the teacher when implementing constructivist approaches represented in a student-oriented instruction, differs markedly from traditional approaches, and hence there is a need to study the changes in teaching practice when using this specific tool.;The methodological framework of this study combined qualitative and quantitative methods. Interpretative case studies for 18 science teachers are presented. A triangulation strategy was used to provide support for the qualitative findings using three different data collection approaches: observation, questionnaire, and interviews. Quantitative data in the form of student performance on higher order conceptual questions and total test scores were collected.;Teachers varied in their implementation of the SWH. Three criteria were constructed to define teachers\u27 levels of implementation: epistemological belief, pedagogical practice, and teacher content knowledge. Different components were included within each criterion, with three scales used to rank each teacher\u27s implementation within each criterion.;The study is pointing to the importance of assisting science teachers with effective pedagogical strategies if there is to be an impact on students\u27 performance on tests. This project was undertaken with partnerships between schools, Area Educational Agencies, and the University. Such partnerships need to be built to promote successful change to science teaching

Towards an authentic argumentation literacy test

A central goal of education is to improve argumentation literacy. How do we know how well this goal is achieved? Can we measure argumentation literacy? The present study is a preliminary step towards measuring the efficacy of education with regards to argumentation literacy. Tests currently in use to determine critical thinking skills are often similar to IQ-tests in that they predominantly measure logical and mathematical abilities. Thus, they may not measure the various other skills required in understanding authentic argumentation. To identify the elements of argumentation literacy, this exploratory study begins by surveying introductory textbooks within argumentation theory, critical thinking, and rhetoric. Eight main abilities have been identified. Then, the study outlines an Argumentation Literacy Test that would comprise these abilities suggested by the literature. Finally, the study presents results from a pilot of a version of such a test and discusses needs for further development

What Do We Know About the World? Rhetorical and Argumentative Perspectives

his book consists of selected papers delivered at “First International Conference on Rhetoric in Croatia: the Days of Ivo Škarić” in May, 2012, and subsequently revised for publication. Through a variety of different routes, the papers explore the role of rhetoric and argumentation in various types of public discourse and present interdisciplinary work connecting linguists, phoneticians, philosophers, law experts and communication scientists in the common ground of rhetoric and argumentation

Human communicative dialogic practices in content and language-based educational interactions with technology

Anthropology has paid close attention to linguistic interaction andforeign language learning. This study approaches the use of dialogic learning that has strong roots in anthropology, sociology and applied linguistics. Following earlier research on teaching content in English in different subjects of teacher training through dialogic interaction, this paper focuses on the specific use of digital technology as a support for analysis through teacher-instructor interaction, as a means of social interaction, and as the creation of self-designed materials for content language integrated learning. Data is obtained from group discussions using digital content. Results indicate that technology represents a shifting pedagogical paradigm to enhance interaction in the Zone of Proximal Development and promote learning content in English through self and peer reflection.The findings of this study may serve as a monitor for content and language integrated learning educators aiming to enhance principled communicative insightful teachingwith technologyMinisterio de Ciencia e Innovació

Managing A Discursive Journey for Classroom Inquiry: Examination of a Teacher’s Discursive Moves

This study presents an analysis of teacher discursive moves (TDMs) that aid students in altering their thinking and talking systems. The participants were a science who handled the immersion inquiry activities. The primary data source was the video recorded in the classroom. This video-based data was analyzed through systematic observation in two phases comprising coding and counting to reveal the mechanics of the discursive journey. Three assertions were made for the dynamics of the discursive journey. First, the teacher enacted a wide range of TDMs incorporating dialogically/monologically oriented, simplified (observe-compare-predict), and rather sophisticated moves (challenging). The challenging moves were the most featured among all analytical TDMs. Second, once higher-order categories were composed by collapsing subcategories of the displayed TDMs, the communicating-framing moves were the most prominent performed moves. Lastly, the teacher created an argumentative atmosphere in which the students had the right to evaluate and judge their classmates and teacher's utterances that modified the epistemic and social authority of the discursive journey. Finally, educational recommendations are offered in the context of teachers noticing the mechanics and dynamics of the discourse journey