Defining and Assessing Critical Thinking: toward an automatic analysis of HiEd students’ written texts

L'obiettivo principale di questa tesi di dottorato è testare, attraverso due studi empirici, l'affidabilità di un metodo volto a valutare automaticamente le manifestazioni del Pensiero Critico (CT) nei testi scritti da studenti universitari. Gli studi empirici si sono basati su una review critica della letteratura volta a proporre una nuova classificazione per sistematizzare le diverse definizioni di CT e i relativi approcci teorici. La review esamina anche la relazione tra le diverse definizioni di CT e i relativi metodi di valutazione. Dai risultati emerge la necessità di concentrarsi su misure aperte per la valutazione del CT e di sviluppare strumenti automatici basati su tecniche di elaborazione del linguaggio naturale (NLP) per superare i limiti attuali delle misure aperte, come l’attendibilità e i costi di scoring. Sulla base di una rubrica sviluppata e implementata dal gruppo di ricerca del Centro di Didattica Museale – Università di Roma Tre (CDM) per la valutazione e l'analisi dei livelli di CT all'interno di risposte aperte (Poce, 2017), è stato progettato un prototipo per la misurazione automatica di alcuni indicatori di CT. Il primo studio empirico condotto su un gruppo di 66 docenti universitari mostra livelli di affidabilità soddisfacenti della rubrica di valutazione, mentre la valutazione effettuata dal prototipo non era sufficientemente attendibile. I risultati di questa sperimentazione sono stati utilizzati per capire come e in quali condizioni il modello funziona meglio. La seconda indagine empirica era volta a capire quali indicatori del linguaggio naturale sono maggiormente associati a sei sottodimensioni del CT, valutate da esperti in saggi scritti in lingua italiana. Lo studio ha utilizzato un corpus di 103 saggi pre-post di studenti universitari di laurea magistrale che hanno frequentato il corso di "Pedagogia sperimentale e valutazione scolastica". All'interno del corso, sono state proposte due attività per stimolare il CT degli studenti: la valutazione delle risorse educative aperte (OER) (obbligatoria e online) e la progettazione delle OER (facoltativa e in modalità blended). I saggi sono stati valutati sia da valutatori esperti, considerando sei sotto-dimensioni del CT, sia da un algoritmo che misura automaticamente diversi tipi di indicatori del linguaggio naturale. Abbiamo riscontrato un'affidabilità interna positiva e un accordo tra valutatori medio-alto. I livelli di CT degli studenti sono migliorati in modo significativo nel post-test. Tre indicatori del linguaggio naturale sono 5 correlati in modo significativo con il punteggio totale di CT: la lunghezza del corpus, la complessità della sintassi e la funzione di peso tf-idf (term frequency–inverse document frequency). I risultati raccolti durante questo dottorato hanno implicazioni sia teoriche che pratiche per la ricerca e la valutazione del CT. Da un punto di vista teorico, questa tesi mostra sovrapposizioni inesplorate tra diverse tradizioni, prospettive e metodi di studio del CT. Questi punti di contatto potrebbero costituire la base per un approccio interdisciplinare e la costruzione di una comprensione condivisa di CT. I metodi di valutazione automatica possono supportare l’uso di misure aperte per la valutazione del CT, specialmente nell'insegnamento online. Possono infatti facilitare i docenti e i ricercatori nell'affrontare la crescente presenza di dati linguistici prodotti all'interno di piattaforme educative (es. Learning Management Systems). A tal fine, è fondamentale sviluppare metodi automatici per la valutazione di grandi quantità di dati che sarebbe impossibile analizzare manualmente, fornendo agli insegnanti e ai valutatori un supporto per il monitoraggio e la valutazione delle competenze dimostrate online dagli studenti.The main goal of this PhD thesis is to test, through two empirical studies, the reliability of a method aimed at automatically assessing Critical Thinking (CT) manifestations in Higher Education students’ written texts. The empirical studies were based on a critical review aimed at proposing a new classification for systematising different CT definitions and their related theoretical approaches. The review also investigates the relationship between the different adopted CT definitions and CT assessment methods. The review highlights the need to focus on open-ended measures for CT assessment and to develop automatic tools based on Natural Language Processing (NLP) technique to overcome current limitations of open-ended measures, such as reliability and costs. Based on a rubric developed and implemented by the Center for Museum Studies – Roma Tre University (CDM) research group for the evaluation and analysis of CT levels within open-ended answers (Poce, 2017), a NLP prototype for the automatic measurement of CT indicators was designed. The first empirical study was carried out on a group of 66 university teachers. The study showed satisfactory reliability levels of the CT evaluation rubric, while the evaluation carried out by the prototype was not yet sufficiently reliable. The results were used to understand how and under what conditions the model works better. The second empirical investigation was aimed at understanding which NLP features are more associated with six CT sub-dimensions as assessed by human raters in essays written in the Italian language. The study used a corpus of 103 students’ pre-post essays who attended a Master's Degree module in “Experimental Education and School Assessment” to assess students' CT levels. Within the module, we proposed two activities to stimulate students' CT: Open Educational Resources (OERs) assessment (mandatory and online) and OERs design (optional and blended). The essays were assessed both by expert evaluators, considering six CT sub-dimensions, and by an algorithm that automatically calculates different kinds of NLP features. The study shows a positive internal reliability and a medium to high inter-coder agreement in expert evaluation. Students' CT levels improved significantly in the post-test. Three NLP indicators significantly correlate with CT total score: the Corpus Length, the Syntax Complexity, and an adapted measure of Term Frequency- Inverse Document Frequency. The results collected during this PhD have both theoretical and practical implications for CT research and assessment. From a theoretical perspective, this thesis shows unexplored similarities among different CT traditions, perspectives, and study methods. These similarities could be exploited to open up an interdisciplinary dialogue among experts and build up a shared understanding of CT. Automatic assessment methods can enhance the use of open-ended measures for CT assessment, especially in online teaching. Indeed, they can support teachers and researchers to deal with the growing presence of linguistic data produced within educational 4 platforms. To this end, it is pivotal to develop automatic methods for the evaluation of large amounts of data which would be impossible to analyse manually, providing teachers an

Creative writing and Critical Thinking Enhancement at Higher Education

[EN] Currently, educational policymakers identify Critical Thinking (CT) as an essential aspect of progress and knowledge growth in any field and in the broad society. Peer interactions and individual writing are helpful pedagogical strategies for CT development that could be enhanced by the use of technologies (Guiller, Durndell, & Ross, 2008). Starting from the above-mentioned evidence, a university module was designed by combining collaborative and creative writing with the critical analysis of literary texts and the fruition of figurative arts. 123 students worked in groups and their CT level was assessed at different times of the course. Most of the students’ groups showed an increase in their CT level whilst a few did not. The difficulties in the management of group dynamics corresponded to the decreasing CT level. On the contrary, groups able to organize their work improved their CT throughout the course. Additionally, students’ work modes had an impact on their performance at different CT tasks. The research results could be used to improve university course design for CT education.Poce, A.; Amenduni, F. (2019). Creative writing and Critical Thinking Enhancement at Higher Education. En HEAD'19. 5th International Conference on Higher Education Advances. Editorial Universitat Politècnica de València. 459-467. https://doi.org/10.4995/HEAD19.2019.9221OCS45946

Could vocational education benefit from augmented reality and hypervideo technologies? An exploratory interview study

Context: This study investigates the perspective of vocational educators on the possibility of adopting augmented reality (AR) and hypervideo (HV) technologies to support their teaching practice. Vocational education and training (VET) is particularly concerned with the learning of resources (knowledge, skills and attitudes) that are immediately transposable into conduct and procedures in the workplace. AR and HV can provide means to answer this requirement, but both technological solutions are still not so diffused in VET. The purpose of this study is to inquire into the perception of educators on the main advantages and disadvantages of using AR and HV to support teaching-and-learning. Methods: A semi-structured interview protocol has been proposed to 73 teachers, intercompany trainers and in-company trainers in 10 professions (at least two per category within each profession). The interview was organized in two main steps: A need analysis, in which the most important and difficult operative skills are identified for the interviewee\u27s profession; and a discussion of advantages and disadvantages of AR and HV. Content analysis was applied to the interview transcriptions. Results: The results show that the main advantages reported in the literature for the two technologies – such as the ability to switch between 2D and 3D and carry out simulations – are also found in the VET context by educators. For HV the main technical advantages (such as the use of active points, and non-linear navigation of video content) were autonomously recognised, while the potential of the instrument to support reflection has not been clearly identified. Conclusions: AR and HV are considered as tools able to support apprentices\u27 procedural learning especially with regard to the operational skills which were judged by the educators to be most relevant for VET. (DIPF/Orig.

Rethinking high-tech tools for cultural heritage education. In-training teachers’ reflections on Musetech web app

MuseTech is a web app implemented in the frame of DICHE Erasmus + Project. The aim was to train teachers on new methods to develop 4Cs through the use of cultural heritage education and technologies. In this paper we investigate how much in-training teachers are interested in 3.0 web technologies proposed in the MuseTech app, and which kind of opinions and expectations they have about the use of 3.0 technologies for cultural heritage education. Teachers showed a strong interest for high tech cultural heritage educational resources, especially because of the following affordances: ubiquity, multi-sensoriality experience, and immersivity. Through a self-assessment of their skills they declared to develop their research and design skills while stimulating their IT, creativity and critical thinking skills. The MuseTech case has provided teachers with the opportunity to discuss their ideas with peers and to employ technologically tools. Ripensare la tecnologia nell’educazione al patrimonio culturale. Riflessioni di futuri insegnanti sulla web app MuseTechMuseTech è un’applicazione web realizzata nell’ambito del Progetto DICHE Erasmus +, che permette agli insegnanti di essere formati sui nuovi metodi per sviluppare capacità comunicative, di collaborazione, di creatività e di pensiero critico (4Cs) attraverso l’educazione al patrimonio culturale e l’uso delle tecnologie. Abbiamo indagato quanto gli insegnanti in formazione siano interessati alle tecnologie web 3.0 proposte nell’app MuseTech, e che tipo di opinioni e aspettative abbiano circa l’uso di tali tecnologie per l’educazione al patrimonio culturale. Gli insegnanti hanno mostrato un grande interesse specialmente per le caratteristiche di: ubiquità, multi-sensorialità e immersività. Autovalutando le proprie competenze, hanno inoltre dichiarato di aver sviluppato le proprie capacità di ricerca e progettazione stimolando al tempo stesso quelle di IT, creatività e pensiero critico. MuseTech ha inoltre fornito agli insegnanti l’opportunità di discutere le proprie idee con i pari e di utilizzare strumenti tecnologici

Using different learning methodologies and tools to exploit the educational impact of a University Art Collection: a pilot phase at Roma Tre University (IT)

The Centre for Museum Studies based at University Roma Tre (IT) has designed and implemented a museum educational programme aimed at promoting the “Tito Rossini” contemporary art collection donated by the artist in 2017 to the Department of Education. The educational programme foresees the use of innovative and digital education methodologies and tools (e.g. Digital Storytelling, Augmented Reality and Gamification) and is addressed to university and external users. A pilot study with university students was carried out in order to verify the achievement of some objectives of the programme designed for the exhibition. This experience addresses the need for personalised educational museum paths in view of promoting social inclusion and transverse skills, focusing on critical thinking in particular

Professional development of in-training museum educators: an experience of curriculum improvement in time of a pandemic

[EN] The present paper aims to illustrate the reorganization of two post-graduate courses, “Museum Education. Theoretical aspects” and “Advanced Studies in Museum Education” promoted by CDM (Center for Museum Studies) - Dept. of Education at Roma Tre University, carried out during the Covid-19 pandemic, and to analyze the education strategies adopted in terms of museum professionals development to face the Covid-19 museum and universities 2020 Italian lockdown.The results emerging from the quantitative evaluation of the module “Museum and Social Networks”, taking into consideration the activities and digital tools proposed, show the efficacy of the courses reorganization, in terms of transverse and professional skills development in university students, critical thinking and collaboration in particular. Moreover, the data analysis give useful indications in term of university online lectures, laboratory activities and practices in e-learning mode, evaluation tools and methodologies aimed at solliciting professional development of in-training museum educators in university learning context.Poce, A.; Valente, M.; Re, MR.; Amenduni, F.; De Medio, C. (2021). Professional development of in-training museum educators: an experience of curriculum improvement in time of a pandemic. En 7th International Conference on Higher Education Advances (HEAd'21). Editorial Universitat Politècnica de València. 1377-1384. https://doi.org/10.4995/HEAd21.2021.12881OCS1377138

Supporting Virtual Mobility Skills in a MOOC: Preliminary Results

In the last years, the concept of Virtual Mobility has receiving a growing attention from educational policy makers and institutions, because it has the potential to make more accessible and effective students and teachers mobility in Higher Education. Virtual Mobility could be defined as institutional ICT-supported activities that trigger or facilitate international collaborative experiences in the context of teaching and/or learning. Despite the interest, there is still a few empirical researches regarding actual effectiveness of Virtual Mobility implementation and which technological solutions could be adopted. The present paper describes a research project aimed at designing an Open and Accessible Virtual Mobility Massive Open Online Course, by involving students and teachers from six European countries and higher education institutions. 716 participants completed and assessed the Open Virtual Mobility MOOC. Participants expressed a positive evaluation of different MOOCs features: (a) Badges; (b) Technical features; (c) Gamification. Four out of eight MOOCs obtained the highest evaluation: (a) Collaborative learning; (b) Autonomy-drive learning; (c) Open-mindedness; (d) Intercultural skills. Future research trajectories would be described

Could Vocational Education Benefit From Augmented Reality and Hypervideo Technologies? An Exploratory Interview Study

Context: This study investigates the perspective of vocational educators on the possibility of adopting augmented reality (AR) and hypervideo (HV) technologies to support their teaching practice. Vocational education and training (VET) is particularly concerned with the learning of resources (knowledge, skills and attitudes) that are immediately transposable into conduct and procedures in the workplace. AR and HV can provide means to answer this requirement, but both technological solutions are still not so diffused in VET. The purpose of this study is to inquire into the perception of educators on the main advantages and disadvantages of using AR and HV to support teaching-and-learning.  Methods: A semi-structured interview protocol has been proposed to 73 teachers, intercompany trainers and in-company trainers in 10 professions (at least two per category within each profession). The interview was organized in two main steps: A need analysis, in which the most important and difficult operative skills are identified for the interviewee's profession; and a discussion of advantages and disadvantages of AR and HV. Content analysis was applied to the interview transcriptions.  Results: The results show that the main advantages reported in the literature for the two technologies – such as the ability to switch between 2D and 3D and carry out simulations – are also found in the VET context by educators. For HV the main technical advantages (such as the use of active points, and non-linear navigation of video content) were autonomously recognised, while the potential of the instrument to support reflection has not been clearly identified.  Conclusions: AR and HV are considered as tools able to support apprentices' procedural learning especially with regard to the operational skills which were judged by the educators to be most relevant for VET.

A Virtual Reality Etruscan Museum Exhibition–Preliminary Results Of The Participants’ Experience

The current global health emergency has posed the need to reflect upon how to guarantee high standard of quality in 100% virtual exhibition. In this case study, we present one of the possible solutions to design a VR museum exhibition for educational purposes. The Centre for Museum Studies designed “The E-Trouria App”, a VR exhibition which is aimed at providing participants with personalised learning path based on an Etruscan museum collection in Rome. The App was designed by combining different pedagogical methods such as Digital Storytelling and Reflective Questioning. The goals of the research were to understand visitors' evaluation of their experience. 20 postgraduate students (F = 17; M = 3; Average age = 36 years) in Museum Education took part in the pre-pilot experimentation. Participants expressed very positive evaluation on the visit and their features (narratives, soundtrack and multimodality). Participants were emotionally engaged during the visit and the most reported emotions were “pleasure” and “wondering”. Future research steps are illustrated

Assessing Critical Thinking in Open-ended Answers: An Automatic Approach

The role of Higher Education (HE) is growingly acknowledged for the promotion of Critical Thinking (CT). Constructed-response tasks (CRT) are recognized to be necessary for the CT assessment, though they present problems related to scoring quality and cost (Ku, 2009). Researchers (Liu, Frankel, Roohr, 2014) have proposed using automated scoring to address the above concerns. The present work is aimed at comparing the features of different Natural Language Processing (NLP) techniques adopted to improve the reliability of a prototype designed to automatically assess six sub-skills of CT in CRT: use of language, argumentation, relevance, importance, critical evaluation and novelty (Poce, 2017). We will present the first (1.0) and the second (2.0) version of the CT prototype and their respective reliability results. Our research question is the following: Which level of reliability are shown respectively by the 1.0 and 2.0 automatic CT assessment prototype compared to expert human evaluation? Data collection is realized in two moments, to measure respectively the CT prototype 1.0 and 2.0 reliability from a total of 264 participants and 592 open-ended answers. Two human assessors rated all of these responses on each of the subskills on a scale of 1-5. Similarly, NLP approaches are adopted to compute a feature on each dimension. Quadratic Weighted Kappa and Pearson product-moment correlation were used to evaluate the between-human agreement and human-NLP agreement. Preliminary findings based on the first data set suggest adequate level of between-human rating agreement and a lower level human-NLP agreement (r .43 for the subscales of Relevance and Importance). We are continuing the analysis of the data collected in the 2nd step and expect to complete them in June 2020