A Framework for Assessing Reflective Writing Produced Within the Context of Computer Science Education

Reflective writing is known to be an effective activity to increase students' learning. However, there is limited literature in reflective writing assessment criteria in the context of computer science (CS) education. In this paper, we aim to explore a meaningful reflective writing assessment characteristics. That has been used to assess reflective text by CS educators. This paper has two contributions: (a) we developed a Reflective Writing Framework (RWF) for the main criteria has been used to assess reflective text in CS education from the findings of a semi-structure questionnaire; (b) the RWF was tested empirically using a pilot test of the manual annotation used to modify the framework. This analysis resulted in an inter-rater reliability of 0.78 being achieved. The overall goal of this research is to develop a Learning Analytics (LA) tool which can automatically detect the categories of the RWF present in a text to assess the student authors’ reflective writing in relation to CS

Evaluating an Automated Analysis using Machine Learning and Natural Language Processing Approaches to Classify Computer Science Students' Reflective Writing

Reflection writing is a common practice in higher education. However, manual analysis of written reflections is time-consuming. This study presents an automated analysis of reflective writing to analyze reflective writing in CS education based on conceptual Reflective Writing Framework (RWF) and application of natural language processing and machine learning algorithm. This paper investigates two groups of features extraction (n-grams and PoS n-grams) and random forest (RF) algorithm that utilize such features to detect the presence or absence of the seven indicators (description of an experience, understandings, feelings, reasoning, perspective, new learning, and future action). The automated analysis of reflective writing is evaluated based on 74 CS student essays (1113 sentences) that are from the final year project reports in CS’s students. Results showed the seven indicators can be reliably distinguished by their features and these indicators can be used in an automated reflective writing analysis for determining the level of students’ reflective writing. Finally, we consider the implications of how the conceptualization of refection quality and providing individualized learning support to students in order to help them develop reflective skills

Validating the Reflective Writing Framework (RWF) for assessing reflective writing in computer science education through manual annotation

The accuracy of a framework for annotating reflective writing can be increased through the evaluation and revision of the annotation scheme to ensure the reliability and validity of the framework. To our knowledge, there is a lack of literature related to the accuracy of any reflective writing framework in Computer Science (CS) education. This paper describes a manual annotation scheme, applied during four pilot studies, to validate the authors’ novel Reflective Writing Framework (RWF) for CS education. The results show, through the pilot studies, that the accuracy of Inter-Rater Reliability (IRR) increases from 0.5 to 0.8, which was substantial and close to an almost perfect agreement. This paper contributes to CS education through the reliability and validity of the RWF that can be potentially used for generating an Intelligent Tutoring Systems (ITS) using machine learning algorithms