Deliverable 4.4 - Review effective feedback and formative assessment in e-learning

This systematic review is written in the frame of a European Union funded project called Supporting Lifelong learning with ICT Inquiry-Based Education (LIBE). Through this project, an e-learning environment will be developed for young low educational achievers (aged 16-24). A crucial part in every educational intervention is assessment and feedback. Hence, to contribute to this project, this study aims to review literature regarding computer based feedback and formative assessment in e-learning. Three different approaches are considered as formative assessment, namely: ‘diagnostic testing’ (DT), ‘data based decision making’ (DBDM) and ‘assessment for learning’ (AFL). The search and selection procedure for this review resulted in 19 studies about the effects of feedback and formative assessment in elearning. Four interrelated themes emerged within this body of literature: (1) the effect on achievement, (2) the effect on motivation, (3) self-regulated learning and (4) the effect of learner characteristics. Results are elaborated around those themes and implications for the LIBE project are drawn

The use of feedback in web-based instruction : achievement, feedback study time, and efficiency

The purpose of this research study is to compare the effects of an instructional treatment that presented adaptive feedback based on students’ perceptions about their answer correctness with a nonadaptive treatment on student performance, feedback study time, and lesson efficiency in a computer-based environment. Because of advances in technology, the instruction was delivered in a Web-based environment. Two versions of the Web-based lesson were designed. The lesson consisted of a pretest, a tutorial, and a posttest. The pre- and posttest were similar in that they tested students’ ability to classify defined concepts, a higher cognitive task. The tutorial presented instructional text with inserted verbal information questions. Undergraduate volunteers enrolled in entry level education courses in the Watson School of Education were randomly assigned to one of the two treatment groups. One group received varied feedback information based upon the combined assessment of response correctness and the student’s response certitude. The other group received feedback information that did not vary. Results indicate that the effects of adaptive feedback were not significantly different from the effects of nonadaptive feedback on concept learning. In the adaptive group, high certitude wrongs, low certitude corrects, and low certitude wrong responses resulted in higher feedback study time than high certitude correct responses. High certitude significantly correlated with fine discrimination errors in concept learning. In terms of feedback efficiency, adaptive feedback was significantly more efficient than nonadaptive feedback; however, for overall lesson efficiency, there were no significant differences between the two treatment groups. These results are discussed in terms of cost-benefit implications for the design of effective Web-based instruction. Implications for future research are discussed in reference to results of this study and past research

Computer-based assessments of high school mathematics in Myanmar

Computer-based assessment (CBA) is a versatile educational tool in the twenty-first century. It offers many new opportunities for innovation in educational assessment through rich new assessment tasks and improves the learning progress of students. Educators have begun to benefit from CBA as it reduces the timing in reporting scores and increases assessment efficiency that enables immediate feedback. However, assessments in Myanmar high schools are mainly in paper-and-pencil test (PPT). Due to the large class and a limited number of teachers, this regular assessment causes them more workload in administering tests and providing scores and feedback. As a result, teachers spend most of their time assessing, scoring, and providing feedback. These activities negatively affect allocated hours of teaching and learning, which, in turn, are ineffective on the learning progress of students. The aims are: (1) to examine high schools in Myanmar whether computer-based assessment, this is, linear-online test (LOT) and computer-adaptive test (CAT) is more effective test mode than PPT as a formative assessment for the learning progress of students; (2) to identify contextual scales that influence students learning progress due to computer-based assessments and regular paper-and-pencil test. Of intervention design with explanatory mix-method, this study applied counter-balanced quasi-experimental research to compare effects of computer-based and paper-based assessments in terms of the achievement improvement of students. This study conducted surveys among students and teachers, followed by semi-structured interviews from five high schools in Yangon Region, Myanmar. Students from these high schools took the computer-based test and paper-based format as formative assessments. For constructing an online formative assessment test, both the Concerto platform and online Monkey Survey were applied, and through the Rasch Dichotomous model, items are assembled in the item-banks of the computer-based assessment. The analysis of variance (ANOVA) was applied to examine the effect of the test modes. The results of the computer-based test mode showed that students who received their specific scores and feedback immediately improved their mathematics achievement significantly higher than those who received the delayed score and feedback from the paper-based test mode. Structural equation modelling is used to analyse the structural relationship between measured variables. This model shows that positive attitude of students towards either computer-based or paper-based is the ultimate mechanism for more remarkable achievement. Although the two test delivery media may affect different groups of participant students in different ways, this concerns equity issues. For example, findings showed that educational background of parents, students’ gender, and attitude towards paper-based assessment or computer-based assessment could influence or affect the achievement of students. In addition, the specific practices of teachers towards formative assessment influence the attitude of students. The attitude of teachers concerning computers and technology affects the attitude of students towards innovative assessment formats. As shown by hierarchical linear modelling, the cross-level interaction effect from the teacher-level on the slope of the attitude of students towards paper-based assessment and their achievement improvement is specific practices of formative assessment. In addition, this study showed that the attitude of students to information and communications technology (ICT) and the attitude of teachers to formative assessment and ICT directly affect the achievement improvement of students. This thesis reveals significant gaps in understanding concerning formative assessment in Myanmar and contributes to the theoretical, practical, and methodological implications in mathematics assessment and learning. In addition, the findings provide (albeit for Myanmar educational systems) a practical resource for assessment developers and a useful framework for the discussion of innovative assessment formats and use in computer-based settings.Thesis (Ph.D.) -- University of Adelaide, School of Education, 202

The Effect of Force Feedback on Student Reasoning about Gravity, Mass, Force and Motion

The purpose of this study was to examine whether force feedback within a computer simulation had an effect on reasoning by fifth grade students about gravity, mass, force, and motion, concepts which can be difficult for learners to grasp. Few studies have been done on cognitive learning and haptic feedback, particularly with young learners, but there is an extensive base of literature on children\u27s conceptions of science and a number of studies focus specifically on children\u27s conceptions of force and motion. This case study used a computer-based paddleball simulation with guided inquiry as the primary stimulus. Within the simulation, the learner could adjust the mass of the ball and the gravitational force. The experimental group used the simulation with visual and force feedback; the control group used the simulation with visual feedback but without force feedback. The proposition was that there would be differences in reasoning between the experimental and control groups, with force feedback being helpful with concepts that are more obvious when felt. Participants were 34 fifth-grade students from three schools. Students completed a modal (visual, auditory, and haptic) learning preference assessment and a pretest. The sessions, including participant experimentation and interviews, were audio recorded and observed. The interviews were followed by a written posttest. These data were analyzed to determine whether there were differences based on treatment, learning style, demographics, prior gaming experience, force feedback experience, or prior knowledge. Work with the simulation, regardless of group, was found to increase students\u27 understanding of key concepts. The experimental group appeared to benefit from the supplementary help that force feedback provided. Those in the experimental group scored higher on the posttest than those in the control group. The greatest difference between mean group scores was on a question concerning the effects of increased gravitational force

Automated Writing Evaluation for non-native speaker English academic writing: The case of IADE and its formative feedback

This dissertation presents an innovative approach to the development and empirical evaluation of Automated Writing Evaluation (AWE) technology used for teaching and learning. It introduces IADE (Intelligent Academic Discourse Evaluator), a new web-based AWE program that analyzes research article Introduction sections and generates immediate, individualized, discipline-specific feedback. The major purpose of the dissertation was to implement IADE as a formative assessment tool complementing L2 graduate-level academic writing instruction and to investigate the effectiveness and appropriateness of its automated evaluation and feedback. To achieve this goal, the study sought evidence of IADE\u27s Language Learning Potential, Meaning Focus, Learner Fit, and Impact qualities outlined in Chapelle\u27s (2001) CALL evaluation conceptual framework. A mixed-methods approach with a concurrent transformative strategy was employed. Quantitative data consisted of Likert-scale, yes/no, and open-ended survey responses; automated and human scores for first and last drafts; pre-/post test scores; and frequency counts for draft submission and for access to IADE\u27s Help Options. Qualitative data contained students\u27 first and last drafts as well as transcripts of think-aloud protocols and Camtasia computer screen recordings, observations, and semi-structured interviews. The findings indicate that IADE can be considered an effective formative assessment tool suitable for implementation in the targeted instructional context. Its effectiveness was a result of combined strengths of its Language Learning Potential, Meaning Focus, Learner Fit, and Impact qualities, which were all enhanced by the program\u27s automated feedback. The strength of Language Learning Potential was supported by evidence of noticing of and focus on discourse form, improved rhetorical quality of writing, increased learning gains, and relative helpfulness of practice and modified interaction. Learners\u27 focus on the functional meaning of discourse and construction of such meaning served as evidence of strong Meaning Focus. IADE\u27s automated feedback characteristics and Help Options were appropriate for targeted learners, which speaks of adequate Learner Fit. Finally, despite some negative effects caused by IADE\u27s numerical feedback, overall Impact, exerted at affective, intrinsic, pragmatic, and cognitive levels, was found to be positive due to the color-coded type of feedback. The results of this study provide valuable empirical knowledge to the areas of L2 academic writing, AWE, formative assessment, and I/CALL. They have important practical and theoretical implications and are informative for future research as well as for the design and application of new learning technologies

A systemic functional perspective on automated writing evaluation: formative feedback on causal discourse

Making explanations is a very important communicative function in academic literacy; several disciplines including science are dominated by causal explanations (Mohan & Slater, 2004; Slater, 2004; Wellington & Osborne, 2001). For academic success, students need to write about causes and effects well with the help of their instructors, which means that formative assessment of causal discourse is necessary (Slater & Mohan, 2010). However, manual evaluation of causal discourse is time-consuming and impractical for writing instructors. For this reason, automated evaluation of causal discourse, which current automated writing evaluation (AWE) systems cannot perform, is required. Addressing these needs, this dissertation aimed to develop an automated causal discourse evaluation tool (ACDET) and empirically evaluate learners’ causal discourse development with ACDET in academic writing classes. ACDET was developed using three approaches: a functional linguistic approach, a hybrid natural language processing approach combining rule-based and statistical approaches, and a pedagogical approach. The linguistic approach helped identify causal discourse features by analyzing a small corpus of texts about causes and effects of economic events. ACDET detects seven types of causal discourse features and generates formative feedback based on them: causal conjunctions, causal adverbs, causal prepositions, causal verbs, causal adjectives, and causal nouns. The natural language processing approach allowed for assigning part-of-speech tags to sentences and words and creating hand-coded rules for the detection of causal discourse features. The pedagogical approach determined feedback features of ACDET, and it was informed by the theoretical perspectives of the Interaction Hypothesis and Systemic Functional Linguistics and findings of research on causal discourse development. Causal discourse development with ACDET was empirically evaluated through a qualitative study in which four research questions investigated two criteria of computer-assisted language learning evaluation framework: language learning potential (i.e., focus on causal discourse form, interactional modifications, and causal discourse development) and focus on causal meaning. Participants of the study were 32 English as a second language learners who were students in two academic writing classes. Data consisted of pre- and post-tests, ACDET’s text-level feedback reports, cause-and-effect assignment drafts, screen capturing recordings, semi-structured interviews, and questionnaires. The findings indicate language learning potential of ACDET: ACDET drew learners’ attention to causal discourse form and created opportunities for interactional modifications, however, resulted in limited causal discourse development. Findings also reveal that ACDET drew learners’ attention to causal meaning. This study is an important attempt in the field of AWE to analyze meaning in written discourse automatically and provide causal discourse specific feedback. The fact that empirical evaluation of ACDET was based on process-oriented data revealing how students used ACDET in class is noteworthy. The findings of this study have important implications for the refinement of ACDET, the development of AWE systems, and research on causal discourse development

Informing Writing: The Benefits of Formative Assessment

Examines whether classroom-based formative writing assessment - designed to provide students with feedback and modified instruction as needed - improves student writing and how teachers can improve such assessment. Suggests best practices

Formative peer assessment in a CSCL environment

In this case study our aim was to gain more insight in the possibilities of qualitative formative peer assessment in a computer supported collaborative learning (CSCL) environment. An approach was chosen in which peer assessment was operationalised in assessment assignments and assessment tools that were embedded in the course material. The course concerned a higher education case-based virtual seminar, in which students were asked to conduct research and write a report in small multidisciplinary teams. The assessment assignments contained the discussion of assessment criteria, the assessment of a group report of a fellow group, and writing an assessment report. A list of feedback rules was one of the assessment tools. A qualitative oriented study was conducted, focussing on the attitude of students towards peer assessment and practical use of peer assessment assignments and tools. Results showed that students’ attitude towards peer assessment was positive and that assessment assignments had added value. However, not all students fulfilled all assessment assignments. Recommendations for implementation of peer assessment in CSCL environments as well as suggestions for future research are discussed

Peer assessment as collaborative learning

Peer assessment is an important component of a more participatory culture of learning. The articles collected in this special issue constitute a representative kaleidoscope of current research on peer assessment. In this commentary, we argue that research on peer assessment is currently in a stage of adolescence, grappling with the developmental tasks of identity formation and affiliation. Identity formation may be achieved by efforts towards a shared terminology and joint theory building, whereas affiliation may be reached by a more systematic consideration of research in related fields. To reach identity formation and affiliation, preliminary ideas for a cognitively toned, process-related model of peer assessment and links to related research fields, especially to research on collaborative learning, are presented