Writing Assignments with a Metacognitive Component Enhance Learning in a Large Introductory Biology Course

Writing assignments, including note taking and written recall, should enhance retention of knowledge, whereas analytical writing tasks with metacognitive aspects should enhance higher-order thinking. In this study, we assessed how certain writing-intensive “interventions,” such as written exam corrections and peer-reviewed writing assignments using Calibrated Peer Review and including a metacognitive component, improve student learning. We designed and tested the possible benefits of these approaches using control and experimental variables across and between our three-section introductory biology course. Based on assessment, students who corrected exam questions showed significant improvement on postexam assessment compared with their nonparticipating peers. Differences were also observed between students participating in written and discussion-based exercises. Students with low ACT scores benefited equally from written and discussion-based exam corrections, whereas students with midrange to high ACT scores benefited more from written than discussion-based exam corrections. Students scored higher on topics learned via peer-reviewed writing assignments relative to learning in an active classroom discussion or traditional lecture. However, students with low ACT scores (17–23) did not show the same benefit from peer-reviewed written essays as the other students. These changes offer significant student learning benefits with minimal additional effort by the instructors

Selecting Effective Examples to Train Students for Peer Review of Open‐Ended Problem Solutions

Background Students conducting peer review on authentic artifacts require training. In the training studied here, individual students reviewed (score and provide feedback on) a randomly selected prototypical solution to a problem. Afterwards, they are shown a side-by-side comparison of their review and an expert’s review, along with prompts to reflect on the differences and similarities. Individuals were then assigned a peer team’s solution to review. Purpose This paper explores how the characteristics of five different prototypical solutions used in training (and their associated expert evaluations) impacted students’ abilities to score peer teams’ solutions. Design/Method An expert rater scored the prototypical solutions and 147 student teams’ solutions that were peer reviewed using an eight item rubric. Differences between the scores assigned by the expert and a student to a prototypical solution and an actual team solution were used to compute a measure of the student’s improvement as a peer reviewer from training to actual peer review. ANOVA testing with Tukey’s post-hoc analysis was done to identify statistical differences in improvement based on the prototypical solutions students saw during the training phase. Results Statistically significant differences were found in the amount of error a student made during peer review between high and low quality prototypical solutions seen by students during training. Specifically, a lower quality training solution (and associated expert evaluation) resulted in more accurate scoring during peer review. Conclusions While students typically ask to see exemplars of “good solutions”, this research suggests that there is likely greater value, for the purpose of preparing students to score peers’ solutions, in students seeing a low-quality solution and its corresponding expert review

Toward Informative Assessment and a Culture of Evidence

Examines how campuses in the Strengthening Pre-collegiate Education in Community Colleges initiative combined traditional and innovative measures of student performance such as "think-aloud" protocol and pre-post testing to improve teaching and learning

RATE TO MEASURE MATHEMATICS TEACHING: USING THE MANY-FACET RASCH MODELING TO REEVALUATE THE MATHEMATICS CLASSROOM OBSERVATION PROTOCOL FOR PRACTICES (MCOP2)

Rater-mediated classroom observation protocols are increasingly being used for teaching performance assessments, which makes identifying and controlling for various rater effects a central issue to ensure the rating quality. A series of validation studies under the classical test theory framework, including content validity, interrater reliability, and structure analysis, have been completed for the 16-item Mathematics Classroom Observation Protocol for Practices (MCOP2). However, the MCOP2 data have never been investigated under the Rasch framework. Due to the methodological limitations of the CTT approach for rater-mediated assessments, it is imperative to examine the MCOP2 validity and reliability using the MFRM modeling technique to implement dimensionality analysis, item-level analysis, rater effects control, and ratee and rater ability level calibration. To that end, two existing samples of the MCOP2 data were obtained and analyzed, where twelve raters were asked to rate 237 math classroom observations, using the MCOP2 classroom observation protocol. The data were analyzed under the MFRM framework, using Facets 3.83.3. Results of the Facets analysis showed that both the MCOP2 subscales (i.e., Student Engagement & Teacher Facilitation) were valid, unidimensional, and highly reliable rater-mediated performance measures across raters, ratees, and study samples. However, rater-item bias analyses revealed a type of intra-rater inconsistency, where some raters tended to rate more severely than other raters on certain items while more leniently on some other items. The overall findings are promising in that they provide systematic preliminary psychometric evidence for the viability of the MCOP2 protocol to be used for math teachers’ self-assessment and/or peer-assessment along with other designated raters in the future studies

Assessment and feedback for large classes in transnational engineering education: student–staff partnership-based innovative approach

Assessment and feedback (A&F) are two major components of any educational program and must be properly in place to ensure student learning and quality of experience. However, these important components come under severe challenges of meeting student expectations in the large class size context. When the program delivery relates to a transnational educational (TNE) scenario, the additional constraints on staff–student physical interaction, regional time differences and cultural background gaps introduce additional challenges: Conducting proper assessments and provide timely and constructive feedback to the students. In this paper, the authors propose a novel assessment and feedback framework which exploits having a large student number as a positive factor by introducing staff–student partnership to implement efficient assessment and feedback strategies. Authors propose to use students for peer-review, assessment design, evaluation rubric design and tutorial-based feedback. The students also take part in preparing feedback clusters based on which the instructor provides pseudo-personalised video feedback. Through feedback clusters, authors introduce the trade-off between individual feedback and generic feedback. The results of the study are particularly promising in terms of student satisfaction and learning enhancement

Calibrated Peer Reviews in Requirements Engineering Instruction: Application and Experiences

Instructing Requirements Engineering (RE) is a challenging task due to the absence of single absolute and correct solutions computer science students so often strive for. Instead, there is often a variety of compromise solutions for each RE problem. Therefore, it is essential that aspiring Software Engineers are exposed to as many solution alternatives as possible to experience the implications of RE decisions. To facilitate this, we propose a learning-by-multiple-examples process, in which we make use of a calibrated peer review grading model for assignments. Paired with a think-pair-share model of semester-long, industry-realistic, project-based low-stakes milestones, we were able to generate a rich collaborative learning atmosphere. In this paper, we report the course design and experiences from the application of calibrated peer reviews in an undergraduate RE course. Qualitative and quantitative application results show that calibrated peer reviews significantly improve students’ learning outcomes

Sensitivity to reward and adolescents’ unhealthy snacking and drinking behavior: the role of hedonic eating styles and availability

Background: Although previous research found a positive association between sensitivity to reward (SR) and adolescents' unhealthy snacking and drinking behavior, mechanisms explaining these associations remain to be explored. The present study will therefore examine whether the associations between SR and unhealthy snack and/or sugar-sweetened beverage (SSB) intake are mediated by external and/or emotional eating and if this mediation is moderated by availability at home or at school. Methods: Cross-sectional data on snacking, availability of snacks at home and at school, SR (BAS drive scale) and external and emotional eating (Dutch eating behavior questionnaire) of Flemish adolescents (n = 1104, mean age = 14.7 +/- 0.8 years; 51 % boys; 18.0 % overweight) in 20 schools spread across Flanders were collected. Moderated mediation analyses were conducted using generalized structural equation modeling in three steps: (1) direct association between SR and unhealthy snack or SSB intake, (2) mediation of either external or emotional eating and (3) interaction of home or school availability and emotional or external eating. Results: Partial mediation of external eating (a*b = 0.69, p < 0.05) and of emotional eating (a*b = 0.92, p < 0.01) in the relation between SR and intake of unhealthy snacks was found (step 2). The relation between SR and SSB intake was not mediated by external or emotional eating (step 2). No moderation effects of home or school availability were found (step 3). Conclusion: Our findings indicate that the association between SR and the consumption of unhealthy snacks is partially explained by external and emotional eating in a population-based sample of adolescents irrespective of the home or school availability of these foods

THE ROLE OF SIMULATION IN SUPPORTING LONGER-TERM LEARNING AND MENTORING WITH TECHNOLOGY

Mentoring is an important part of professional development and longer-term learning. The nature of longer-term mentoring contexts means that designing, developing, and testing adaptive learning sys-tems for use in this kind of context would be very costly as it would require substantial amounts of fi-nancial, human, and time resources. Simulation is a cheaper and quicker approach for evaluating the impact of various design and development decisions. Within the Artificial Intelligence in Education (AIED) research community, however, surprisingly little attention has been paid to how to design, de-velop, and use simulations in longer-term learning contexts. The central challenge is that adaptive learning system designers and educational practitioners have limited guidance on what steps to consider when designing simulations for supporting longer-term mentoring system design and development deci-sions. My research work takes as a starting point VanLehn et al.’s [1] introduction to applications of simulated students and Erickson et al.’s [2] suggested approach to creating simulated learning envi-ronments. My dissertation presents four research directions using a real-world longer-term mentoring context, a doctoral program, for illustrative purposes. The first direction outlines a framework for guid-ing system designers as to what factors to consider when building pedagogical simulations, fundamen-tally to answer the question: how can a system designer capture a representation of a target learning context in a pedagogical simulation model? To illustrate the feasibility of this framework, this disserta-tion describes how to build, the SimDoc model, a pedagogical model of a longer-term mentoring learn-ing environment – a doctoral program. The second direction builds on the first, and considers the issue of model fidelity, essentially to answer the question: how can a system designer determine a simulation model’s fidelity to the desired granularity level? This dissertation shows how data from a target learning environment, the research literature, and common sense are combined to achieve SimDoc’s medium fidelity model. The third research direction explores calibration and validation issues to answer the question: how many simulation runs does it take for a practitioner to have confidence in the simulation model’s output? This dissertation describes the steps taken to calibrate and validate the SimDoc model, so its output statistically matches data from the target doctoral program, the one at the university of Saskatchewan. The fourth direction is to demonstrate the applicability of the resulting pedagogical model. This dissertation presents two experiments using SimDoc to illustrate how to explore pedagogi-cal questions concerning personalization strategies and to determine the effectiveness of different men-toring strategies in a target learning context. Overall, this dissertation shows that simulation is an important tool in the AIED system design-ers’ toolkit as AIED moves towards designing, building, and evaluating AIED systems meant to support learners in longer-term learning and mentoring contexts. Simulation allows a system designer to exper-iment with various design and implementation decisions in a cost-effective and timely manner before committing to these decisions in the real world