Toward Better Training in Peer Assessment: Does Calibration Help?

For peer assessments to be helpful, student reviewers need to submit reviews of good quality. This requires certain training or guidance from teaching staff, lest reviewers read each other\u27s work uncritically, and assign good scores but offer few suggestions. One approach to improving the review quality is calibration. Calibration refers to comparing students\u27 individual reviews to a standard—usually a review done by teaching staff on the same reviewed artifact. In this paper, we categorize two modes of calibration for peer assessment and discuss our experience with both of them in a pilot study with Expertiza system

The Influence of Participation in a Multi-Disciplinary Collaborative Service Learning Project on the Effectiveness of Team Members in a 100-level Mechanical Engineering Class

Engineers need to develop professional skills, including the ability to work successfully in teams and to communicate within and outside of their discipline, in addition to required technical skills. A collaborative multi-disciplinary service learning project referred to as Ed+gineering was implemented in a 100-level mechanical engineering course. In this collaboration, mechanical engineering students, primarily in the second semester of their freshman year or first semester of their second year, worked over the course of a semester with education students taking a foundations course to develop and deliver engineering lessons to fourth or fifth graders. Students in comparison engineering classes worked on a team project focused on experimental design for a small satellite system. The purpose of this study was to determine if participating in the Ed+gineering collaboration had a positive effect on teamwork effectiveness and satisfaction when compared to the comparison class. In both team projects, the five dimensions of the Comprehensive Assessment of Team Member Effectiveness (CATME) system were used as a quantitative assessment. The five dimensions of CATME Behaviorally Anchored Ratings Scale (BARS) (contribution to the team’s work, interacting with teammates, keeping the team on track, expecting quality, and having relevant Knowledge, Skills, and Abilities - KSAs) were measured. Additionally, within the CATME platform team satisfaction, team interdependence and team cohesiveness were measured. ANCOVA analysis was used to assess the quantitative data from CATME. Preliminary results suggest that students in the treatment classes had higher team member effectiveness and overall satisfaction scores than students in the comparison classes. Qualitative data from reflections written at the completion of the aforementioned projects were used to explore these results

Can We Make Our Robot Play Soccer? Influence of Collaborating with Preservice Teachers and Fifth Graders on Undergraduate Engineering Students\u27 Learning During a Robotic Design Process (Work in Progress)

This work-in-progress paper describes engineering students’ experiences in an NSF-funded project that partnered undergraduate engineering students with pre-service teachers to plan and deliver robotics lessons to fifth graders at a local school. This project aims to address an apparent gap between what is taught in academia and industry’s expectations of engineers to integrate perspectives from outside their field to solve modern societal problems requiring a multidisciplinary approach. Working in small teams over Zoom, participating engineering, education, and fifth grade students designed, built, and coded bio-inspired COVID companion robots. The goal for the engineering students was to build new interprofessional skills, while reinforcing technical skills. The collaborative activities included: (1) training with Hummingbird BitTM hardware (e.g. sensors, servo motors) and coding platform, (2) preparing robotics lessons for fifth graders that explained the engineering design process (EDP), and (3) guiding the fifth graders in the design of their robots. Additionally, each undergraduate engineering student designed a robot following the theme developed with their preservice teacher and fifth grade partners. The intervention took place in Spring 2021 amidst the COVID-19 pandemic, necessitating the investigators to make critical decisions to address challenges of implementing the intervention in an online setting. This paper describes those decisions as it investigates how the cross-disciplinary, mixed-aged collaboration with preservice teachers and fifth graders impacted undergraduate engineering students’ learning and investment during the design process of their robots. Preliminary results of a regression analysis revealed a relationship between the engineering students’ robot rankings and post-scores on the design process knowledge survey (r = 0.92). Consistencies and a few anomalies in this pattern were explained using qualitative reflections which were analyzed to determine students’ level of investment in the project, overall perceptions, and the extent to which they focused on the fifth graders’ ideas in their designs. In general, robot quality was linked to both undergraduate engineering students’ level of investment and whether they focused on the fifth graders’ ideas in their designs. Engineering students’ overall perceptions of the project were generally positive, appreciating the role of cross-disciplinary and mixed-aged collaborations in their learning to brainstorm innovative solutions and interact effectively with professionals outside of engineering as they embark on tackling societal problems in the real world

Experiences During the Implementation of Two Different Project-Based Learning Assignments in a Fluid Mechanics Course.

There is growing evidence of the effectiveness of project-based learning (PBL) in preparing students to solve complex problems. In PBL implementations in engineering, students are treated as professional engineers facing projects centered around real-world problems, including the complexity and uncertainty that influence such problems. Not only does this help students to analyze and solve an authentic real-world task, promoting critical thinking, but also students learn from each other, learning valuable communication and teamwork skills. Faculty play an important part by assuming non-conventional roles (e.g., client, senior professional engineer, consultant) to help students throughout this instructional and learning approach. Typically in PBLs, students work on projects over extended periods of time that culminate in realistic products or presentations. In order to be successful, students need to learn how to frame a problem, identify stakeholders and their requirements, design and select concepts, test them, and so on. Two different implementations of PBL projects in a fluid mechanics course are presented in this paper. This required, junior-level course has been taught since 2014 by the same instructor. The first PBL project presented is a complete design of pumped pipeline systems for a hypothetical plant. In the second project, engineering students partnered with pre-service teachers to design and teach an elementary school lesson on fluid mechanics concepts. With the PBL implementations, it is expected that students: 1) engage in a deeper learning process where concepts can be reemphasized, and students can realize applicability; 2) develop and practice teamwork skills; 3) learn and practice how to communicate effectively to peers and to those from other fields; and 4) increase their confidence working on open-ended situations and problems. The goal of this paper is to present the experiences of the authors with both PBL implementations. It explains how the projects were scaffolded through the entire semester, including how the sequence of course content was modified, how team dynamics were monitored, the faculty roles, and the end products and presentations. Students\u27 experiences are also presented. To evaluate and compare students’ learning and satisfaction with the team experience between the two PBL implementations, a shortened version of the NCEES FE exam and the Comprehensive Assessment of Team Member Effectiveness (CATME) survey were utilized. Students completed the FE exam during the first week and then again during the last week of the semester in order to assess students’ growth in fluid mechanics knowledge. The CATME survey was completed mid-semester to help faculty identify and address problems within team dynamics, and at the end of the semester to evaluate individual students’ teamwork performance. The results showed that no major differences were observed in terms of the learned fluid mechanics content, however, the data showed interesting preliminary observations regarding teamwork satisfaction. Through reflective assignments (e.g., short answer reflections, focus groups), student perceptions of the PBL implementations are discussed in the paper. Finally, some of the challenges and lessons learned from implementing both projects multiple times, as well as access to some of the PBL course materials and assignments will be provided

Simulation-Based Training of the Rapid Evaluation and Management of Acute Stroke (STREAM)-A Prospective Single-Arm Multicenter Trial

Introduction: Acute stroke care delivered by interdisciplinary teams is time-sensitive. Simulation-based team training is a promising tool to improve team performance in medical operations. It has the potential to improve process times, team communication, patient safety, and staff satisfaction. We aim to assess whether a multi-level approach consisting of a stringent workflow revision based on peer-to-peer review and 2-3 one-day in situ simulation trainings can improve acute stroke care processing times in high volume neurocenters within a 6 months period. Methods and Analysis: The trial is being carried out in a pre-test-post-test design at 7 tertiary care university hospital neurocenters in Germany. The intervention is directed at the interdisciplinary multiprofessional stroke teams. Before and after the intervention, process times of all direct-to-center stroke patients receiving IV thrombolysis (IVT) and/or endovascular therapy (EVT) will be recorded. The primary outcome measure will be the door-to-needle time of all consecutive stroke patients directly admitted to the neurocenters who receive IVT. Secondary outcome measures will be intervention-related process times of the fraction of patients undergoing EVT and effects on team communication, perceived patient safety, and staff satisfaction via a staff questionnaire. Interventions: We are applying a multi-level intervention in cooperation with three STREAM multipliers from each center. First step is a central meeting of the multipliers at the sponsor's institution with the purposes of algorithm review in a peer-to-peer process that is recorded in a protocol and an introduction to the principles of simulation training and debriefing as well as crew resource management and team communication. Thereafter, the multipliers cooperate with the stroke team trainers from the sponsor's institution to plan and execute 2-3 one-day simulation courses in situ in the emergency department and CT room of the trial centers whereupon they receive teaching materials to perpetuate the trainings

Quantitative Signal Intensity in Fluid-Attenuated Inversion Recovery and Treatment Effect in the WAKE-UP Trial

International audienceBackground and Purpose— Relative signal intensity of acute ischemic stroke lesions in fluid-attenuated inversion recovery (fluid-attenuated inversion recovery relative signal intensity [FLAIR-rSI]) magnetic resonance imaging is associated with time elapsed since stroke onset with higher intensities signifying longer time intervals. In the randomized controlled WAKE-UP trial (Efficacy and Safety of MRI-Based Thrombolysis in Wake-Up Stroke Trial), intravenous alteplase was effective in patients with unknown onset stroke selected by visual assessment of diffusion weighted imaging fluid-attenuated inversion recovery mismatch, that is, in those with no marked fluid-attenuated inversion recovery hyperintensity in the region of the acute diffusion weighted imaging lesion. In this post hoc analysis, we investigated whether quantitatively measured FLAIR-rSI modifies treatment effect of intravenous alteplase. Methods— FLAIR-rSI of stroke lesions was measured relative to signal intensity in a mirrored region in the contralesional hemisphere. The relationship between FLAIR-rSI and treatment effect on functional outcome assessed by the modified Rankin Scale (mRS) after 90 days was analyzed by binary logistic regression using different end points, that is, favorable outcome defined as mRS score of 0 to 1, independent outcome defined as mRS score of 0 to 2, ordinal analysis of mRS scores (shift analysis). All models were adjusted for National Institutes of Health Stroke Scale at symptom onset and stroke lesion volume. Results— FLAIR-rSI was successfully quantified in stroke lesions in 433 patients (86% of 503 patients included in WAKE-UP). Mean FLAIR-rSI was 1.06 (SD, 0.09). Interaction of FLAIR-rSI and treatment effect was not significant for mRS score of 0 to 1 ( P =0.169) and shift analysis ( P =0.086) but reached significance for mRS score of 0 to 2 ( P =0.004). We observed a smooth continuing trend of decreasing treatment effects in relation to clinical end points with increasing FLAIR-rSI. Conclusions— In patients in whom no marked parenchymal fluid-attenuated inversion recovery hyperintensity was detected by visual judgement in the WAKE-UP trial, higher FLAIR-rSI of diffusion weighted imaging lesions was associated with decreased treatment effects of intravenous thrombolysis. This parallels the known association of treatment effect and elapsing time of stroke onset