Self- and peer assessment may not be an accurate measure of PBL tutorial process

<p>Abstract</p> <p>Background</p> <p>Universidade Cidade de São Paulo adopted a problem-based learning (PBL) strategy as the predominant method for teaching and learning medicine. Self-, peer- and tutor marks of the educational process are taken into account as part of the final grade, which also includes assessment of content. This study compared the different perspectives (and grades) of evaluators during tutorials with first year medical students, from 2004 to 2007 (n = 349), from seven semesters.</p> <p>Methods</p> <p>The tutorial evaluation method was comprised of the students' self assessment (SA) (10%), tutor assessment (TA) (80%) and peer assessment (PA) (10%) to calculate a final educational process grade for each tutorial. We compared these three grades from each tutorial for seven semesters using ANOVA and a post hoc test.</p> <p>Results</p> <p>A total of 349 students participated with 199 (57%) women and 150 (42%) men. The SA and PA scores were consistently greater than the TA scores. Moreover, the SA and PA groups did not show statistical difference in any semester evaluated, while both differed from tutor assessment in all semesters (Kruskal-Wallis, Dunn's test). The Spearman rank order showed significant (p < 0.0001) and positive correlation for the SA and PA groups (r = 0.806); this was not observed when we compared TA with PA (r = 0.456) or TA with SA (r = 0.376).</p> <p>Conclusion</p> <p>Peer- and self-assessment marks might be reliable but not valid for PBL tutorial process, especially if these assessments are used for summative assessment, composing the final grade. This article suggests reconsideration of the use of summative assessment for self-evaluation in PBL tutorials.</p

The use of global rating scales for OSCEs in veterinary medicine

OSCEs (Objective Structured Clinical Examinations) are widely used in health professions to assess clinical skills competence. Raters use standardized binary checklists (CL) or multi-dimensional global rating scales (GRS) to score candidates performing specific tasks. This study assessed the reliability of CL and GRS scores in the assessment of veterinary students, and is the first study to demonstrate the reliability of GRS within veterinary medical education. Twelve raters from two different schools (6 from University of Calgary [UCVM] and 6 from Royal (Dick) School of Veterinary Studies [R(D)SVS] were asked to score 12 students (6 from each school). All raters assessed all students (video recordings) during 4 OSCE stations (bovine haltering, gowning and gloving, equine bandaging and skin suturing). Raters scored students using a CL, followed by the GRS. Novice raters (6 R(D)SVS) were assessed independently of expert raters (6 UCVM). Generalizability theory (G theory), analysis of variance (ANOVA) and t-tests were used to determine the reliability of rater scores, assess any between school differences (by student, by rater), and determine if there were differences between CL and GRS scores. There was no significant difference in rater performance with use of the CL or the GRS. Scores from the CL were significantly higher than scores from the GRS. The reliability of checklist scores were .42 and .76 for novice and expert raters respectively. The reliability of the global rating scale scores were .7 and .86 for novice and expert raters respectively. A decision study (D-study) showed that once trained using CL, GRS could be utilized to reliably score clinical skills in veterinary medicine with both novice and experienced raters

The Generation R Study: design and cohort update 2010

The Generation R Study is a population-based prospective cohort study from fetal life until young adulthood. The study is designed to identify early environmental and genetic causes of normal and abnormal growth, development and health during fetal life, childhood and adulthood. The study focuses on four primary areas of research: (1) growth and physical development; (2) behavioural and cognitive development; (3) diseases in childhood; and (4) health and healthcare for pregnant women and children. In total, 9,778 mothers with a delivery date from April 2002 until January 2006 were enrolled in the study. General follow-up rates until the age of 4 years exceed 75%. Data collection in mothers, fathers and preschool children included questionnaires, detailed physical and ultrasound examinations, behavioural observations, and biological samples. A genome wide association screen is available in the participating children. Regular detailed hands on assessment are performed from the age of 5 years onwards. Eventually, results forthcoming from the Generation R Study have to contribute to the development of strategies for optimizing health and healthcare for pregnant women and children

Skewed X-inactivation is common in the general female population

X-inactivation is a well-established dosage compensation mechanism ensuring that X-chromosomal genes are expressed at comparable levels in males and females. Skewed X-inactivation is often explained by negative selection of one of the alleles. We demonstrate that imbalanced expression of the paternal and maternal X-chromosomes is common in the general population and that the random nature of the X-inactivation mechanism can be sufficient to explain the imbalance. To this end, we analyzed blood-derived RNA and whole-genome sequencing data from 79 female children and their parents from the Genome of the Netherlands project. We calculated the median ratio of the paternal over total counts at all X-chromosomal heterozygous single-nucleotide variants with coverage ≥10. We identified two individuals where the same X-chromosome was inactivated in all cells. Imbalanced expression of the two X-chromosomes (ratios ≤0.35 or ≥0.65) was observed in nearly 50% of the population. The empirically observed skewing is explained by a theoretical model where X-inactivation takes place in an embryonic stage in which eight cells give rise to the hematopoietic compartment. Genes escaping X-inactivation are expressed from both alleles and therefore demonstrate less skewing than inactivated genes. Using this characteristic, we identified three novel escapee genes (SSR4, REPS2, and SEPT6), but did not find support for many previously reported escapee genes in blood. Our collective data suggest that skewed X-inactivation is common in the general population. This may contribute to manifestation of symptoms in carriers of recessive X-linked disorders. We recommend that X-inactivation results should not be used lightly in the interpretation of X-linked variants

Specialty Training&rsquo;s Organizational Readiness for curriculum Change (STORC): validation of a questionnaire

Lindsay Bank,1,2 Mari&euml;lle Jippes,3 Jimmie Leppink,4 Albert JJA Scherpbier,4 Corry den Rooyen,5 Scheltus J van Luijk,6 Fedde Scheele1,2,7 1Department of Healthcare Education, OLVG Hospital, 2Faculty of Earth and Life Sciences, Athena Institute for Transdisciplinary Research, VU University, Amsterdam, 3Department of Plastic Surgery, Erasmus Medical Center, Rotterdam, 4Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Maastricht University, Maastricht, 5Movation BV, Maarssen, 6Department of Healthcare Education, Maastricht University Medical Center+, Maastricht, 7School of Medical Sciences, Institute for Education and Training, VU University Medical Center, Amsterdam, the Netherlands Background: The field of postgraduate medical education (PGME) is continuously evolving as a result of social demands and advancing educational insights. Change experts contend that organizational readiness for change (ORC) is a critical precursor for successful implementation of change initiatives. However, in PGME, assessing change readiness is rarely considered while it could be of great value for managing educational change such as curriculum change. Therefore, in a previous Delphi study the authors developed an instrument for assessing ORC in PGME: Specialty Training&rsquo;s Organizational Readiness for curriculum Change (STORC). In this study, the psychometric properties of this questionnaire were further explored.Methods: In 2015, STORC was distributed among clinical teaching teams in the Netherlands. The authors conducted a confirmatory factor analysis on the internal factor structure of STORC. The reliability of the measurements was estimated by calculating Cronbach&rsquo;s alpha for all subscales. Additionally, a behavioral support-for-change measure was distributed as well to assess correlations with change-related behavior.Results: In total, the STORC questionnaire was completed by 856 clinical teaching team members from 39 specialties. Factor analysis led to the removal of 1 item but supported the expected factor structure with very good fit for the other 43 items. Supportive behavior was positively correlated to a higher level of ORC.Discussion: In this study, additional steps to collect validity evidence for the STORC questionnaire were taken successfully. The final subscales of STORC represent the core components of ORC in the literature. By breaking down this concept into multiple measurable aspects, STORC could help to enable educational leaders to diagnose possible hurdles in implementation processes and to perform specifically targeted interventions when needed. Keywords: organizational readiness for change, postgraduate medical education, curriculum change, questionnaire development, change management&nbsp

Are they ready? Organizational readiness for change among clinical teaching teams

Lindsay Bank,1,2 Mari&euml;lle Jippes,3 Jimmie Leppink,4 Albert JJA Scherpbier,4 Corry den Rooyen,5 Scheltus J van Luijk,6 Fedde Scheele1,2,7 1Department of Healthcare Education, OLVG Hospital, 2Faculty of Earth and Life Sciences, Athena Institute for Transdisciplinary Research, VU University, Amsterdam, 3Department of Plastic Surgery, Erasmus Medical Centre, Rotterdam, 4Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Maastricht University, Maastricht, 5Movation BV, Maarssen, 6Department of Healthcare Education, Maastricht University Medical Center+, Maastricht, 7School of Medical Sciences, Institute for Education and Training, VU University Medical Center, Amsterdam, the Netherlands Introduction: Curriculum change and innovation are inevitable parts of progress in postgraduate medical education (PGME). Although implementing change is known to be challenging, change management principles are rarely looked at for support. Change experts contend that organizational readiness for change (ORC) is a critical precursor for the successful implementation of change initiatives. Therefore, this study explores whether assessing ORC in clinical teaching teams could help to understand how curriculum change takes place in PGME.Methods: Clinical teaching teams in hospitals in the Netherlands were requested to complete the Specialty Training&rsquo;s Organizational Readiness for curriculum Change, a questionnaire to measure ORC in clinical teaching teams. In addition, change-related behavior was measured by using the &ldquo;behavioral support-for-change&rdquo; measure. A two-way analysis of variance was performed for all response variables of interest. Results: In total, 836 clinical teaching team members were included in this study: 288 (34.4%) trainees, 307 (36.7%) clinical staff members, and 241 (28.8%) program directors. Overall, items regarding whether the program director has the authority to lead scored higher compared with the other items. At the other end, the subscales &ldquo;management support and leadership,&rdquo; &ldquo;project resources,&rdquo; and &ldquo;implementation plan&rdquo; had the lowest scores in all groups.Discussion: The study brought to light that program directors are clearly in the lead when it comes to the implementation of educational innovation. Clinical teaching teams tend to work together as a team, sharing responsibilities in the implementation process. However, the results also reinforce the need for change management support in change processes in PGME. Keywords: organizational readiness for change, postgraduate medical education, curriculum change, change management, questionnaire, innovatio

Objective structured clinical examinations (OSCEs), psychiatry and the clinical assessment of skills and competencies (CASC) : same evidence, different judgement

Background: The Objective Structured Clinical Examination (OSCE), originally developed in the 1970’s, has been hailed as the "gold standard" of clinical assessments for medical students and is used within medical schools throughout the world. The Clinical assessment of Skills and Competencies (CASC) is an OSCE used as a clinical examination gateway, granting access to becoming a senior Psychiatrist in the UK. Discussion: Van der Vleuten’s utility model is used to examine the CASC from the viewpoint of a senior psychiatrist. Reliability may be equivalent to more traditional examinations. Whilst the CASC is likely to have content validity, other forms of validity are untested and authenticity is poor. Educational impact has the potential to change facets of psychiatric professionalism and influence future patient care. There are doubts about acceptability from candidates and more senior psychiatrists. Summary: Whilst OSCEs may be the best choice for medical student examinations, their use in post graduate psychiatric examination in the UK is subject to challenge on the grounds of validity, authenticity and educational impact

Web-assisted assessment of professional behaviour in problem-based learning: more feedback, yet no qualitative improvement?

Although other web-based approaches to assessment of professional behaviour have been studied, no publications studying the potential advantages of a web-based instrument versus a classic, paper-based method have been published to date. This study has two research goals: it focuses on the quantity and quality of comments provided by students and their peers (two researchers independently scoring comments as correct and incorrect in relation to five commonly used feedback rules (and resulting in an aggregated score of the five scores) on the one, and on the feasibility, acceptability and perceived usefulness of the two approaches on the other hand (using a survey). The amount of feedback was significantly higher in the web-based group than in the paper based group for all three categories (dealing with work, others and oneself). Regarding the quality of feedback, the aggregated score for each of the three categories was not significantly different between the two groups, neither for the interim, nor for the final assessment. Some, not statistically significant, but nevertheless noteworthy trends were nevertheless noted. Feedback in the web-based group was more often unrelated to observed behaviour for several categories for both the interim and final assessment. Furthermore, most comments relating to the category ‘Dealing with oneself’ consisted of descriptions of a student’s attendance, thereby neglecting other aspects of personal functioning. The survey identified significant differences between the groups for all questionnaire items regarding feasibility, acceptability and perceived usefulness in favour of the paper-based form. The use of a web-based instrument for professional behaviour assessment yielded a significantly higher number of comments compared to the traditional paper-based assessment. Unfortunately, the quality of the feedback obtained by the web-based instrument as measured by several generally accepted feedback criteria did not parallel this increase