Predictive validity of the UK clinical aptitude test in the final years of medical school:a prospective cohort study

Peer reviewedPublisher PD

The academic backbone: longitudinal continuities in educational achievement from secondary school and medical school to MRCP(UK) and the specialist register in UK medical students and doctors

Background: Selection of medical students in the UK is still largely based on prior academic achievement, although doubts have been expressed as to whether performance in earlier life is predictive of outcomes later in medical school or post-graduate education. This study analyses data from five longitudinal studies of UK medical students and doctors from the early 1970s until the early 2000s. Two of the studies used the AH5, a group test of general intelligence (that is, intellectual aptitude). Sex and ethnic differences were also analyzed in light of the changing demographics of medical students over the past decades. Methods: Data from five cohort studies were available: the Westminster Study (began clinical studies from 1975 to 1982), the 1980, 1985, and 1990 cohort studies (entered medical school in 1981, 1986, and 1991), and the University College London Medical School (UCLMS) Cohort Study (entered clinical studies in 2005 and 2006). Different studies had different outcome measures, but most had performance on basic medical sciences and clinical examinations at medical school, performance in Membership of the Royal Colleges of Physicians (MRCP(UK)) examinations, and being on the General Medical Council Specialist Register. Results: Correlation matrices and path analyses are presented. There were robust correlations across different years at medical school, and medical school performance also predicted MRCP(UK) performance and being on the GMC Specialist Register. A-levels correlated somewhat less with undergraduate and post-graduate performance, but there was restriction of range in entrants. General Certificate of Secondary Education (GCSE)/O-level results also predicted undergraduate and post-graduate outcomes, but less so than did A-level results, but there may be incremental validity for clinical and post-graduate performance. The AH5 had some significant correlations with outcome, but they were inconsistent. Sex and ethnicity also had predictive effects on measures of educational attainment, undergraduate, and post-graduate performance. Women performed better in assessments but were less likely to be on the Specialist Register. Non-white participants generally underperformed in undergraduate and post-graduate assessments, but were equally likely to be on the Specialist Register. There was a suggestion of smaller ethnicity effects in earlier studies. Conclusions: The existence of the Academic Backbone concept is strongly supported, with attainment at secondary school predicting performance in undergraduate and post-graduate medical assessments, and the effects spanning many years. The Academic Backbone is conceptualized in terms of the development of more sophisticated underlying structures of knowledge ('cognitive capital’ and 'medical capital’). The Academic Backbone provides strong support for using measures of educational attainment, particularly A-levels, in student selection

Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: meta-regression of six UK longitudinal studies

Background: Measures used for medical student selection should predict future performance during training. A problem for any selection study is that predictor-outcome correlations are known only in those who have been selected, whereas selectors need to know how measures would predict in the entire pool of applicants. That problem of interpretation can be solved by calculating construct-level predictive validity, an estimate of true predictor-outcome correlation across the range of applicant abilities. Methods: Construct-level predictive validities were calculated in six cohort studies of medical student selection and training (student entry, 1972 to 2009) for a range of predictors, including A-levels, General Certificates of Secondary Education (GCSEs)/O-levels, and aptitude tests (AH5 and UK Clinical Aptitude Test (UKCAT)). Outcomes included undergraduate basic medical science and finals assessments, as well as postgraduate measures of Membership of the Royal Colleges of Physicians of the United Kingdom (MRCP(UK)) performance and entry in the Specialist Register. Construct-level predictive validity was calculated with the method of Hunter, Schmidt and Le (2006), adapted to correct for right-censorship of examination results due to grade inflation. Results: Meta-regression analyzed 57 separate predictor-outcome correlations (POCs) and construct-level predictive validities (CLPVs). Mean CLPVs are substantially higher (.450) than mean POCs (.171). Mean CLPVs for first-year examinations, were high for A-levels (.809; CI: .501 to .935), and lower for GCSEs/O-levels (.332; CI: .024 to .583) and UKCAT (mean = .245; CI: .207 to .276). A-levels had higher CLPVs for all undergraduate and postgraduate assessments than did GCSEs/O-levels and intellectual aptitude tests. CLPVs of educational attainment measures decline somewhat during training, but continue to predict postgraduate performance. Intellectual aptitude tests have lower CLPVs than A-levels or GCSEs/O-levels. Conclusions: Educational attainment has strong CLPVs for undergraduate and postgraduate performance, accounting for perhaps 65% of true variance in first year performance. Such CLPVs justify the use of educational attainment measure in selection, but also raise a key theoretical question concerning the remaining 35% of variance (and measurement error, range restriction and right-censorship have been taken into account). Just as in astrophysics, ‘dark matter’ and ‘dark energy’ are posited to balance various theoretical equations, so medical student selection must also have its ‘dark variance’, whose nature is not yet properly characterized, but explains a third of the variation in performance during training. Some variance probably relates to factors which are unpredictable at selection, such as illness or other life events, but some is probably also associated with factors such as personality, motivation or study skills

The educational impact of assessment: a comparison of DOPS and MCQs

Aim: To evaluate the impact of two different assessment formats on the approaches to learning of final year veterinary students. The relationship between approach to learning and examination performance was also investigated. Method: An 18-item version of the Study Process Questionnaire (SPQ) was sent to 87 final year students. Each student responded to the questionnaire with regards to DOPS (Direct Observation of Procedural Skills) and a Multiple Choice Examination (MCQ). Semi-structured interviews were conducted with 16 of the respondents to gain a deeper insight into the students’ perception of assessment. Results: Students’ adopted a deeper approach to learning for DOPS and a more surface approach with MCQs. There was a positive correlation between an achieving approach to learning and examination performance. Analysis of the qualitative data revealed that deep, surface and achieving approaches were reported by the students and seven major influences on their approaches to learning were identified: motivation, purpose, consequence, acceptability, feedback, time pressure and the individual difference of the students. Conclusions: The format of DOPS has a positive influence on approaches to learning. There is a conflict for students between preparing for final examinations and preparing for clinical practice

Chemistry courses as the turning point for premedical students

Previous research has documented that negative experiences in chemistry courses are a major factor that discourages many students from continuing in premedical studies. This adverse impact affects women and students from under-represented minority (URM) groups disproportionately. To determine if chemistry courses have a similar effect at a large public university, we surveyed 1,036 students from three entering cohorts at the University of California, Berkeley. We surveyed students at the beginning of their first year at the university and again at the end of their second year. All subjects had indicated an interest in premedical studies at the time they entered the university. We conducted follow-up interviews with a stratified sub-set of 63 survey respondents to explore the factors that affected their level of interest in premedical studies. Using a 10-point scale, we found that the strength of interest in premedical studies declined for all racial/ethnic groups. In the follow-up interviews, students identified chemistry courses as the principal factor contributing to their reported loss of interest. URM students especially often stated that chemistry courses caused them to abandon their hopes of becoming a physician. Consistent with reports over more than 50 years, it appears that undergraduate courses in chemistry have the effect of discouraging otherwise qualified students, as reflected in their admission to one of the most highly selective public universities in the US, from continuing in premedical studies, especially in the case of URM students. Reassessment of this role for chemistry courses may be overdue

Integrating 360° behavior-orientated feedback in communication skills training for medical undergraduates: concept, acceptance and students’ self-ratings of communication competence

BACKGROUND: Feedback is considered a key didactic element in medical education, especially for teaching of communication skills. This study investigates the impact of a best evidence-based practice feedback concept within the context of communication skills training (CST). We evaluate this concept for acceptance and changes in students self-ratings of communication competence. METHODS: Our CST integrating feedback process comprises a short theoretical introduction presenting standards for good communication and a constructive 360° feedback from three perspectives: feedback from peers, from standardized patients (SPs), and from a trainer. Feed-forward process was facilitated for documenting suggestions for improvements based on observable behaviors to maximize learning benefits. Our CST was applied to four groups of eight or nine students. We assessed the data on students’ acceptance using a 6-point scale ranging from very good (1) to poor (6), applied a forced choice question to rank didactic items, and assessed changes in student’ self-ratings of their communication competence on a 10-cm visual analogue scale (VAS). RESULTS: Thirty-four medical undergraduates (82 % female, 18 % male) in their first clinical year, with an average age of 21.4 years (SD = 1.0), participated in the new training. The concept achieved high acceptance from good to very good: overall impression (M = 1.56), sufficient interaction for discussion (M = 1.15), and constructive learning atmosphere (M = 1.18). Specific elements, such as practical training with SPs (M = 1.18) and feedback by SPs (M = 1.12), showed highest acceptance. The forced choice ranking placed all feedback elements at the top of the list (feedback (FB) by SPs, rank 2; FB by trainer, rank 3; FB by colleagues, rank 4), whereas theoretical elements were at the bottom (theoretical introduction, rank 7; memory card, rank 9). Overall, student self-ratings of communication competence significantly improved in nine of the ten communication items assessed by VAS and showed a pre-post effect size of ES = 0.74 on a global rating. CONCLUSIONS: This study demonstrates that the training concept based on 360° behavioral feedback was well accepted and generated significant changes in student self-ratings of their communication competence. Further research is needed to determine the effects on objective communication performance. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (doi:10.1186/s12909-016-0792-0) contains supplementary material, which is available to authorized users

The Dreyfus model of clinical problem-solving skills acquisition: a critical perspective

Context: The Dreyfus model describes how individuals progress through various levels in their acquisition of skills and subsumes ideas with regard to how individuals learn. Such a model is being accepted almost without debate from physicians to explain the ‘acquisition’ of clinical skills. Objectives: This paper reviews such a model, discusses several controversial points, clarifies what kind of knowledge the model is about, and examines its coherence in terms of problem-solving skills. Dreyfus’ main idea that intuition is a major aspect of expertise is also discussed in some detail. Relevant scientific evidence from cognitive science, psychology, and neuroscience is reviewed to accomplish these aims. Conclusions: Although the Dreyfus model may partially explain the ‘acquisition’ of some skills, it is debatable if it can explain the acquisition of clinical skills. The complex nature of clinical problem-solving skills and the rich interplay between the implicit and explicit forms of knowledge must be taken into consideration when we want to explain ‘acquisition’ of clinical skills. The idea that experts work from intuition, not from reason, should be evaluated carefully

The impact of preparatory activities on medical school selection outcomes: a cross-sectional survey of applicants to the University of Adelaide Medical School in 2007

Background: Selection into medical school is highly competitive with more applicants than places. Little is known about the preparation that applicants undertake for this high stakes process. The study aims to determine what preparatory activities applicants undertake and what difficulties they encounter for each stage of the application process to medical school and in particular what impact these have on the outcome. Methods: A cross-sectional survey of 1097 applicants who applied for a place in the University of Adelaide Medical School in 2007 and participated in the UMAT (Undergraduate Medicine and Health Sciences Admission Test) and oral assessment components of the selection process. The main outcome measures were an offer of an interview and offer of a place in the medical school and were analysed using logistic regression. Results: The odds of a successful outcome increased with each additional preparatory activity undertaken for the UMAT (odds ratio 1.22, 95% confidence interval 1.11 to 1.33; P < 0.001) and the oral assessment (1.36, 1.19 to 1.55; P < 0.001) stage of selection. The UMAT preparatory activities associated with the offer of an interview were attendance of a training course by a private organisation (1.75, 1.35 to 2.27: P < 0.001), use of online services of a private organisation (1.58, 1.23 to 2.04; P < 0.001), and familiarising oneself with the process (1.52, 1.15 to 2.00; p = 0.021). The oral assessment activities associated with an offer of a place included refining and learning a personal resume (9.73, 2.97 to 31.88; P < 0.001) and learning about the course structure (2.05, 1.29 to 3.26; P = 0.022). For the UMAT, applicants who found difficulties with learning for this type of test (0.47, 0.35 to 0.63: P < 0.001), with the timing of UMAT in terms of school exams (0.48, 0.5 to 0.66; P < 0.001) and with the inability to convey personal skills with the UMAT (0.67, 0.52 to 0.86; P = 0.026) were significantly less likely to be offered an interview. Conclusions: Medical schools make an enormous effort to undertake a selection process that is fair and equitable and which selects students most appropriate for medical school and the course they provide. Our results indicate that performance in the selection processes can be improved by training. However, if these preparatory activities may be limited to those who can access them, the playing field is not even and increasing equity of access to medical schools will not be achieved.Caroline O Laurence, Ian T Zajac, Michelle Lorimer, Deborah A Turnbull and Karen E Sumne