Development and validation of the ACE tool: Assessing medical trainees' competency in evidence based medicine

BACKGROUND: While a variety of instruments have been developed to assess knowledge and skills in evidence based medicine (EBM), few assess all aspects of EBM - including knowledge, skills attitudes and behaviour - or have been psychometrically evaluated. The aim of this study was to develop and validate an instrument that evaluates medical trainees’ competency in EBM across knowledge, skills and attitude. METHODS: The ‘Assessing Competency in EBM’ (ACE) tool was developed by the authors, with content and face validity assessed by expert opinion. A cross-sectional sample of 342 medical trainees representing ‘novice’, ‘intermediate’ and ‘advanced’ EBM trainees were recruited to complete the ACE tool. Construct validity, item difficulty, internal reliability and item discrimination were analysed. RESULTS: We recruited 98 EBM-novice, 108 EBM-intermediate and 136 EBM-advanced participants. A statistically significant difference in the total ACE score was observed and corresponded to the level of training: on a 0-15-point test, the mean ACE scores were 8.6 for EBM-novice; 9.5 for EBM-intermediate; and 10.4 for EBM-advanced (p < 0.0001). Individual item discrimination was excellent (Item Discrimination Index ranging from 0.37 to 0.84), with internal reliability consistent across all but three items (Item Total Correlations were all positive ranging from 0.14 to 0.20). CONCLUSION: The 15-item ACE tool is a reliable and valid instrument to assess medical trainees’ competency in EBM. The ACE tool provides a novel assessment that measures user performance across the four main steps of EBM. To provide a complete suite of instruments to assess EBM competency across various patient scenarios, future refinement of the ACE instrument should include further scenarios across harm, diagnosis and prognosis

Evidence-based practice educational intervention studies: A systematic review of what is taught and how it is measured

Abstract Background Despite the established interest in evidence-based practice (EBP) as a core competence for clinicians, evidence for how best to teach and evaluate EBP remains weak. We sought to systematically assess coverage of the five EBP steps, review the outcome domains measured, and assess the properties of the instruments used in studies evaluating EBP educational interventions. Methods We conducted a systematic review of controlled studies (i.e. studies with a separate control group) which had investigated the effect of EBP educational interventions. We used citation analysis technique and tracked the forward and backward citations of the index articles (i.e. the systematic reviews and primary studies included in an overview of the effect of EBP teaching) using Web of Science until May 2017. We extracted information on intervention content (grouped into the five EBP steps), and the outcome domains assessed. We also searched the literature for published reliability and validity data of the EBP instruments used. Results Of 1831 records identified, 302 full-text articles were screened, and 85 included. Of these, 46 (54%) studies were randomised trials, 51 (60%) included postgraduate level participants, and 63 (75%) taught medical professionals. EBP Step 3 (critical appraisal) was the most frequently taught step (63 studies; 74%). Only 10 (12%) of the studies taught content which addressed all five EBP steps. Of the 85 studies, 52 (61%) evaluated EBP skills, 39 (46%) knowledge, 35 (41%) attitudes, 19 (22%) behaviours, 15 (18%) self-efficacy, and 7 (8%) measured reactions to EBP teaching delivery. Of the 24 instruments used in the included studies, 6 were high-quality (achieved ≥3 types of established validity evidence) and these were used in 14 (29%) of the 52 studies that measured EBP skills; 14 (41%) of the 39 studies that measured EBP knowledge; and 8 (26%) of the 35 studies that measured EBP attitude. Conclusions Most EBP educational interventions which have been evaluated in controlled studies focus on teaching only some of the EBP steps (predominantly critically appraisal of evidence) and did not use high-quality instruments to measure outcomes. Educational packages and instruments which address all EBP steps are needed to improve EBP teaching

Molecular Surveillance of True Nontypeable Haemophilus influenzae: An Evaluation of PCR Screening Assays

BackgroundUnambiguous identification of nontypeable Haemophilus influenzae (NTHi) is not possible by conventional microbiology. Molecular characterisation of phenotypically defined NTHi isolates suggests that up to 40% are Haemophilus haemolyticus (Hh); however, the genetic similarity of NTHi and Hh limits the power of simple molecular techniques such as PCR for species discrimination.Methodology/Principal FindingsHere we assess the ability of previously published and novel PCR-based assays to identify true NTHi. Sixty phenotypic NTHi isolates, classified by a dual 16S rRNA gene PCR algorithm as NTHi (n = 22), Hh (n = 27) or equivocal (n = 11), were further characterised by sequencing of the 16S rRNA and recA genes then interrogated by PCR-based assays targeting the omp P2, omp P6, lgtC, hpd, 16S rRNA, fucK and iga genes. The sequencing data and PCR results were used to define NTHi for this study. Two hpd real time PCR assays (hpd#1 and hpd#3) and the conventional iga PCR assay were equally efficient at differentiating study-defined NTHi from Hh, each with a receiver operator characteristic curve area of 0.90 [0.83; 0.98]. The hpd#1 and hpd#3 assays were completely specific against a panel of common respiratory bacteria, unlike the iga PCR, and the hpd#3 assay was able to detect below 10 copies per reaction.Conclusions/SignificanceOur data suggest an evolutionary continuum between NTHi and Hh and therefore no single gene target could completely differentiate NTHi from Hh. The hpd#3 real time PCR assay proved to be the superior method for discrimination of NTHi from closely related Haemophilus species with the added potential for quantification of H. influenzae directly from specimens. We suggest the hpd#3 assay would be suitable for routine NTHi surveillance and to assess the impact of antibiotics and vaccines, on H. influenzae carriage rates, carriage density, and disease

Impact of Capsular Switch on Invasive Pneumococcal Disease Incidence in a Vaccinated Population

BACKGROUND: Despite the dramatic decline in the incidence of invasive pneumococcal disease (IPD) observed since the introduction of conjugate vaccination, it is feared that several factors may undermine the future effectiveness of the vaccines. In particular, pathogenic pneumococci may switch their capsular types and evade vaccine-conferred immunity. METHODOLOGY/PRINCIPAL FINDINGS: Here, we first review the literature and summarize the available epidemiological data on capsular switch for S. pneumoniae. We estimate the weekly probability that a persistently carried strain may switch its capsule from four studies, totalling 516 children and 6 years of follow-up, at 1.5x10(-3)/week [4.6x10(-5)-4.8x10(-3)/week]. There is not enough power to assess an increase in this frequency in vaccinated individuals. Then, we use a mathematical model of pneumococcal transmission to quantify the impact of capsular switch on the incidence of IPD in a vaccinated population. In this model, we investigate a wide range of values for the frequency of vaccine-selected capsular switch. Predictions show that, with vaccine-independent switching only, IPD incidence in children should be down by 48% 5 years after the introduction of the vaccine with high coverage. Introducing vaccine-selected capsular switch at a frequency up to 0.01/week shows little effect on this decrease; yearly, at most 3 excess cases of IPD per 10(6) children might occur due to switched pneumococcal strains. CONCLUSIONS: Based on all available data and model predictions, the existence of capsular switch by itself should not impact significantly the efficacy of pneumococcal conjugate vaccination on IPD incidence. This optimistic result should be tempered by the fact that the selective pressure induced by the vaccine is currently increasing along with vaccine coverage worldwide; continued surveillance of pneumococcal populations remains of the utmost importance, in particular during clinical trials of the new conjugate vaccines

Positively Selected Codons in Immune-Exposed Loops of the Vaccine Candidate OMP-P1 of Haemophilus influenzae

The high levels of variation in surface epitopes can be considered as an evolutionary hallmark of immune selection. New computational tools enable analysis of this variation by identifying codons that exhibit high rates of amino acid changes relative to the synonymous substitution rate. In the outer membrane protein P1 of Haemophilus influenzae, a vaccine candidate for nontypeable strains, we identified four codons with this attribute in domains that did not correspond to known or assumed B- and T-cell epitopes of OMP-P1. These codons flank hypervariable domains and do not appear to be false positives as judged from parsimony and maximum likelihood analyses. Some closely spaced positively selected codons have been previously considered part of a transmembrane domain, which would render this region unsuited for inclusion in a vaccine. Secondary structure analysis, three-dimensional structural database searches, and homology modeling using FadL of E. coli as a structural homologue, however, revealed that all positively selected codons are located in or near extracellular looping domains. The spacing and level of diversity of these positively selected and exposed codons in OMP-P1 suggest that vaccine targets based on these and conserved flanking residues may provide broad coverage in H. influenzae

The Population Genetics of dN/dS

Evolutionary pressures on proteins are often quantified by the ratio of substitution rates at non-synonymous and synonymous sites. The dN/dS ratio was originally developed for application to distantly diverged sequences, the differences among which represent substitutions that have fixed along independent lineages. Nevertheless, the dN/dS measure is often applied to sequences sampled from a single population, the differences among which represent segregating polymorphisms. Here, we study the expected dN/dS ratio for samples drawn from a single population under selection, and we find that in this context, dN/dS is relatively insensitive to the selection coefficient. Moreover, the hallmark signature of positive selection over divergent lineages, dN/dS>1, is violated within a population. For population samples, the relationship between selection and dN/dS does not follow a monotonic function, and so it may be impossible to infer selection pressures from dN/dS. These results have significant implications for the interpretation of dN/dS measurements among population-genetic samples

A prospective study evaluating the integration of a multifaceted evidence-based medicine curriculum into early years in an undergraduate medical school

BACKGROUND: The importance of ensuring medical students are equipped with the skills to be able to practice evidence-based medicine (EBM) has been increasingly recognized in recent years. However, there is limited information on an effective EBM curriculum for undergraduate medical schools. This study aims to test the feasibility of integrating a multifaceted EBM curriculum in the early years of an undergraduate medical school. This was subsequently evaluated using the validated Fresno test and students' self-reported knowledge and attitudes as they progressed through the curriculum. METHODS: EBM was integrated horizontally and vertically into the curriculum into the first 2 years of undergraduate medical school. First year medical students were recruited to participate in the study. The 212-point Fresno test was administered along with a locally developed questionnaire at baseline before EBM teaching in year one and at the end of EBM teaching in year two. RESULTS: Thirty-one students participated at baseline and 55 students participated at the end of second year EBM teaching. For the 18 students who completed the Fresno at both time points, the average score increased by 38.7 marks (p < 0.001) after EBM teaching. Students felt confident in formulating clinical questions and in critically appraising journal articles after EBM teaching. EBM was perceived to be important to their future practice as a doctor and for improving patient outcomes at both time points. CONCLUSIONS: It has been feasible to integrate a multifaceted, EBM curriculum from the first year of an undergraduate medical program. Early evaluation of the curriculum using the Fresno test has shown a significant increase in students' EBM knowledge. The curriculum also demonstrated an increase in students' perceptions of the clinical relevance of EBM in their developing practice