Using item response theory to explore the psychometric properties of extended matching questions examination in undergraduate medical education

BACKGROUND: As assessment has been shown to direct learning, it is critical that the examinations developed to test clinical competence in medical undergraduates are valid and reliable. The use of extended matching questions (EMQ) has been advocated to overcome some of the criticisms of using multiple-choice questions to test factual and applied knowledge. METHODS: We analysed the results from the Extended Matching Questions Examination taken by 4th year undergraduate medical students in the academic year 2001 to 2002. Rasch analysis was used to examine whether the set of questions used in the examination mapped on to a unidimensional scale, the degree of difficulty of questions within and between the various medical and surgical specialties and the pattern of responses within individual questions to assess the impact of the distractor options. RESULTS: Analysis of a subset of items and of the full examination demonstrated internal construct validity and the absence of bias on the majority of questions. Three main patterns of response selection were identified. CONCLUSION: Modern psychometric methods based upon the work of Rasch provide a useful approach to the calibration and analysis of EMQ undergraduate medical assessments. The approach allows for a formal test of the unidimensionality of the questions and thus the validity of the summed score. Given the metric calibration which follows fit to the model, it also allows for the establishment of items banks to facilitate continuity and equity in exam standards

Assessing health and well-being among older people in rural South Africa

Background: The population in developing countries is ageing, which is likely to increase the burden of noncommunicable diseases and disability. Objective: To describe factors associated with self-reported health, disability and quality of life (QoL) of older people in the rural northeast of South Africa. Design: Cross-sectional survey of 6,206 individuals aged 50 and over. We used multivariate analysis to examine relationships between demographic variables and measures of self-reported health (Health Status), functional ability (WHODASi) and quality of life (WHOQoL). Results: About 4,085 of 6,206 people eligible (65.8%) completed the interview. Women (Odds Ratio (OR) 1.30, 95% CI 1.09, 1.55), older age (OR2.59, 95% CI 1.97, 3.40), lower education (OR1.62, 95% CI 1.31,2.00), single status (OR1.18, 95% CI 1.01, 1.37) and not working at present (OR1.29, 95% CI 1.06, 1.59) were associated with a low health status. Women were also more likely to report a higher level of disability (OR1.38, 95% CI 1.14, 1.66), as were older people (OR2.92, 95% CI 2.25, 3.78), those with no education (OR1.57, 95% CI 1.26, 1.97), with single status (OR1.25, 95% CI 1.06, 1.46) and not working at present (OR1.33, 95% CI 1.06, 1.66). Older age (OR1.35, 95% CI 1.06, 1.74), no education (OR1.39, 95% CI 1.11, 1.73), single status (OR1.28, 95% CI 1.10, 1.49), a low household asset score (OR1.52, 95% CI 1.19, 1.94) and not working at present (OR1.32; 95% CI 1.07, 1.64) were all associated with lower quality of life. Conclusions: This study presents the first population-based data from South Africa on health status, functional ability and quality of life among older people. Health and social services will need to be restructured to provide effective care for older people living in rural South Africa with impaired functionality and other health problems

BRIEF REPORT: Brief Instrument to Assess Geriatrics Knowledge of Surgical and Medical Subspecialty House Officers

Initiatives are underway to increase geriatrics training in nonprimary care disciplines. However, no validated instrument exists to measure geriatrics knowledge of house officers in surgical specialties and medical subspecialties. METHODS : A 23-item multiple-choice test emphasizing inpatient care and common geriatric syndromes was developed through expert panels and pilot testing, and administered to 305 residents and fellows at 4 institutions in surgical disciplines (25% of respondents), emergency medicine (29%), medicine subspecialties (19%), internal medicine (12%), and other disciplines (15%). RESULTS : Three items decreased internal reliability. The remaining 20 items covered 17 topic areas. Residents averaged 62% correct on the test. Internal consistency was appropriate (Cronbach's Α coefficient=0.60). Validity was supported by the use of expert panels to develop content, and by overall differences in scores by level of training ( P <.0001) and graded improvement in test performance, with 58%, 63%, 62%, and 69% correct responses among HO1, HO2, HO3, and HO4s, respectively. CONCLUSIONS : This reliable, valid measure of clinical geriatrics knowledge can be used by a wide variety of surgical and medical graduate medical education programs to guide curriculum reform or evaluate program performance to meet certification requirements. The instrument is now available on the web.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/73902/1/j.1525-1497.2006.00433.x.pd

Prevalence Rates of Mental Disorders in Chilean Prisons

PMCID: PMC371883

Predicting residents' performance: A prospective study

BACKGROUND: Objective criteria for predicting residents' performance do not exist. The purpose of this study was to test the hypothesis that global assessment by an intern selection committee (ISC) would correlate with the future performance of residents. METHODS: A prospective study of 277 residents between 1992 and 1999. Global assessment at the time of interview was compared to subsequent clinical (assessed by chief residents) and cognitive performance (assessed by the American Board of Pediatrics in-service training examination). RESULTS: ISC ratings correlated significantly with clinical performance at 24 and 36 months of training (r = 0.58, P < .001; and r = 0.60, P < .001 respectively). ISC ratings also correlated significantly with in-service exam scores in the 1(st), 2(nd), and 3(rd) years of training (r = 0.35, P = .0016; r = 0.39, P = 0.0003; r = 0.50, P = 0.005 respectively). CONCLUSIONS: Global assessment by an ISC predicted residents' clinical and cognitive performances

Urinary MicroRNA Profiling in the Nephropathy of Type 1 Diabetes

Background: Patients with Type 1 Diabetes (T1D) are particularly vulnerable to development of Diabetic nephropathy (DN) leading to End Stage Renal Disease. Hence a better understanding of the factors affecting kidney disease progression in T1D is urgently needed. In recent years microRNAs have emerged as important post-transcriptional regulators of gene expression in many different health conditions. We hypothesized that urinary microRNA profile of patients will differ in the different stages of diabetic renal disease. Methods and Findings: We studied urine microRNA profiles with qPCR in 40 T1D with >20 year follow up 10 who never developed renal disease (N) matched against 10 patients who went on to develop overt nephropathy (DN), 10 patients with intermittent microalbuminuria (IMA) matched against 10 patients with persistent (PMA) microalbuminuria. A Bayesian procedure was used to normalize and convert raw signals to expression ratios. We applied formal statistical techniques to translate fold changes to profiles of microRNA targets which were then used to make inferences about biological pathways in the Gene Ontology and REACTOME structured vocabularies. A total of 27 microRNAs were found to be present at significantly different levels in different stages of untreated nephropathy. These microRNAs mapped to overlapping pathways pertaining to growth factor signaling and renal fibrosis known to be targeted in diabetic kidney disease. Conclusions: Urinary microRNA profiles differ across the different stages of diabetic nephropathy. Previous work using experimental, clinical chemistry or biopsy samples has demonstrated differential expression of many of these microRNAs in a variety of chronic renal conditions and diabetes. Combining expression ratios of microRNAs with formal inferences about their predicted mRNA targets and associated biological pathways may yield useful markers for early diagnosis and risk stratification of DN in T1D by inferring the alteration of renal molecular processes. © 2013 Argyropoulos et al

Genetics of callous-unemotional behavior in children

Callous-unemotional behavior (CU) is currently under consideration as a subtyping index for conduct disorder diagnosis. Twin studies routinely estimate the heritability of CU as greater than 50%. It is now possible to estimate genetic influence using DNA alone from samples of unrelated individuals, not relying on the assumptions of the twin method. Here we use this new DNA method (implemented in a software package called Genome-wide Complex Trait Analysis, GCTA) for the first time to estimate genetic influence on CU. We also report the first genome-wide association (GWA) study of CU as a quantitative trait. We compare these DNA results to those from twin analyses using the same measure and the same community sample of 2,930 children rated by their teachers at ages 7, 9 and 12. GCTA estimates of heritability were near zero, even though twin analysis of CU in this sample confirmed the high heritability of CU reported in the literature, and even though GCTA estimates of heritability were substantial for cognitive and anthropological traits in this sample. No significant associations were found in GWA analysis, which, like GCTA, only detects additive effects of common DNA variants. The phrase ‘missing heritability’ was coined to refer to the gap between variance associated with DNA variants identified in GWA studies versus twin study heritability. However, GCTA heritability, not twin study heritability, is the ceiling for GWA studies because both GCTA and GWA are limited to the overall additive effects of common DNA variants, whereas twin studies are not. This GCTA ceiling is very low for CU in our study, despite its high twin study heritability estimate. The gap between GCTA and twin study heritabilities will make it challenging to identify genes responsible for the heritability of CU

Standard setting: Comparison of two methods

BACKGROUND: The outcome of assessments is determined by the standard-setting method used. There is a wide range of standard – setting methods and the two used most extensively in undergraduate medical education in the UK are the norm-reference and the criterion-reference methods. The aims of the study were to compare these two standard-setting methods for a multiple-choice question examination and to estimate the test-retest and inter-rater reliability of the modified Angoff method. METHODS: The norm – reference method of standard -setting (mean minus 1 SD) was applied to the 'raw' scores of 78 4th-year medical students on a multiple-choice examination (MCQ). Two panels of raters also set the standard using the modified Angoff method for the same multiple-choice question paper on two occasions (6 months apart). We compared the pass/fail rates derived from the norm reference and the Angoff methods and also assessed the test-retest and inter-rater reliability of the modified Angoff method. RESULTS: The pass rate with the norm-reference method was 85% (66/78) and that by the Angoff method was 100% (78 out of 78). The percentage agreement between Angoff method and norm-reference was 78% (95% CI 69% – 87%). The modified Angoff method had an inter-rater reliability of 0.81 – 0.82 and a test-retest reliability of 0.59–0.74. CONCLUSION: There were significant differences in the outcomes of these two standard-setting methods, as shown by the difference in the proportion of candidates that passed and failed the assessment. The modified Angoff method was found to have good inter-rater reliability and moderate test-retest reliability

Assessment of higher order cognitive skills in undergraduate education: modified essay or multiple choice questions? Research paper

Background: Reliable and valid written tests of higher cognitive function are difficult to produce, particularly for the assessment of clinical problem solving. Modified Essay Questions (MEQs) are often used to assess these higher order abilities in preference to other forms of assessment, including multiple-choice questions (MCQs). MEQs often form a vital component of end-of-course assessments in higher education. It is not clear how effectively these questions assess higher order cognitive skills. This study was designed to assess the effectiveness of the MEQ to measure higher-order cognitive skills in an undergraduate institution. Methods: An analysis of multiple-choice questions and modified essay questions (MEQs) used for summative assessment in a clinical undergraduate curriculum was undertaken. A total of 50 MCQs and 139 stages of MEQs were examined, which came from three exams run over two years. The effectiveness of the questions was determined by two assessors and was defined by the questions ability to measure higher cognitive skills, as determined by a modification of Bloom's taxonomy, and its quality as determined by the presence of item writing flaws. Results: Over 50% of all of the MEQs tested factual recall. This was similar to the percentage of MCQs testing factual recall. The modified essay question failed in its role of consistently assessing higher cognitive skills whereas the MCQ frequently tested more than mere recall of knowledge. Conclusion: Construction of MEQs, which will assess higher order cognitive skills cannot be assumed to be a simple task. Well-constructed MCQs should be considered a satisfactory replacement for MEQs if the MEQs cannot be designed to adequately test higher order skills. Such MCQs are capable of withstanding the intellectual and statistical scrutiny imposed by a high stakes exit examination.Edward J Palmer, Peter G Devit