Using item response theory to explore the psychometric properties of extended matching questions examination in undergraduate medical education

BACKGROUND: As assessment has been shown to direct learning, it is critical that the examinations developed to test clinical competence in medical undergraduates are valid and reliable. The use of extended matching questions (EMQ) has been advocated to overcome some of the criticisms of using multiple-choice questions to test factual and applied knowledge. METHODS: We analysed the results from the Extended Matching Questions Examination taken by 4th year undergraduate medical students in the academic year 2001 to 2002. Rasch analysis was used to examine whether the set of questions used in the examination mapped on to a unidimensional scale, the degree of difficulty of questions within and between the various medical and surgical specialties and the pattern of responses within individual questions to assess the impact of the distractor options. RESULTS: Analysis of a subset of items and of the full examination demonstrated internal construct validity and the absence of bias on the majority of questions. Three main patterns of response selection were identified. CONCLUSION: Modern psychometric methods based upon the work of Rasch provide a useful approach to the calibration and analysis of EMQ undergraduate medical assessments. The approach allows for a formal test of the unidimensionality of the questions and thus the validity of the summed score. Given the metric calibration which follows fit to the model, it also allows for the establishment of items banks to facilitate continuity and equity in exam standards

Prevalence and predictors of video game addiction: a study based on a national representative sample of gamers

Video gaming has become a popular leisure activity in many parts of the world, and an increasing number of empirical studies examine the small minority that appears to develop problems as a result of excessive gaming. This study investigated prevalence rates and predictors of video game addiction in a sample of gamers, randomly selected from the National Population Registry of Norway (N =3389). Results showed there were 1.4 % addicted gamers, 7.3 % problem gamers, 3.9 % engaged gamers, and 87.4 % normal gamers. Gender (being male) and age group (being young) were positively associated with addicted-, problem-, and engaged gamers. Place of birth (Africa, Asia, South- and Middle America) were positively associated with addicted- and problem gamers. Video game addiction was negatively associated with conscientiousness and positively associated with neuroticism. Poor psychosomatic health was positively associated with problem- and engaged gaming. These factors provide insight into the field of video game addiction, and may help to provide guidance as to how individuals that are at risk of becoming addicted gamers can be identified

Rasch analysis of the Psychiatric Out-Patient Experiences Questionnaire (POPEQ)

<p>Abstract</p> <p>Background</p> <p>The Psychiatric Out-Patient Experiences Questionnaire (POPEQ) is an 11-item core measure of psychiatric out-patients experiences of the perceived outcome of the treatment, the quality of interaction with the clinician, and the quality of information provision. The POPEQ was found to have evidence for reliability and validity following the application of classical test theory but has not previously been assessed by Rasch analysis.</p> <p>Methods</p> <p>Two national postal surveys of psychiatric outpatients took place in Norway in 2004 and 2007. The performance of the POPEQ, including item functioning and differential item functioning, was assessed by Rasch analysis. Principal component analysis of item residuals was used to assess the presence of subdimensions.</p> <p>Results</p> <p>6,677 (43.3%) and 11,085 (35.2%) psychiatric out patients responded to the questionnaire in 2004 and 2007, respectively. All items in the scale were retained after the Rasch analysis. The resulting scale had reasonably good fit to the Rasch model. The items performed the same for the two survey years and there was no differential item functioning relating to patient characteristics. Principal component analysis of the residuals confirmed that the measure to a high degree is unidimensional. However, the data also reflects three potential subscales, each relating to one of the three included aspects of health care.</p> <p>Conclusions</p> <p>The POPEQ had excellent psychometric properties and Rasch analysis further supported the construct validity of the scale by also identifying the three subdimensions originally included as components in the instrument development. The 11-item instrument is recommended in future research on psychiatric out-patient experiences. Future development may lead to the construction of more precise measures of the three subdomains that the POPEQ is based on.</p

Alcohol and cannabis use among adolescents in Flemish secondary school in Brussels: effects of type of education

<p>Abstract</p> <p>Background</p> <p>Research regarding socio-economic differences in alcohol and drug use in adolescence yields mixed results. This study hypothesizes that (1) when using education type as a proxy of one's social status, clear differences will exist between students from different types of education, regardless of students' familial socio-economic background; (2) and that the effects of education type differ according to their cultural background.</p> <p>Methods</p> <p>Data from the Brussels youth monitor were used, a school survey administered among 1,488 adolescents from the 3rd to 6th year of Flemish secondary education. Data were analyzed using multilevel logistic regression models.</p> <p>Results</p> <p>Controlling for their familial background, the results show that native students in lower educational tracks use alcohol and cannabis more often than students in upper educational tracks. Such a relationship was not found for students from another ethnic background.</p> <p>Conclusion</p> <p>Results from this study indicate that research into health risks should take into account both adolescents' familial background and individual social position as different components of youngsters' socio-economic background.</p

Do agreements between adolescent and parent reports on family socioeconomic status vary with household financial stress?

<p>Abstract</p> <p>Background</p> <p>Many studies compared the degree of concordance between adolescents' and parents' reports on family socioeconomic status (SES). However, none of these studies analyzed whether the degree of concordance varies by different levels of household financial stress. This research examines whether the degree of concordance between adolescents' and parent reports for the three traditional SES measures (parental education, parental occupation and household income) varied with parent-reported household financial stress and relative standard of living.</p> <p>Methods</p> <p>2,593 adolescents with a mean age of 13 years, and one of their corresponding parents from the Taiwan Longitudinal Youth Project conducted in 2000 were analyzed. Consistency of adolescents' and parents' reports on parental educational attainment, parental occupation and household income were examined by parent-reported household financial stress and relative standard of living.</p> <p>Results</p> <p>Parent-reported SES variables are closely associated with family financial stress. For all levels of household financial stress, the degree of concordance between adolescent's and parent's reports are highest for parental education (κ ranging from 0.87 to 0.71) followed by parental occupation (κ ranging from 0.50 to 0.34) and household income (κ ranging from 0.43 to 0.31). Concordance for father's education and parental occupation decreases with higher parent-reported financial stress. This phenomenon was less significant for parent-reported relative standard of living.</p> <p>Conclusions</p> <p>Though the agreement between adolescents' and parents' reports on the three SES measures is generally judged to be good in most cases, using adolescents reports for family SES may still be biased if analysis is not stratified by family financial stress.</p

Internal construct validity of the Shirom-Melamed Burnout Questionnaire (SMBQ)

<p>Abstract</p> <p>Background</p> <p>Burnout is a mental condition defined as a result of continuous and long-term stress exposure, particularly related to psychosocial factors at work. This paper seeks to examine the psychometric properties of the Shirom-Melamed Burnout Questionnaire (SMBQ) for validation of use in a clinical setting.</p> <p>Methods</p> <p>Data from both a clinical (319) and general population (319) samples of health care and social insurance workers were included in the study. Data were analysed using both classical and modern test theory approaches, including Confirmatory Factor Analysis (CFA) and Rasch analysis.</p> <p>Results</p> <p>Of the 638 people recruited into the study 416 (65%) persons were working full or part time. Data from the SMBQ failed a CFA, and initially failed to satisfy Rasch model expectations. After the removal of 4 of the original items measuring tension, and accommodating local dependency in the data, model expectations were met. As such, the total score from the revised scale is a sufficient statistic for ascertaining burnout and an interval scale transformation is available. The scale as a whole was perfectly targeted to the joint sample. A cut point of 4.4 for severe burnout was chosen at the intersection of the distributions of the clinical and general population.</p> <p>Conclusion</p> <p>A revised 18 item version of the SMBQ satisfies modern measurement standards. Using its cut point it offers the opportunity to identify potential clinical cases of burnout.</p

The Imperforate Anus Psychosocial Questionnaire (IAPSQ): Its construction and psychometric properties

The origin of the present study was to develop the liaison work between the disciplines of child and adolescent psychiatry and paediatric surgery and nursing, so as to improve the quality of treatment and care of a group of children with imperforate anus (IA) and their families. Imperforate anus is a congenital disease involving a deformity of the anorectum. The early surgery and invasive follow-up treatment associated with IA may affect the child psychosocially, including the child-parent relationship. By developing and testing a questionnaire for children born with anorectal anomalies, a tool for measuring psychosocial functioning can be realized

Randomised controlled trial of school-based humanistic counselling for emotional distress in young people: Feasibility study and preliminary indications of efficacy

The purpose of this study was to test the feasibility of a randomised controlled trial comparing six weeks of humanistic school-based counselling versus waiting list in the reduction of emotional distress in young people, and to obtain initial indications of efficacy. Following a screening procedure, young people (13 - 15 years old) who experienced emotional distress were randomised to either humanistic counselling or waiting list in this multi-site study. Outcomes were assessed using a range of self-report mental health measures, with the emotional symptoms subscale of the Strengths and Difficulties Questionnaire (SDQ) acting as the primary outcome indicator. Recruitment procedures were successful, with 32 young people consenting to participate in the trial and 27 completing endpoint measures. Trial procedures were acceptable to all involved in the research. No significant differences were found between the counselling and waiting list groups in reductions in levels of emotional symptoms (Hedges' g = 0.03), but clients allocated to counselling showed significantly greater improvement in prosocial behaviour (g = 0.89) with an average effect size (g) across the nine outcome measures of 0.25. Participants with higher levels of depressive symptoms showed significantly greater change. This study suggested that a randomised controlled trial of counselling in schools is acceptable and feasible, although initial indications of efficacy are mixed

Development and testing of the Measure of Innovation-Specific Implementation Intentions (MISII) using Rasch measurement theory

© 2018 The Author(s). Background: Implementation is proposed to be a multiphase, multilevel process. After a period of exploration, an adoption decision is made, typically at the upper management or policy level. Nevertheless, movement through each of the subsequent phases of the implementation process involves clinicians or providers at the individual level to adopt the innovation and then change their behavior to use/deliver the innovation. Multiple behavioral change theories propose that intentions are a critical determinant of implementation behavior. However, there is a need for the development and testing of pragmatic measures of providers' intentions to use a specific innovation or evidence-based practice (EBP). Methods: Nine items were developed to assess providers' intentions to use a specific innovation or EBP. Motivational interviewing was the EBP in the study. Items were administered, as part of larger survey, to 179 providers across 38 substance use disorder treatment (SUDT) programs within five agencies in California, USA. Rasch analysis was conducted using RUMM2030 software to assess the items, their overall fit to the Rasch model, the response scale used, individual item fit, differential item functioning (DIF), and person separation. Results: Following a stepwise process, the scale was reduced from nine items to three items to increase the feasibility and acceptability of the scale while maintaining suitable psychometric properties. The three-item unidimensional scale showed good person separation (PSI =.872), no disordering of thresholds, and no evidence of uniform or non-uniform DIF. Rasch analysis supported the viability of the scale as a measure of implementation intentions. Conclusions: The Measure of Innovation-Specific Implementation Intentions (MISII) is a sound measure of providers' intentions to use a specific innovation or EBP. Future evaluation of convergent, divergent, and predictive validity are needed. The study also demonstrates the value of Rasch analysis for testing the psychometric properties of pragmatic implementation measures

An evaluation of the structural validity of the Shoulder Pain and Disability Index (SPADI) using the Rasch model

Purpose: The Shoulder Pain and Disability Index (SPADI) has been extensively evaluated for its psychometric properties using classic test theory (CTT). The purpose of this study was to evaluate its structural validity using Rasch model analysis. Methods: Responses to the SPADI from 1030 patients referred for physiotherapy with shoulder pain and enrolled in a prospective cohort study were available for Rasch model analysis. Overall fit, individual person and item fit, response format, dependence, unidimensionality, targeting, reliability and differential item functioning (DIF) were examined. Results: The SPADI pain subscale initially demonstrated a misfit due to DIF by age and gender. After iterative analysis it showed good fit to the Rasch model with acceptable targeting and unidimensionality (overall fit (chi-square statistic 57.2, p=0.1); mean item fit residual 0.19 (1.5) and mean person fit residual 0.44 (1.1); person separation index (PSI) of 0.83). The disability subscale however shows significant misfit due to uniform DIF even after iterative analyses were used to explore different solutions to the sources of misfit (overall fit (chi-square statistic 57.2, p=0.1); mean item fit residual -0.54 (1.26) and mean person fit residual -0.38 (1.0); PSI 0.84). Conclusions: Rasch Model analysis of the SPADI has identified some strengths and limitations not previously observed using CTT methods. The SPADI should be treated as two separate subscales. The SPADI is a widely used outcome measure in clinical practice and research, however the scores derived from it must be interpreted with caution. The pain subscale fits the Rasch model expectations well. The disability subscale does not fit the Rasch model and its current format does not meet the criteria for true interval-level measurement required for use as a primary endpoint in clinical trials. Clinicians should therefore exercise caution when interpreting score changes on the disability subscale and attempt to compare their scores to age and sex stratified data