Item Response Theory for Peer Assessment

As an assessment method based on a constructivist approach, peer assessment has become popular in recent years. However, in peer assessment, a problem remains that reliability depends on the rater characteristics. For this reason, some item response models that incorporate rater parameters have been proposed. Those models are expected to improve the reliability if the model parameters can be estimated accurately. However, when applying them to actual peer assessment, the parameter estimation accuracy would be reduced for the following reasons. 1) The number of rater parameters increases with two or more times the number of raters because the models include higher-dimensional rater parameters. 2) The accuracy of parameter estimation from sparse peer assessment data depends strongly on hand-tuning parameters, called hyperparameters. To solve these problems, this article presents a proposal of a new item response model for peer assessment that incorporates rater parameters to maintain as few rater parameters as possible. Furthermore, this article presents a proposal of a parameter estimation method using a hierarchical Bayes model for the proposed model that can learn the hyperparameters from data. Finally, this article describes the effectiveness of the proposed method using results obtained from a simulation and actual data experiments

An Item Response Theory for Peer Assessment

Non

Applying Item Response Theory (IRT) Modeling to an Observational Measure of Childhood Pragmatics: The Pragmatics Observational Measure-2

Assessment of pragmatic language abilities of children is important across a number of childhood developmental disorders including ADHD, language impairment and Autism Spectrum Disorder. The Pragmatics Observational Measure (POM) was developed to investigate children's pragmatic skills during play in a peer-peer interaction. To date, classic test theory methodology has reported good psychometric properties for this measure, but the POM has yet to be evaluated using item response theory. The aim of this study was to evaluate the POM using Rasch analysis. Person and item fit statistics, response scale, dimensionality of the scale and differential item functioning were investigated. Participants included 342 children aged 5-11 years from New Zealand; 108 children with ADHD were playing with 108 typically developing peers and 126 typically developing age, sex and ethnic matched peers played in dyads in the control group. Video footage of this interaction was recorded and later analyzed by an independent rater unknown to the children using the POM. Rasch analysis revealed that the rating scale was ordered and used appropriately. The overall person (0.97) and item (0.99) reliability was excellent. Fit statistics for four individual items were outside acceptable parameters and were removed. The dimensionality of the measure showed two distinct elements (verbal and non-verbal pragmatic language) of a unidimensional construct. These findings have led to a revision of the first edition of POM, now called the POM-2. Further empirical work investigating the responsiveness of the POM-2 and its utility in identifying pragmatic language impairments in other childhood developmental disorders is recommended

Utilizing Multidimensional Item Response Theory to Examine Social Capital

This study introduces the use of mixed-format multidimensional item response theory (MIRT) analysis for assessing the latent factor structure of the Social Capital Rating Scale (SCRS). The rating scale, an instrument developed for measuring the parent involvement and the peer network of high school students, contains twenty-two items selected from the student questionnaire of the National Educational Longitudinal Study (NELS:88) Database. The psychometric properties and dimensionality of the scale are evaluated with MIRT framework. Using the sample of grade eight students from NELS:88 (N = 27,394), the two-factor structure of the SCRS (factor 1: within-family social capital; factor 2: peer social capital) was confirmed with the two-parameter IRT model (2PL): RMSEA = 0.049, CFI = 0.915, TLI = 0.905, SRMSR = 0.057. Discussion includes methodological implications for social capital scale, focusing on assessment of measurement invariance in differential item functioning etc

Factor validation and Rasch analysis of the individual recovery outcomes counter

Objective: The Individual Recovery Outcomes Counter is a 12-item personal recovery self assessment tool for adults with mental health problems. Although widely used across Scotland, limited research into its psychometric properties has been conducted. We tested its' measurement properties to ascertain the suitability of the tool for continued use in its present form.Materials and methods: Anonymised data from the assessments of 1,743 adults using mental health services in Scotland were subject to tests based on principles of Rasch measurement theory, principal components analysis and confirmatory factor analysis.Results: Rasch analysis revealed that the 6-point response structure of the Individual Recovery Outcomes Counter was problematic. Re-scoring on a 4-point scale revealed well ordered items that measure a single, recovery-related construct, and has acceptable fit statistics. Confirmatory factor analysis supported this. Scale items covered around 75% of the recovery continuum; those individuals least far along the continuum were least well addressed.Conclusions: A modified tool worked well for many, but not all, service users. The study suggests specific developments are required if the Individual Recovery Outcomes Counter is to maximise its' utility for service users and provide meaningful data for service providers.*Implications for Rehabilitation*Agencies and services working with people with mental health problems aim to help them with their recovery.*The individual recovery outcomes counter has been developed and is used widely in Scotland to help service users track their progress to recovery.*Using a large sample of routinely collected data we have demonstrated that a number of modifications are needed if the tool is to adequately measure recovery.*This will involve consideration of the scoring system, item content and inclusion, and theoretical basis of the tool

Examining \u3cem\u3eDSM\u3c/em\u3e Criteria for Trichotillomania in A Dimensional Framework: Implications for \u3cem\u3eDSM-5\u3c/em\u3e And Diagnostic Practice

Background: Diagnosis of Trichotillomania (TTM) requires meeting several criteria that aim to embody the core pathology of the disorder. These criteria are traditionally interpreted monothetically, in that they are all equally necessary for diagnosis. Alternatively, a dimensional conceptualization of psychopathology allows for examination of the relatedness of each criterion to the TTM latent continuum. Objectives: First, to examine the ability of recently removed criteria (B and C) to identify the latent dimensions of TTM psychopathology, such that they discriminate between individuals with low and high degrees of hair pulling severity. Second, to determine the impact of removing criteria B and C on the information content of remaining diagnostic criteria. Third, to determine the psychometric properties of remaining TTM diagnostic criteria that remain largely unchanged in DSM-5; that is, whether they measure distinct or overlapping levels of TTM psychopathology. Fourth, to determine whether information content derived from diagnostic criteria aid in the prediction of disease trajectory (i.e., can relapse propensity be predicted from criteria endorsement patterns). Method: Statistics derived from Item Response Theory were used to examine diagnostic criteria endorsement in 91 adults with TTM who underwent psychotherapy. Results: The removal of two criteria in DSM-5 and psychometric validity of remaining criteria was supported. Additionally, individual trait parameters were used to predict treatment progress, uncovering predictive power where none previously existed. Conclusions: Diagnostic criteria for TTM should be examined in dimensional models, which allow for nuanced and sensitive measurement of core symptomology in treatment contexts

Applying Item Response Theory (IRT) Modeling to an Observational Measure of Childhood Pragmatics: The Pragmatics Observational Measure-2

Assessment of pragmatic language abilities of children is important across a number of childhood developmental disorders including ADHD, language impairment and Autism Spectrum Disorder. The Pragmatics Observational Measure (POM) was developed to investigate children’s pragmatic skills during play in a peer–peer interaction. To date, classic test theory methodology has reported good psychometric properties for this measure, but the POM has yet to be evaluated using item response theory. The aim of this study was to evaluate the POM using Rasch analysis. Person and item fit statistics, response scale, dimensionality of the scale and differential item functioning were investigated. Participants included 342 children aged 5–11 years from New Zealand; 108 children with ADHD were playing with 108 typically developing peers and 126 typically developing age, sex and ethnic matched peers played in dyads in the control group. Video footage of this interaction was recorded and later analyzed by an independent rater unknown to the children using the POM. Rasch analysis revealed that the rating scale was ordered and used appropriately. The overall person (0.97) and item (0.99) reliability was excellent. Fit statistics for four individual items were outside acceptable parameters and were removed. The dimensionality of the measure showed two distinct elements (verbal and non-verbal pragmatic language) of a unidimensional construct. These findings have led to a revision of the first edition of POM, now called the POM-2. Further empirical work investigating the responsiveness of the POM-2 and its utility in identifying pragmatic language impairments in other childhood developmental disorders is recommended

Development of the Motivational Interviewing Supervision and Training Scale

The movement to use empirically supported treatments has increased the need for researchers and supervisors to evaluate therapists’ adherence to and the quality with which they implement those interventions. Few empirically supported approaches exist for providing these types of evaluations. This is also true for motivational interviewing, an empirically supported intervention important in the addictions field. This study describes the development and psychometric evaluation of the Motivational Interviewing Supervision and Training Scale (MISTS), a measure intended for use in training and supervising therapists implementing motivational interviewing. Satisfactory interrater reliability was found (generalizability coefficient p2 = .79), and evidence was found supporting the convergent and discriminant validity of the MISTS. Recommendations for refinement of the measure and future research are discussed

Addressing Youth Perceptions of Harm in Marijuana Prevention Programming

The inverse relationship between perception of harm and substance use is clearly supported by decades of research – youth are less likely to engage in substance use when it is seen as harmful. However, despite strong theoretical and practical reasons to focus on perception of harm as a change-producer in prevention programming, little is known about what is effective in impacting perception of harm for youth marijuana use. To investigate the impact of existing prevention efforts designed to influence youth perception of harm and, consequently, youth marijuana use, we reviewed seven privately- or federally-funded online registries (e.g., Blueprints for Healthy Youth Development, Substance Abuse and Mental Health Administration’s National Registry of Evidence Based Programs and Practices) to identify evidence-based programs with marijuana-related outcomes for youth. We found 36 registry-identified programs with demonstrated impact on youth marijuana use. Although many of these programs may have actively or passively sought to alter perception of harm, only ten measured marijuana- or drug-related perception of harm as an intermediate outcome. Drawing on the commonalities of evidence-based programs with significant impacts on youth marijuana perception of harm, as well as lessons learned from other health behavior change efforts, we recommend best practices to provide state and local decision-makers with information on altering youth perception of harm for marijuana and on evaluating the impact of these efforts