Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial

Many research designs require the assessment of inter-rater reliability (IRR) to demonstrate consistency among observational ratings provided by multiple coders. However, many studies use incorrect statistical procedures, fail to fully report the information necessary to interpret their results, or do not address how IRR affects the power of their subsequent analyses for hypothesis testing. This paper provides an overview of methodological issues related to the assessment of IRR with a focus on study design, selection of appropriate statistics, and the computation, interpretation, and reporting of some commonly-used IRR statistics. Computational examples include SPSS and R syntax for computing Cohens kappa and intra-class correlations to assess IRR

Conducting Simulation Studies in the R Programming Environment

Simulation studies allow researchers to answer specific questions about data analysis, statistical power, and best-practices for obtainingaccurate results in empirical research. Despite the benefits that simulation research can provide, many researchers are unfamiliar with available tools for conducting their own simulation studies. The use of simulation studies need not be restricted toresearchers with advanced skills in statistics and computer programming, and such methods can be implemented by researchers with a variety of abilities and interests. The present paper provides an introduction to methods used for running simulationstudies using the R statistical programming environment and is written for individuals with minimal experience running simulation studies or using R. The paper describes the rationale and benefits of using simulations and introduces R functions relevant for many simulation studies. Three examples illustrate different applications for simulation studies, including (a) the use of simulations to answer a novel question about statistical analysis, (b) the use of simulations to estimate statistical power, and (c) the use of simulations to obtain confidence intervals of parameter estimates throughbootstrapping. Results and fully annotated syntax from these examples are provided

Proceedings of the 3rd Biennial Conference of the Society for Implementation Research Collaboration (SIRC) 2015: advancing efficient methodologies through community partnerships and team science

It is well documented that the majority of adults, children and families in need of evidence-based behavioral health interventionsi do not receive them [1, 2] and that few robust empirically supported methods for implementing evidence-based practices (EBPs) exist. The Society for Implementation Research Collaboration (SIRC) represents a burgeoning effort to advance the innovation and rigor of implementation research and is uniquely focused on bringing together researchers and stakeholders committed to evaluating the implementation of complex evidence-based behavioral health interventions. Through its diverse activities and membership, SIRC aims to foster the promise of implementation research to better serve the behavioral health needs of the population by identifying rigorous, relevant, and efficient strategies that successfully transfer scientific evidence to clinical knowledge for use in real world settings [3]. SIRC began as a National Institute of Mental Health (NIMH)-funded conference series in 2010 (previously titled the “Seattle Implementation Research Conference”; $150,000 USD for 3 conferences in 2011, 2013, and 2015) with the recognition that there were multiple researchers and stakeholdersi working in parallel on innovative implementation science projects in behavioral health, but that formal channels for communicating and collaborating with one another were relatively unavailable. There was a significant need for a forum within which implementation researchers and stakeholders could learn from one another, refine approaches to science and practice, and develop an implementation research agenda using common measures, methods, and research principles to improve both the frequency and quality with which behavioral health treatment implementation is evaluated. SIRC’s membership growth is a testament to this identified need with more than 1000 members from 2011 to the present.ii SIRC’s primary objectives are to: (1) foster communication and collaboration across diverse groups, including implementation researchers, intermediariesi, as well as community stakeholders (SIRC uses the term “EBP champions” for these groups) – and to do so across multiple career levels (e.g., students, early career faculty, established investigators); and (2) enhance and disseminate rigorous measures and methodologies for implementing EBPs and evaluating EBP implementation efforts. These objectives are well aligned with Glasgow and colleagues’ [4] five core tenets deemed critical for advancing implementation science: collaboration, efficiency and speed, rigor and relevance, improved capacity, and cumulative knowledge. SIRC advances these objectives and tenets through in-person conferences, which bring together multidisciplinary implementation researchers and those implementing evidence-based behavioral health interventions in the community to share their work and create professional connections and collaborations

Remotely Assessing Mechanisms of Behavioral Change in Community Substance Use Disorder Treatment to Facilitate Measurement-Informed Care: Pilot Longitudinal Questionnaire Study

BackgroundResearch shows that improvements in coping strategies, abstinence self-efficacy, craving, and depression are potential mechanisms of behavioral change (MOBC) in treatments for substance use disorders (SUDs). However, little is known about how these insights regarding MOBC can be applied to SUD treatment settings. One way to facilitate MOBC-informed care in frontline settings could be to measure and monitor changes in MOBC throughout treatment using brief, frequent questionnaires that patients complete by using mobile technologies (eg, smartphones). The results derived from these questionnaires could potentially be used for clinical monitoring (ie, measurement-based care) to better understand whether individual patients are experiencing treatment-related improvements on key clinical targets. ObjectiveThis study evaluated whether brief, weekly MOBC questionnaires completed by patients remotely can potentially provide clinically meaningful information about changes in MOBC in the context of real-world, community-based SUD treatment. MethodsA total of 30 patients (14/30, 47% female; 13/30, 43% racial or ethnic minority) in a community SUD treatment clinic participated in a pilot study where they were invited to complete brief, weekly questionnaires that assessed various MOBC, including coping strategies, abstinence self-efficacy, craving, depression, and therapeutic alliance. Questionnaires were typically completed remotely via smartphone for up to 6 months; 618 questionnaires were completed in total. Participants also completed longer, psychometrically validated measures of the same MOBC at baseline and 6-month research appointments. Statistical analyses tested whether brief, weekly, remotely completed MOBC questionnaires exhibited characteristics that would be desirable for real-world longitudinal clinical monitoring, including a tendency to detect within-person changes in MOBC over time; cross-sectional and longitudinal associations with longer, psychometrically validated measures completed at research appointments; and similar patterns of associations with 6-month percentage of days abstinent as longer, psychometrically validated MOBC measures completed at research appointments. ResultsThe results of this study indicated that the brief, weekly, remotely completed MOBC measures exhibited characteristics that are desirable for clinical monitoring, including a tendency to vary longitudinally (within patients over time) more often than measures of alcohol and drug consumption, generally having medium to large cross-sectional and longitudinal correlations with longer psychometrically validated measures of MOBC completed at research appointments, and generally having similar patterns of association with 6-month percentage of days abstinent from alcohol and drugs as longer psychometrically validated MOBC measures completed at research appointments. ConclusionsThe results of this pilot study provide initial evidence that incorporating brief, weekly, and remotely completed MOBC questionnaires into community SUD treatment may be a viable approach for facilitating MOBC-informed care. Such questionnaires can potentially support measurement-based care by providing meaningful information about within-patient changes in clinical domains that are often directly targeted in SUD treatments and predict long-term substance use outcomes

Psychometric Properties of the Important People Instrument With College Student Drinkers

Alcohol use in college students is associated with a number of negative consequences, and specific characteristics of college students’ social networks have been linked to hazardous or harmful alcohol use. College represents a time when social networks are changing significantly, both in terms of composition and relative influence of network members. However, the reliability and validity of one of the most common measures for assessing alcohol-specific social support, the Important People Instrument (IPI; Clifford & Longabaugh, 1991), have not been established in college student samples. The aim of the current study was to examine the psychometric properties of the IPI administered in computerized and paper-and-pencil formats within a non-treatment-seeking sample of college drinkers (N = 197). Test-retest reliability estimates for the overall sample indicated that all indices had acceptable reliability. While the models tested within a confirmatory factor analysis (CFA) framework exhibited mediocre fit, a three-factor model appeared to offer the best fit overall. Consistent with previous findings with treatment-seeking samples, negative consequences of drinking were positively associated with network drinking behavior, but not network support for drinking or network general support. The IPI can provide reliable and valid information about network drinking behavior, network support for drinking, and general support among college students, although there is room for improvement in the measure. Future research may improve the measurement of alcohol-specific social support by measuring additional domains of the construct or by taking more contextualized approaches

Psychometric properties of the Important People Instrument with college student drinkers.

Alcohol use in college students is associated with a number of negative consequences, and specific characteristics of college students’ social networks have been linked to hazardous or harmful alcohol use. College represents a time when social networks are changing significantly, both in terms of composition and relative influence of network members. However, the reliability and validity of one of the most common measures for assessing alcohol-specific social support, the Important People Instrument (IPI; Clifford & Longabaugh, 1991), have not been established in college student samples. The aim of the current study was to examine the psychometric properties of the IPI administered in computerized and paper-and-pencil formats within a non-treatment-seeking sample of college drinkers (N = 197). Test-retest reliability estimates for the overall sample indicated that all indices had acceptable reliability. While the models tested within a confirmatory factor analysis (CFA) framework exhibited mediocre fit, a three-factor model appeared to offer the best fit overall. Consistent with previous findings with treatment-seeking samples, negative consequences of drinking were positively associated with network drinking behavior, but not network support for drinking or network general support. The IPI can provide reliable and valid information about network drinking behavior, network support for drinking, and general support among college students, although there is room for improvement in the measure. Future research may improve the measurement of alcohol-specific social support by measuring additional domains of the construct or by taking more contextualized approaches

Trajectories of drinking urges and the initiation of abstinence during cognitive-behavioral alcohol treatment

BACKGROUND AND AIMS: Drinking urges during treatment for alcohol use disorders (AUDs) are common, can cause distress and predict relapse. Clients may have little awareness of how their drinking urges might be expected to change during AUD treatment in general and in response to initiating abstinence. The aim of the present study was to test whether drinking urges change on a daily level during treatment and after initiating abstinence. DESIGN: Secondary data analysis was performed using daily drinking urge ratings from two randomized clinical trials. SETTING AND PARTICIPANTS: Women (n = 98) and men (n = 79) with AUDs in separate clinical trials of out-patient AUD-focused cognitive-behavioral therapy. MEASUREMENTS: Daily dichotomous indicators of any drinking urges or acute escalations in urges (i.e. at least two more urges compared with the previous day) were examined using generalized linear mixed growth-curve modeling. FINDINGS: Participants who initiated abstinence reported reductions in urges immediately thereafter (log odds ratios: women B = -0.701, P \u3c 0.001; men B = -0.628, P = 0.018), followed by additional, gradual reductions over time (women B = -0.118, P \u3c 0.001; men B = -0.141, P \u3c 0.001). Participants who entered treatment abstaining from alcohol also reported significant reductions in urges over time (women B = -0.147, P \u3c 0.001; men B = -0.142, P \u3c 0.001). Participants who drank throughout treatment had smaller (women B = -0.042, P = 0.012) or no reductions in urges (men B = 0.015, P = 0.545). There was no evidence that urges increased systematically in response to initiating abstinence. CONCLUSIONS: Drinking urges during out-patient behavioral treatment for alcohol use disorders may be maintained in part by alcohol consumption. Initiating abstinence is associated with reductions in drinking urges immediately and then more gradually over time

Psychometric Properties of the Adolescent Reinforcement Survey Schedule-Alcohol Use Version with College Student Drinkers

Background: Behavioral economic theories of drinking posit that the reinforcing value of engaging in activities with versus without alcohol influences drinking behavior. Measures of the reinforcement value of drugs and alcohol have been used in previous research, but little work has examined the psychometric properties of these measures. Objectives: The present study aims to evaluate the factor structure, test-retest reliability, and concurrent validity of an alcohol-only version of the Adolescent Reinforcement Survey Schedule (ARSS-AUV). Methods: A sample of 157 college student drinkers completed the ARSS-AUV at two time points 2-3 days apart. Test-retest reliability, hierarchical factor analysis, and correlations with other drinking measures were examined. Results: Single, unidimensional general factors accounted for a majority of the variance in alcohol and alcohol-free reinforcement items. Residual factors emerged that typically represented alcohol or alcohol-free reinforcement while doing activities with friends, romantic or sexual partners, and family members. Individual ARSS-AUV items had fair-to-good test-retest reliability, while general and residual factors had excellent test-retest reliability. General alcohol reinforcement and alcohol reinforcement from friends and romantic partners were positively correlated with past-year alcohol consumption, heaviest drinking episode, and alcohol-related negative consequences. Alcohol-free reinforcement indices were unrelated to alcohol use or consequences. Conclusions/Importance: The ARSS-AUV appears to demonstrate good reliability and mixed concurrent validity among college student drinkers. The instrument may provide useful information about alcohol reinforcement from various activities and people and could provide clinically-relevant information for prevention and treatment programs

Understanding measures of racial discrimination and microaggressions among American Indian and Alaska Native college students in the Southwest United States

Abstract Background Racial discrimination, including microaggressions, contributes to health inequities, yet research on discrimination and microaggressions has focused on single measures without adequate psychometric evaluation. To address this gap, we examined the psychometric performance of three discrimination/microaggression measures among American Indian and Alaska Native (AI/AN) college students in a large Southwestern city. Methods Students (N = 347; 65% female; ages 18–65) completed the revised-Everyday Discrimination Scale, Microaggressions Distress Scale, and Experiences of Discrimination measure. The psychometric performance of these measures was evaluated using item response theory and confirmatory factor analyses. Associations of these measures with age, gender, household income, substance use, and self-rated physical health were examined. Results Discrimination and microaggression items varied from infrequently to almost universally endorsed and each measure was unidimensional and moderately correlated with the other two measures. Most items contributed information about the overall severity of discrimination and collectively provided information across a continuum from everyday microaggressions to physical assault. Greater exposure to discrimination on each measure had small but significant associations with more substance use, lower income, and poorer self-rated physical health. The Experiences of Discrimination measure included more severe forms of discrimination, while the revised-Everyday Discrimination Scale and the Microaggressions Distress Scale represented a wider range of severity. Conclusions In clinical practice, these measures can index varying levels of discrimination for AI/ANs, particularly for those in higher educational settings. This study also informs the measurement of racial discrimination and microaggressions more broadly