The Choices, Challenges, and Lessons Learned from a Multi-Method Social-Emotional / Character Assessment in and Out of School Time Setting

Out-of-School-Time (OST) programs are increasingly recognized as a venue to actively engage children and youth in character development activities, but little guidance exists as to how to assess individual children and youth in OST environments for the sake of evaluating their character development. This research brief uses an illustrative case study to reflect upon the experience of selecting and completing a strength-based, multi-modal social-emotional / character assessment that used a direct assessment and a multiple informant behavior rating scale in an OST setting. Insights derived from the case study reveal opportunities and challenges associated with each assessment modality. This paper shares lessons learned with those conducting individual assessments in OST environments and with those seeking to improve our capacity to complete screening, formative, and summative assessments of social-emotional and character constructs in OST youth development programs to help children

Predictors of rater bias in the assessment of social-emotional competence

The Devereux Student Strengths Assessment Mini (DESSA-Mini) (LeBuffe, Shapiro, & Naglieri, 2014) efficiently monitors the growth of Social-Emotional Competence (SEC) in the routine implementation of Social Emotional Learning programs. The DESSAMini is used to assess approximately half a million children around the world. Since behavior rating scales can have ‘rater bias’, this paper examines rater characteristics that contribute to DESSA-Mini ratings. Rater characteristics and DESSA-Mini ratings were collected from elementary school classroom teachers (n=72) implementing TOOLBOX in a racially/ethnically diverse California school district. Teachers rated 1,676 students, who scored similarly to a national reference group. Multilevel modeling analysis showed that only 16% of variance in DESSA-mini ratings was attributable to raters. Relationships between teacher characteristics and ratings were estimated to examine rater variance. Collectively, four characteristics of teachers (perceived barriers to student learning, sense of their ‘typical’ student’s level of SEC, anticipation of SEL program implementation challenges, and intentions to fully implement a newly adopted SEL program) accounted for bias in teacher-generated DESSA scores, leaving only 10% of the variance unexplained. Identified sources of ‘rater bias’ can be controlled for in research and addressed through thoughtful program selection, training, and implementation.peer-reviewe

The Choices, Challenges, and Lessons Learned from a Multi-Method Social-Emotional / Character Assessment in and Out of School Time Setting

Out-of-School-Time (OST) programs are increasingly recognized as a venue to actively engage children and youth in character development activities, but little guidance exists as to how to assess individual children and youth in OST environments for the sake of evaluating their character development. This research brief uses an illustrative case study to reflect upon the experience of selecting and completing a strength-based, multi-modal social-emotional / character assessment that used a direct assessment and a multiple informant behavior rating scale in an OST setting. Insights derived from the case study reveal opportunities and challenges associated with each assessment modality. This paper shares lessons learned with those conducting individual assessments in OST environments and with those seeking to improve our capacity to complete screening, formative, and summative assessments of social-emotional and character constructs in OST youth development programs to help children

Do Strengths Matter?: The Role of Strengths in Improving Mental Health Outcomes for Youth with Emotional and Behavioral Problems In Contact with a Public System of Care

The aim of this dissertation was to examine the extent of association, if any, between strengths (including those of the client, the client's family, and the client's environment) and psychiatric symptom severity for "youth" (children and adolescents aged 5-22) in contact with a mental health and/or substance abuse agency in a public System of Care. This dissertation examined a public systems dataset that included administrative assessment data for every client aged 5-22 (n=2049) who entered into a public System of Care from June 2010-August 2013. The assessment data was collected by clinicians at entry to the system (initial assessment), six months after initial assessment, and twelve months after initial assessment. Behavioral/social learning theory and related research in applied and experimental settings (e.g. Goldiamond, 1974; Madden, 2013; Staats, 2012) would suggest that growth in strengths would be associated with reduction in symptoms. However, time order cannot be determined with the current data and thus only associational relationships are discussed. This dissertation is a "three paper" dissertation including a dissertation introduction, three inter-related papers, and a dissertation conclusion. The dissertation introduction provides a background to the entire dissertation and is intended to orient to the reader to terminological and conceptual issues relevant to all three papers (each paper might also cover some of the literature included in the dissertation introduction, as relevant to the specific paper). Paper one reviews the literature on prevalence of emotional and behavioral problems among youth in the United States, orients the reader to the history and current policies of Systems of Care (SOC), and reviews the literature on strength-based assessment. Then, paper one reports the prevalence of clinician-reported strengths in the public System of Care, explores the psychometric properties of the assessment's Strengths section, and describes three alternative modeling options to explore group differences (by gender, age, and ethnicity) of youth strengths. Youth assessed at entry to the System of Care were reported to enter with considerable strengths, similar to other studies reporting on prevalence of strengths in a System of Care (e.g. CMHI, 2011; Dunleavey et al., 2011). The confirmatory factor analysis found that the hypothesized two factor model was a better fit for the data than a single factor model, though the fit of the two factor model was only adequate and the interpretation of the factors were questionable. Individual strength items were assessed as the most informative for future analyses, compared to the factors or the composite score. Paper two examines whether, at initial assessment, client strengths are associated with psychiatric symptoms. Higher levels of family relationships, peer relationships, and educational strengths were significantly associated with lower levels of psychiatric symptoms, after accounting for variation due to clinician, program, reported trauma exposure and youth demographics. Paper three reviews some studies that report on the relationship between strengths and mental health outcomes in Systems of Care samples. This paper then examines 1) the average change in psychiatric symptom severity twelve months after entry (initial assessment) to the System of Care, 2) whether initial strengths are associated with rate of change in psychiatric symptoms, 3) whether rate of change in strengths at six month and twelve month assessment are associated with rate of change in psychiatric symptoms, and 4) explores the contribution of variations related to clinician, youth, and time. Results suggest that self-reported growth in peer relationships, relationship permanence, and education strengths twelve months after entry to the System of Care was associated with larger reported reductions in symptoms at twelve months, adjusting for clustering, time, and youth demographics. Only associational relationships are reported; causal direction cannot be established in the current data. The percentage of youth referred for services, where they were referred from and to, the percentage who went to services, type, quantity, and quality of interventions offered and used (if any), and drop-out rates and reasons are not reliably tracked in the System of Care. This dissertation focuses on one aspect of service delivery in public mental health systems, namely the role of strengths, if any, in Systems of Care for youth with emotional and behavioral problems. The dissertation conclusion discusses overall themes that emerged, notes limitations relevant to the entire dissertation, and discusses implications for social workers and others

The Strengths of Youth in a Public Behavioral Health System: Measurement Choices, Prevalence Rates, and Group Differences.

Youth with severe emotional and behavioral problems receiving services in public behavioral health systems have strengths that are understudied in research and underutilized in practice. This study explores four alternative strategies (individual item scores, the number of "actionable" strengths, subscales, and a total composite) for summarizing the strengths of youth assessed with the Child and Adolescent Needs and Strengths (CANS) in a large, urban, public behavioral health system. The paper examines whether these summarization strategies produce divergent understandings of the prevalence of strengths across gender, age, and racial groups. Analyses suggest that youth enter this system with high levels of strengths. There are few group differences in strengths across the diverse summarization strategies. Though the practice-preferred method of using individual strengths items provides the most interpretable information about strengths, the aggregation strategies may be useful for programs and systems. Implications for policy and practice are discussed

The Strengths of Youth in a Public Behavioral Health System: Measurement Choices, Prevalence Rates, and Group Differences.

Youth with severe emotional and behavioral problems receiving services in public behavioral health systems have strengths that are understudied in research and underutilized in practice. This study explores four alternative strategies (individual item scores, the number of "actionable" strengths, subscales, and a total composite) for summarizing the strengths of youth assessed with the Child and Adolescent Needs and Strengths (CANS) in a large, urban, public behavioral health system. The paper examines whether these summarization strategies produce divergent understandings of the prevalence of strengths across gender, age, and racial groups. Analyses suggest that youth enter this system with high levels of strengths. There are few group differences in strengths across the diverse summarization strategies. Though the practice-preferred method of using individual strengths items provides the most interpretable information about strengths, the aggregation strategies may be useful for programs and systems. Implications for policy and practice are discussed