Translating the Statistical Representation of the Effects of Education Interventions Into More Readily Interpretable Forms

This paper is directed to researchers who conduct and report education intervention studies. Its purpose is to stimulate and guide them to go a step beyond reporting the statistics that emerge from their analysis of the differences between experimental groups on the respective outcome variables. With what is often very minimal additional effort, those statistical representations can be translated into forms that allow their magnitude and practical significance to be more readily understood by the practitioners, policymakers, and even other researchers who are interested in the intervention that was evaluated

Culturally sensitive substance use treatment for racial/ethnic minority youth: A meta-analytic review

This systematic review and meta-analysis synthesized findings from studies examining culturally sensitive substance use treatment for racial/ethnic minority youth. An extensive literature search located eight eligible studies using experimental or quasi-experimental designs. The meta-analysis quantitatively synthesized findings comparing seven culturally sensitive treatment conditions to seven alternative conditions on samples composed of at least 90% racial/ethnic minority youth. The results from the meta-analysis indicated that culturally sensitive treatments were associated with significantly larger reductions in post-treatment substance use levels relative to their comparison conditions (g=0.37, 95% CI [0.12, 0.62], k=7, total number participants=723). The average time between pretest and posttest was 21weeks (SD=11.79). There was a statistically significant amount of heterogeneity across the seven studies (Q=26.5, p=0.00, τ2=0.08, I2=77.4%). Differential effects were not statistically significant when contrasts were active generic counterparts of treatment conditions (direct “bona fide” comparisons; g=−0.08, 95% CI [−0.51, 0.35]) and ‘treatment as usual’ conditions (g=0.39, 95% CI [−0.14, 0.91]). Strong conclusions from the review were hindered by the small number of available studies for synthesis, variability in comparison conditions across studies, and lack of diversity in the adolescent clients served in the studies. Nonetheless, this review suggests that culturally sensitive treatments offer promise as an effective way to address substance use among racial/ethnic minority youth. •The effectiveness of culturally sensitive substance use treatments for racial/ethnic minority was examined.•Meta-analysis was used to synthesize findings from seven studies using controlled designs.•Culturally sensitive treatments were associated with greater reductions in substance use.•Comparison conditions varied considerably across studies.•Conclusions were hindered by the small number of studies available for synthesis

Post-basic technical and vocational education and training (TVET) interventions to improve employability and employment of TVET graduates in low- and middle-income countries: a systematic review

BACKGROUND The increase of low-income, low-skilled youth in the labour market, particularly in developing countries, is a major concern internationally. In some regions of the world, young people are nearly three times as likely as adults to be unemployed. They are also more likely to work in the informal labour market than adults, in low quality jobs that offer limited socio-economic security, training opportunities, and working conditions. This enormous unlocked potential represents a substantial loss of opportunity for both individuals and society. With increasing emphasis being given to work- and skills-based solutions to economic competition and poverty in the developing world, comes a renewed focus on technical and vocational education and training (TVET) as a means to expand opportunities for marginalised youth. Although several reviews have attempted to summarise the existing research in this area, there are a number of limitations to these reviews. There is a need to systematically examine the evidence base to provide a picture of the types of TVET interventions being used to raise employment, to identify those that are effective and ineffective, and to identify areas in which more research needs to be conducted. OBJECTIVES The main objective of this systematic review was to summarise the available evidence on the effects of TVET interventions for young people in developing countries to inform policy, practice, and research. The questions guiding this study were: - What are the effects of different models of technical and vocational education and training (TVET) interventions on the employment and employability outcomes of young people, aged 15-24 years, in low- and middle-income countries? - What do the findings suggest about moderating effects? SEARCH STRATEGY A systematic and comprehensive search was used to locate both published and unpublished studies. A wide range of major bibliographic databases were electronically searched, along with specialist and grey literature databases, and websites of relevant organisations. Reference lists of previous reviews and included studies were examined. In addition, we conducted forward citation checking exercises and attempted contact with authors and other relevant stakeholders. SELECTION CRITERIA Studies eligible for inclusion in the review were required to meet several eligibility criteria. First, studies must have evaluated a TVET intervention. Second, studies must have investigated outcomes for young people aged 15-24 years. Third, the geographical location of the studies must have been a low-or middle-income country. Fourth, studies must have utilised an experimental or quasi-experimental research design, including random assignment, quasi-random assignment (and groups generated were shown to be equivalent, or there was sufficient information to permit calculation of pre-treatment group equivalence), non-random assignment with matching, or non-random assignment with statistical controls. Fifth, studies must have reported at least one eligible outcome variable measuring employment (e.g., gaining paid employment) or employability (e.g., changing attitudes to work, or gaining job search skills). Finally, the date of publication or reporting of the study must have been between 2000 and 2011. No language restrictions were applied. DATA COLLECTION AND ANALYSIS The electronic literature search yielded a total of 8072 potentially relevant reports, 145 of which were retrieved for full-text screening and nine were judged relevant. Handsearching identified a further 46 eligible reports. A total of 30 studies, reported in 55 publications, met the eligibility criteria. However, due to resources limitations, four of the eight eligible Spanish language papers we identified were not included in the review. Of the 26 studies included in the review, 3 utilised a randomised controlled trial (RCT) design, and 23 utilised a quasi-experimental design. The studies were coded independently by pairs of reviewers using a structured coding tool. Descriptive analysis was undertaken to examine and describe data related to the characteristics of the included studies and interventions. Ten of the 26 studies had data that allowed calculation of effects sizes. The findings from these 10 studies were statistically combined using meta-analytic techniques. The effect sizes were calculated using the standardised mean difference, corrected for small sample bias (i.e., Hedges' g). Analysis of the mean effect size, the heterogeneity of effect sizes, and the relationship between effect size and characteristics of the studies, participants and interventions was conducted. RESULTS The 26 included studies assessed the effectiveness of 20 different TVET interventions from various countries in Latin America, the Caribbean, Europe, East Asia, South Asia and Sub-Saharan Africa. Publication dates ranged between 2001 and 2011. Study settings included ten upper-middle income countries (Argentina, Bosnia and Herzegovina, Brazil, Chile, China, Colombia, Dominican Republic, Latvia, Mexico, Panama and Peru); two lower-middle income countries (India and Bhutan); and one low-income country (Kenya). The following summary of evidence focuses on the results of the statistical analyses of 10 studies included in the review. Employment - The overall mean effect of TVET on paid employment was positive and significant; however, significant heterogeneity was observed (Q = 23.8; df = 7; p = 0.00124; I2 = 70.6%; tau2 = 0.0153). - Four variables were tested for moderating effects. - Evidence of a statistically significant relationship between study quality and effect size was observed (Qb = 6.49; p = 0.0108). It is reasonable, therefore, to conclude that the overall mean effect may be inflated and that our conclusions about treatment effect on paid employment should be based only on those studies rated medium quality (g=0.06; 95% CI [-0.01, 0.12]). - No significant differences in mean effects were observed between studies according to type of TVET intervention (Qb = 1.43; p = 0.231), length of follow-up period (Qb = 0.273; p = 0.601), or gender (Qb = 2.1; p = 0.147). Formal employment The overall mean effect of TVET on formal employment was positive and significant; however, significant heterogeneity was observed (Q = 11.1; df = 4; p = 0.0256; I2 = 63.9%; tau2 = 0.0131). - One variable was tested for moderating effects. - Evidence of a statistically significant relationship between study quality and effect size was observed (Qb = 10.6; p = 0.00116). It is reasonable, therefore, to conclude that the overall mean effect may be inflated and that our conclusions about treatment effect on formal employment should be based only on those studies rated medium quality (g=0.12; 95% CI [0.05, 0.19]). Monthly earnings - The overall mean effect of TVET on earnings was positive and significant; however, significant heterogeneity was observed (Q = 25.5; df = 8; p = 0.00128; I2 = 68.6%; tau2 = 0.00815). - Four variables were tested for moderating effects. - No evidence of a statistically significant relationship between study quality and effect size was observed (Qb = 0.204; p = 0.652. It is reasonable, therefore, to conclude that the overall mean effect is not inflated and that our conclusions about treatment effect on monthly earnings should be based on all studies in the analysis (g=0.127; 95% CI [0.043, 0.21]). - No statistically significant differences in mean effects were observed between studies according to type of TVET intervention (Qb = 0.397; p = 0.529), length of follow-up period (Qb = 0.186; p = 0.666), or gender (Qb = 1.26; p = 0.262). Self-employment earnings - The overall mean effect of TVET on self-employment earnings was negative and non-significant (g=-0.025, 95% CI [-0.11, 0.061]). No significant heterogeneity was observed (Q = 0.206; df = 1; p = 0.65; I2 = 0%; tau2 = 0). This analysis was based on two