The Case for Multisystemic Therapy: Evidence or Orthodoxy?

In this paper, I respond to comments by Henggeler, Schoenwald, Borduin, and Swenson [Henngeler, S. W., Schoenwald, S. K., Borduin, C. M., & Swenson, C. C. (this issue). The Littell paper: Methodological critique and meta-analysis as Trojan horse. Children and Youth Services Review, doi: 10.1016/j.childyouth.2005.07.001] on my recent article, “Lessons from a systematic review of Multisystemic Therapy”. I identify factual and logical errors in their response, show how relevant research has been misinterpreted and misrepresented, and suggest constructive new directions for Multisystemic Therapy and the evidence-based practice movement

Evidence-based or Biased? The Quality of Published Reviews of Evidence-based Practices

Objective To assess methods used to identify, analyze, and synthesize results of empirical research on intervention effects, and determine whether published reviews are vulnerable to various sources and types of bias. Methods Study 1 examined the methods, sources, and conclusions of 37 published reviews of research on effects of a model program. Study 2 compared findings of one published trial with summaries of results of that trial that appeared in published reviews. Results Study 1: Published reviews varied in terms of the transparency of inclusion criteria, strategies for locating relevant published and unpublished data, standards used to evaluate evidence, and methods used to synthesize results across studies. Most reviews relied solely on narrative analysis of a convenience sample of published studies. None of the reviews used systematic methods to identify, analyze, and synthesize results. Study 2: When results of a single study were traced from the original report to summaries in published reviews, three patterns emerged: a complex set of results was simplified, non-significant results were ignored, and positive results were over-emphasized. Most reviews used a single positive statement to characterize results of a study that were decidedly mixed. This suggests that reviews were influenced by confirmation bias, the tendency to emphasize evidence that supports a hypothesis and ignore evidence to the contrary. Conclusions Published reviews may be vulnerable to biases that scientific methods of research synthesis were designed to address. This raises important questions about the validity of traditional sources of knowledge about “what works,” and suggests need for a renewed commitment to using scientific methods to produce valid evidence for practice

Evidence-based or Biased? The Quality of Published Reviews of Evidence-based Practices

Objective To assess methods used to identify, analyze, and synthesize results of empirical research on intervention effects, and determine whether published reviews are vulnerable to various sources and types of bias. Methods Study 1 examined the methods, sources, and conclusions of 37 published reviews of research on effects of a model program. Study 2 compared findings of one published trial with summaries of results of that trial that appeared in published reviews. Results Study 1: Published reviews varied in terms of the transparency of inclusion criteria, strategies for locating relevant published and unpublished data, standards used to evaluate evidence, and methods used to synthesize results across studies. Most reviews relied solely on narrative analysis of a convenience sample of published studies. None of the reviews used systematic methods to identify, analyze, and synthesize results. Study 2: When results of a single study were traced from the original report to summaries in published reviews, three patterns emerged: a complex set of results was simplified, non-significant results were ignored, and positive results were over-emphasized. Most reviews used a single positive statement to characterize results of a study that were decidedly mixed. This suggests that reviews were influenced by confirmation bias, the tendency to emphasize evidence that supports a hypothesis and ignore evidence to the contrary. Conclusions Published reviews may be vulnerable to biases that scientific methods of research synthesis were designed to address. This raises important questions about the validity of traditional sources of knowledge about “what works,” and suggests need for a renewed commitment to using scientific methods to produce valid evidence for practice

The Case for Multisystemic Therapy: Evidence or Orthodoxy?

In this paper, I respond to comments by Henggeler, Schoenwald, Borduin, and Swenson [Henngeler, S. W., Schoenwald, S. K., Borduin, C. M., & Swenson, C. C. (this issue). The Littell paper: Methodological critique and meta-analysis as Trojan horse. Children and Youth Services Review, doi: 10.1016/j.childyouth.2005.07.001] on my recent article, “Lessons from a systematic review of Multisystemic Therapy”. I identify factual and logical errors in their response, show how relevant research has been misinterpreted and misrepresented, and suggest constructive new directions for Multisystemic Therapy and the evidence-based practice movement

Caregivers\u27 Readiness for Change: Predictive Validity in a Child Welfare Sample

Objective: To assess the predictive validity of continuous measures of problem recognition (PR), intentions to change (ITC), and overall readiness for change (RFC) among primary caregivers who received in-home services following substantiated reports of child abuse or neglect. Method: A modified version of the University of Rhode Island Change Assessment scale was included in interviews with a sample of 353 primary caregivers at 4 weeks, 16 weeks, and 1 year after referral for in-home services. Additional data were obtained from administrative records and caseworker surveys. Hierarchical linear and nonlinear models were used to assess relationships between PR, ITC, RFC and changes over time in measures of individual and family functioning (e.g., parenting behaviors, children\u27s behaviors, housing and economic problems, social support, and life events). Bivariate probit regression analysis was used to examine relationships between PR, ITC, RFC and the likelihood of subsequent reports of child maltreatment and out-of-home placements within 1 year after referral. Results: Initial problem recognition and intentions to change predict a few improvements in individual and family functioning, along with significant reductions in the likelihood of additional reports of child maltreatment within 1 year. Initial intentions to change also predict reductions in the substantiation of subsequent reports of maltreatment. An overall measure of readiness for change predicts reductions in the likelihood of out-of-home placement. Conclusions: Problem recognition and intentions to change predict somewhat different outcomes; hence, there are few advantages of a combined measure of readiness for change. Further inquiry is needed to determine whether and how these associations are mediated by intervention processes or other factors in child welfare services populations

Correlates of Problem Recognition and Intentions to Change among Caregivers of Abused and Neglected Children

Objective To identify individual, family, and caseworker characteristics associated with problem recognition (PR) and intentions to change (ITC) in a sample of caregivers who received in-home child welfare services following substantiated reports of child abuse or neglect. Methods Caregivers were interviewed at 4 weeks, 16 weeks, and 1 year after referral for in-home services. In these interviews, the University of Rhode Island Change Assessment scale assessed PR and ITC in relation to caregiving practices. Additional data were obtained from administrative records and surveys of in-home services caseworkers. We used growth models to identify caregiver, family, and caseworker characteristics associated with initial levels of PR and ITC, and with changes in PR and ITC over time. Results Contrary to expectations, there were no overall increases in PR and ITC during the first 4 months of in-home services. PR and ITC scores fluctuated over time, in relation to some external events, case characteristics, and caseworker characteristics; however, we were able to account for small portions of the variance in PR and ITC. Controlling for social desirability bias and other variables in the analysis, negative life events, lack of network support, and the severity of caregiver depression were associated with greater PR; housing problems were associated with lower PR. Caregiver depression and age were associated with ITC. Caregivers whose children had been removed from their homes showed significant increases in ITC in the first few months of in-home services. The duration of caseworkers’ child welfare experience predicted increases in their clients’ PR in the first 4 months and more frequent contacts with an experienced caseworker predicted small, but significant increases in ITC over time. Conclusions PR and ITC are associated with somewhat different case characteristics and may be affected by caseworkers’ experience