McSweeney, Tim

Meadows, P.

Metcalf, H.

Stanley, C.

Turnbull, Paul J.

English

Name not available

   BIROn - Birkbeck Institutional Research Online  Enabling open access to Birkbeck’s published research output    The feasibility of conducting an impact evaluation of the Dedicated Drug Court pilot  Report  http://eprints.bbk.ac.uk/3758   Version: Published    Citation:        © 2009 Crown Copyright    Publisher version    ______________________________________________________________  All articles available through Birkbeck ePrints are protected by intellectual property law, including copyright law. Any use made of the contents should comply with the relevant law. ______________________________________________________________  Deposit Guide  Contact: lib-eprints@bbk.ac.uk Birkbeck ePrints McSweeney, T.; Meadows, P.; Metcalf, H.; Turnbull, P.; Stanley, C.  (2009)  The feasibility of conducting an impact evaluation of the Dedicated Drug Court pilot – Ministry of Justice Research Summary 2/10 Research Summary 2/10© Crown copyright 2010Extracts from this document may be reproduced for non-commercial purposes on condition that the source is acknowledged.First published March 2010ISBN 978-1-84099-380-6Contact info:research@justice.gsi.gov.ukThe views expressed in this Research Summary are those of the authors, not necessarily those of the Ministry of Justice (nor do they reflect Government policy)The feasibility of conducting an impact evaluation of the Dedicated Drug Court pilotSummary based on research undertaken for the MoJ by T. McSweeney, P. Meadows, H. Metcalf, P. Turnbull and C. Stanley (National Institute of Economic and Social Research, and the Institute for Criminal Policy Research, Kings College, London), completed in 2009.ContextThe study aimed to establish whether a robust impact evaluation of the Dedicated Drug Court (DDC) pilot would be possible and, if so, the requirements for such an evaluation1. An impact evaluation would provide an assessment of whether the DDC pilot makes any difference to outcomes, by looking at what happens to those who go through the DDC, compared to what would have happened had they not. The impact is the difference between the two. This would be of use in helping the MoJ determine whether it was possible to measure the impact of the DDC pilot. This in turn would help with decisions on whether to roll out the pilot more widely. The outcomes of interest were:• reducing reoffending by drug-misusing offenders; and• reducing drug misuse.The DDC model was piloted at six magistrates’ courts: Leeds, West London, Barnsley, Bristol, Cardiff and Salford. The model aimed to:• improve the processes and effectiveness of the magistrates’ courts in dealing with drug-misusing offenders; and • reduce drug use and reoffending. It provided a new framework in magistrates’ courts for dealing with drug-misusing offenders who commit low-level ‘acquisitive’ crime to fund their addiction. It did not introduce new treatment or sentencing options. The key elements of the DDC model were as follows.• Exclusivity: handling cases only relating to drug-misusing offenders from (and including) sentence, through review to completion of the sentence and, where possible, breach of their orders.• Continuity: ensuring sustained continuity of magistrates’ bench or District Judge throughout the period an offender comes before the DDC.1. Technical information on the methods used and further information about this study is available on request from the Ministry of Justice.2• Training: the judiciary and court staff receive additional training on working with drug-misusing offenders and the DDC model.• Processes: processes designed to ensure all necessary information is before the court when required.• Partnership: ensuring effective multidisciplinary working with other criminal justice system agencies and professionals – in particular between the court, the probation service and those providing drug treatment or other related services.A number of methods for evaluating the impact of the DDC pilots were considered. These methods provide an understanding of the impact of the DDC on offenders, and a means of estimating what might have happened if the offender had been dealt with by other courts. They were assessed on:• how well they would be able to measure reduced reoffending and drug use outcomes; and• ease of implementation, including resources, time, data quality and availability, and sample size needed.The following designs were considered2.1) A Randomised Control Trial (RCT). In an RCT, those who are eligible for the pilot are randomly assigned to ‘treatment’ or ‘control’ groups, and their outcomes are compared. The treatment group receive the intervention (in this case go through the DDC) and the control group do not (in this case they would receive the standard non-DDC court approach). This approach is common in drug trials when volunteers are randomly assigned and given either the medication being tested or a placebo. The benefit of this design is that the comparator (control) group is created automatically as part of the pilot process.2) A between-area comparison. Offenders who go to a DDC in one court area are compared with offenders in a court area without a DDC. The offenders are closely matched on factors which affect outcomes (e.g. previous and current convictions, gender, age, identified class A drug misuse, need). Areas are matched on characteristics such as criminal justice system (CJS) practice and locality. 3) A within-area comparison. This is where offenders who go to a DDC in one court area are compared with offenders (matched as above) from the same court area who are dealt with by other courts.4) A historical comparison. This is where offenders who go to a DDC are compared with offenders (matched as above) from the same court area who passed through the courts before the existence of the DDC.The latter three designs require some sort of comparator group of offenders, matched as closely as possible on the personal characteristics of the offenders associated with reoffending and the area characteristics (such as criminal justice and drug treatment practices and economic circumstances). This provides a way of minimising the effect of individual differences between offenders. When matching is used, this requires the use of statistical techniques to control for differences between the offender groups. Random assignment (i.e. RCT), if conducted successfully, negates the need for this.ApproachTo identify possible designs, the following factors were assessed:• the extent to which DDC and non-DDC practice differed;• the availability of relevant administrative data;• the potential impact of the DDCs on outcomes (to estimate the sample size required to detect effects); and• for an RCT, consideration was given to the way in which offenders were allocated to the 2. A qualitative process evaluation is being undertaken. This will look at the perceived impact of the DDC pilot including the benefits, an assessment of funding required and the level of support received from partners and stakeholders. This report will be available in Autumn 2010.3DDC in order to assess whether the processes involved would permit genuinely random allocation of cases between DDCs and other courts.Information to support assessment of the above factors was gathered in the following ways:• through interviews with 15 representatives from the six pilot DDCs;• through interviews with 20 representatives from the courts and probation in 15 areas which did not have a DDC;• collation of existing activity data on DDCs;• a literature review of evidence on the effectiveness of drug courts; and• a review of existing health and criminal justice datasets.ResultsImpactThe literature review suggested that international drug court evaluations provided a poor indication of the likely impact of the DDC pilot. This was because the drug courts evaluated elsewhere differed substantially from the DDC model in England and Wales. In particular, many other drug courts involved not only a change of process (which the DDC pilot does) but also recourse to enhanced drug treatment for offenders (which the DDC pilot does not). Bearing this in mind, a minimum impact of five percentage points was assumed when considering potential designs for the DDC pilot, based on the outcomes from other countries and published reconviction data for drug-misusing offenders in Britain. This impact is the assumed difference in rates of reconviction between DDC and non-DDC groups. Due to the lack of evidence from international drug court evaluations regarding the key elements of the DDC approach, this estimate of impact may be optimistic. With a smaller impact, the sample sizes given below would be too small to detect the effect and would need to be increased, with time and cost implications. This means there is a relatively high risk that no effect would be detected, even if one existed. Infeasible designs When implementing the England and Wales DDC model, the pilot sites received guidance to develop local practices from a minimum framework. Interviews with the DDC pilot sites showed that practice across the six sites varied and the key characteristics of the DDC model were not consistently defined or systematically measured and monitored. The target groups and the extent to which offenders were referred to the DDC for sentencing also varied. However, all DDCs sought a degree of continuity of magistrate and reviewed all Drug Rehabilitation Requirements3 (DRRs), with the exception of those offenders who lived outside the court area and were sent to their home court for DRR reviews post-sentence. This meant that there was no obvious group of drug-misusing offenders within a DDC area that had not engaged with the DDC. Therefore a within-area comparison design was not feasible. Historically, some of the DDCs had been operating as quasi-drug courts prior to acquiring formal pilot status and it was not possible to establish when practice had changed. This meant that a historical comparison design was not feasible.Feasible designsThere were no data constraints affecting potential designs. However, the administrative data on continued illicit drug use of offenders were not comprehensive which affected the ability to assess drug misuse outcomes. Offenders were referred to the DDC by a range of people: police, magistrates and the court administration. This would make it difficult to randomly allocate offenders to the DDC or another court and any system of random allocation would run a high risk of implementation failure.An RCT was feasible in theory, but would require a sample size of at least 2,800 in total. At the time of the study, there were around 800 useable cases annually going through the DDCs, so a sample of this size would take around four years to assemble. With follow-up (between one and two years), an 3. The Criminal Justice Act 2003 replaced Drug Treatment and Testing Orders (DTTOs) with a new Community Order with a DRR. This is similar to the DTTO but with greater flexibility around supervision and management, with attendance requirements ranging from one contact to 15 hours of supervision each week depending on the needs, risks and seriousness of an offence. 4RCT would take between 8.5 and 11 years, and would cost between £1,000,000 and £1,500,000.A between-area comparison design was also feasible. It would compare DDCs with other courts nationally, or courts in selected areas without a DDC. This design would require a minimum sample size of 3,200: 1,600 in the DDCs and 1,600 in other areas. To process 1,600 cases through the DDCs would take two years. The evaluation would take an additional four to five years to identify impact over one or two years respectively. If the DDCs were to be compared with courts nationally this would cost between £260,000 and £300,000. If they were to be compared with courts in selected areas this would cost £280,000 to £325,000. The higher cost is due to additional work identifying suitable comparator courts which would need to be similar to the DDCs other than in drug court characteristics.All costs stated above were rough estimates. Exact costs would be determined by a range of factors that were not within the scope of this study to obtain.ImplicationsThere were two feasible designs to consider – an RCT and a between-area design. For any design it would be difficult to measure a reduction in drug misuse. Running an RCT presents a risk of failure during the randomisation process, as it requires compliance and assistance from a large number of practitioners. The between-area comparison design runs the risk of falsely attributing outcome differences to the DDC when other factors are involved. The between-area comparison seemed the most cost-effective and least risky approach. However, the length of time needed for either approach would be considerable. Given the lack of robust evidence on the likely effect and the small degree of difference between DDCs and non-DDCs, there is a high risk that the sample sizes proposed here would not be large enough to detect a reduction in reoffending if it was less than five percentage points. If the effect was smaller than this, the sample size would need to be increased to identify the impact. This would have time and cost implications.

The feasibility of conducting an impact evaluation of the Dedicated Drug Court pilot

Abstract

Similar works

Full text

Available Versions

Name not available

Birkbeck Institutional Research Online