Conditions to Consider in the Use of Randomized Experimental Designs in Evaluation

Debates about the role of randomized experiments in evaluation have been heated at times, which likely has not facilitated and possibly has hindered thoughtful judgments about whether and when to use a randomized experimental design. The challenges of thoughtful deliberation may be especially great for funders and others who influence the choice of an evaluation design but are not immersed in methodological literatures. The current paper offers a non-technical summary of general factors to take into consideration when determining the appropriateness of a randomized design in a forthcoming evaluation or set of evaluations. Four general conditions are described that should be considered with respect to the specific context for the upcoming evaluation(s). These are, first, the expected value of the information that a well-implemented experiment can provide in the specific context; second, the legal and ethical issues that apply in the circumstances at hand; third, the practical constraints (or facilitating factors) that would apply to a randomized experiment in that context; and fourth, the likely value of the experimental findings in relation to and as part of a portfolio of evaluative studies in the specific context

Job Stability and Wage Progression Patterns among Early TANF Leavers

This article reports on first-year employment experiences of a randomly selected sample of 213 Illinois TANF leavers. Aggregate employment levels were 70 percent at exit, and leavers typically generated earnings from a single full-time job. However, employment often was unstable, so that only about one-fourth of leavers had the same job both at exit and when interviewed 10-11 months later. Employment instability resulted from the marginal or temporary nature of many jobs, as well as employment barriers such as health problems and lack of day care. Average wage levels easily exceeded the minimum wage and grew during the first year after exit, but nonetheless often were insufficient to provide incomes above the poverty level. The findings underscore the need to develop post-employment service strategies that assist persons in accessing work-related benefits such as child care and Medicaid, as well as improved income support strategies such as expansion of the Earned Income Tax Credit

Analysis of No-Difference Findings in Evaluation Research

Conclusions of no difference are becoming increasingly important in evaluation research. We delineate three major uses of no-difference findings and analyze their meanings. (1) No-differ ence findings in randomized experiments can be interpreted as support for conclusions of the absence of a meaningful treatment effect, but only if the proper analytic methods are used. (2) Statistically based conclusions in quasi-experiments do not allow causal statements about the treatment impact but do provide a metric to judge the size of the resulting difference. (3) Using no-difference findings to conclude equivalence on control variables is inefficient and potentially misleading. The final section of the article presents alternative methods by which conclusions of no difference may be supported when applicable. These methods include the use of arbitrarily high alpha levels, interval estimation, and power analysis.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/67182/2/10.1177_0193841X8901300604.pd

Providing Appropriate Evaluative Support for Valuing in the Public Interest

Depto. de Ciencia Política y de la AdministraciónFac. de Ciencias Políticas y SociologíaTRUEpu

New directions for evaluation

Publ. comme no 91, fall 2001 de la revue New directions for evaluationBibliogr. à la fin des textesIndex: p. 131-13

Professional Values and Ethics in Evaluation

Depto. de Ciencia Política y de la AdministraciónFac. de Ciencias Políticas y SociologíaTRUEpu

Professional Evaluation in the Public Interest(s)

Efforts to promote professionalism in evaluation, whether through certification, credentialing, or other path, are not new, but there is a new push to adopt sets of essential evaluator competencies, both in the United States and globally, that are intended to advance professionalism of the field. This emphasis on professionalism is relevant to the American Journal of Evaluation Section on Professional Values and Ethics in that this section focuses on how values are to influence evaluation practice and on how our understanding of valuing affects our view of the ethics that should guide evaluators. In his invited contribution to this issue’s section, Tom Schwandt provides an account of professionalism based on a view of evaluators serving a moral purpose in serving society. This account, a democratic professionalism in which citizens are engaged as co-owners of evaluation, emphasizes a professional ethos of evaluation that complements the focus of those working to advance the more technical aspects of professionalism. To introduce Schwandt’s essay, we review some of the recognized promises and challenges for professionalism in evaluation and highlight the promise of Schwandt’s approach for managing the tension in professions between serving professionals and serving society