Heckman, James J.

Heinrich, Carolyn J.

Smith, Jeffrey

English

Upjohn Research

   Upjohn Institute Press   Lessons for Advancing Future Performance Standards Systems    James J. Heckman University of Chicago  Carolyn J. Heinrich University of Wisconsin  Jeffrey Smith University of Michigan           Chapter 10 (pp. 305-309) in: The Performance of Performance Standards James J. Heckman, Carolyn J. Heinrich, Pascal Courty, Gerald Marschke, and Jeffrey Smith, eds. Kalamazoo, MI: W.E. Upjohn Institute for Employment Research, 2011 DOI: 10.17848/9780880993982.ch10  Copyright ©2011. W.E. Upjohn Institute for Employment Research. All rights reserved. The Performance of Performance StandardsJames J. HeckmanCarolyn J. HeinrichPascal CourtyGerald MarschkeJeffrey SmithEditors2011W.E. Upjohn Institute for Employment ResearchKalamazoo, MichiganLibrary of Congress Cataloging-in-Publication DataThe performance of performance standards / James J. Heckman . . . [et al.], editors.  p. cm. Includes bibliographical references and index. ISBN-13: 978-0-88099-292-3 (pbk. : alk. paper) ISBN-10: 0-88099-292-1 (pbk. : alk. paper) ISBN-13: 978-0-88099-294-7 (hardcover : alk. paper) ISBN-10: 0-88099-294-8 (hardcover : alk. paper) 1. Government productivity. 2. Performance standards. 3. Civil service—Personnel management.  I. Heckman, James J. (James Joseph) JF1525.P67P476 2011 352.6'7—dc222011007877  © 2011W.E. Upjohn Institute for Employment Research300 S. Westnedge AvenueKalamazoo, Michigan 49007-4686The facts presented in this study and the observations and viewpoints expressed are the sole responsibility of the author. They do not necessarily represent positions of the W.E. Upjohn Institute for Employment Research.Cover design by Alcorn Publication Design.Index prepared by Diane Worden.Printed in the United States of America.Printed on recycled paper.30510Lessons for Advancing Future Performance Standards SystemsJames J. HeckmanCarolyn J. HeinrichJeffrey SmithThe economic recession that started in 2007 led to renewed inter-est in public employment and training services. At the same time, the accompanying fi nancial crises also elevated concern for how dwindling government budgets could be spent more effi ciently and effectively to maximize returns to public investments in training. As discussed in the introductory chapter, revamping incentive systems in government is a critical step toward improving government performance and our future economic outlook.The chapters in this volume marshal some of the most detailed evi-dence available on how performance standards and incentives infl uence the behavior of program administrators and staff and contribute to pro-gram outcomes or unintended consequences. Since each chapter pres-ents a self-contained summary of its main fi ndings, we do not review the details of each one. Instead, this conclusion presents three main lessons learned from these essays and discusses some of their policy implications.LESSON 1: AGENCIES RESPOND TO INCENTIVESConcerns that performance incentives are disregarded by govern-ment employees because award levels are low or because benefi ts are diffused are not justifi ed. Low-powered cash incentives may, in fact, be high-powered because of the value of the budgetary awards in estab-lishing the reputation of bureaucrats and the recognition that comes chapter10.indd   305 4/27/2011   9:56:16 AM306   Heckman, Heinrich, and Smithwith them. Bonuses and award money create leverage in the social ser-vices community and are thus frequently highly prized (Heinrich 2007). The evidence reported by Courty and Marschke in Chapter 7 demon-strates that agencies made placement, enrollment, and termination deci-sions in ways that were consistent with maximizing their performance as measured by the JTPA performance standards system. The evidence presented by Heinrich in Chapter 8 reveals that although caseworkers claimed to discount performance standards in decision making, they nevertheless selectively enrolled into JTPA people who were likely to contribute to the placement goals rewarded under the performance stan-dards system. Courty and Marschke and Heinrich also present a dark side to the behavior elicited by the JTPA performance standards system. Training centers showed remarkable ingenuity in manipulating the JTPA account-ing system and reporting requirements in their efforts to boost their measured performance. Practices included enrolling persons receiv-ing job search assistance or on-the-job training only after they found a job, using short-term training arrangements in order to maximize the probability of counting a successful placement, holding persons who did not fi nd jobs in dead-end job clubs, and releasing poorly perform-ing trainees from the program at strategic accounting dates when it did least damage to training center performance records. These all represent behaviors that make perfect sense in terms of the performance stan-dards, but they do nothing to raise participant earnings or increase the equity with which program services are distributed. The problem of regulating job-training programs—or any govern-ment agency—is that enforceable regulations cannot be written too fi nely, and simple rules can and will, as shown in this volume, be sub-verted. Along these lines, performance standards systems designers also need to grasp the dynamic properties of performance incentive sys-tems, as discussed in Chapter 5 and illustrated throughout this volume. An incentive system designer’s understanding of how individuals will respond to performance standards will inevitably be imperfect prior to their introduction, and it is only as performance measures and targets are tried, evaluated, modifi ed, and/or discarded that their responses become known (Courty and Marschke 2007). Of course, this type of monitor-ing to assess a measure’s effectiveness and possible distortion requires a considerable investment on the part of incentive system designers, chapter10.indd   306 4/27/2011   9:56:28 AMLessons for Advancing Future Performance Standards Systems   307one that has probably been underestimated in the past. And as Heinrich and Marschke (2010) point out, learning on the part of bureaucrats will advance over time as well, as they come to better know the distinct weaknesses of performance measures and how they can be exploited through their day-to-day experiences. Incentive system designers will have to expect to regularly review and revise the rules and incentives they create if they want to avoid the ineffi cient behavior documented in the chapters in this volume. LESSON 2: CURRENT PERFORMANCE STANDARDS DO NOT PREDICT LONG-RUN GAINSWhile the chapters by Courty and Marschke and Heinrich indi-cate that the JTPA performance standards system effectively motivated agency staff and service providers to meet short-run performance stan-dards, they do not indicate whether the measures themselves are appro-priate to induce the achievement of the primary program goals, i.e., increasing the earnings of program participants (or the value added of the program). Chapter 9 demonstrates that the short-run performance measures featured by the JTPA performance standards system were weakly and often perversely related to the long-term effects of the pro-gram on the earnings and employment of participants. Yet it is these long-term effects that constituted the true goal of the program’s services. The analysis also reveals that the measurement of wages and employ-ment at a later point following termination represented no improve-ment in the performance monitoring system. Neither set of measures was strongly positively related to long-run program impacts on earn-ings and employment, suggesting that the performance standards sys-tem did not promote the program’s key objectives of long-run gains in earnings and employment for participants. We see no reason to expect that the relationships between the performance measures and programs have changed under WIA and thus no reason to think that this problem has gone away.From the perspective of policymakers and taxpayers who would like to maximize the value from government dollars spent on public programs, the goal of incentive system designers should be to choose chapter10.indd   307 4/27/2011   9:56:28 AM308   Heckman, Heinrich, and Smithperformance measures so that the effects of bureaucratic actions on mea-sured performance are aligned with the effects of those same actions on value added. However, we frequently lack the information required to realize this objective in practice. Empirical research, as Heckman and Smith suggest, has focused primarily on estimating measures of statisti-cal association between performance and value added, where the value added of programs has been assessed through the use of randomized experiments or through sophisticated statistical modeling. The dynam-ics of performance standards systems, however, may limit the useful-ness even of these estimates, as the alignment between a performance measure and value added may decrease after a performance measure is introduced and bureaucrats respond by exploring all strategies for rais-ing it, not just those that also increase program value added. Clearly, this will continue to be one of the most vexing issues for performance standards system designers for some time to come.LESSON 3: THE CREAM SKIMMING PROBLEM IS OVERSTATEDThe charge of cream skimming has arisen frequently in public job training programs, including the Workforce Investment Act program. In Chapters 3, 6, and 9, we defi ne various uses of this term and pre-sent evidence that fears about cream skimming are overstated. For most demographic groups, experimental estimates of the earnings impacts of participation in JTPA are uniform over broad skill levels. Only at the lowest skill levels is there any evidence that impacts were smaller than at higher levels of the skill distribution. While there is some evi-dence of a trade-off between serving the most disadvantaged within eligible populations and allocating program resources most effi ciently, it appears to be a modest one given the modest benefi ts realized by program participants.Cream skimming is usually defi ned as arising from purposive screening behavior by bureaucrats. Chapters 6 and 8 present informa-tion about disparities in program participation rates among different demographic groups. Even among eligible persons, there are substantial differences in program participation rates. It is not enough to compare chapter10.indd   308 4/27/2011   9:56:28 AMLessons for Advancing Future Performance Standards Systems   309participation rates among the eligible to determine if cream skimming is an important factor. At issue is how much of the difference in par-ticipation rates conditional on eligibility is due to the voluntary choices of individuals, how much is due to their lack of information about the program, and how much is due to the decisions of bureaucrats. The decompositions of participation rates into components due to eligibil-ity, awareness, and acceptance that are presented in Chapter 6 reveal that personal choices and lack of information play the major role in accounting for demographic disparities in participation. Administrative discretion also has some role to play in determining the participation rates of different groups in public job training programs, but it is not the dominant factor.The evidence in Chapters 6 and 8 indicates that those whose char-acteristics make them more employable and more easily placed were more likely to be screened into the JTPA system. However, the analysis of Chapter 6 reveals that the same characteristics that make a person more attractive in terms of achieving objectives within the job place-ment system also made that person more aware of the program and more willing to apply to it. Thus, if resources are going to be used to improve equity in access to programs with voluntary enrollment, incen-tive system designers and program administrators should consider investing more in (or providing incentives for) increasing awareness of the program among the eligible population.We realize that there will be some limits to the generalizability of these fi ndings and the lessons of this collection of studies to other public program contexts, and that there is inherent variability in the potential of performance standards systems for improving government performance in the many different contexts in which they have been introduced or considered. At the same time, it is clear that there is no diminishing of demand on the part of policymakers or the public for greater accountability and a results-oriented focus of government. The research in this volume suggests that there is considerable work to do in addressing the fl aws of current public sector performance standards systems as well as investing in research that will guide improvements and advancements to these systems as their use expands in the public sector.chapter10.indd   309 4/27/2011   9:56:28 AM310   Heckman, Heinrich, and SmithReferencesCourty, Pascal, and Gerald Marschke. 2007. “A General Test for Distortions in Performance Measures.” Review of Economics and Statistics 90(3): 428–441.Heinrich, Carolyn J. 2007. “False or Fitting Recognition? The Use of High Performance Bonuses in Motivating Organizational Achievements.” Jour-nal of Policy Analysis and Management 26(2): 281–304.Heinrich, Carolyn J., and Gerald R. Marschke. 2010. “Incentives and Their Dynamics in Public Sector Performance Management Systems.” Journal of Policy Analysis and Management 29(1): 183–208. chapter10.indd   310 4/27/2011   9:56:28 AM

Lessons for Advancing Future Performance Standards Systems

https://research.upjohn.org/cgi/viewcontent.cgi?filename=9&article=1225&context=up_press&type=additional

Lessons for Advancing Future Performance Standards Systems

Abstract

Similar works

Full text

Available Versions

Upjohn Research