Learning from Experiments: An Evaluation Plan for CMAQ Projects

Abstract

The Congestion Mitigation/Air Quality Program (CMAQ), established in 1991 by the Intermodal Surface Transportation Efficiency Act (ISTEA) to provide about $1 billion per year to fund transportation projects that improve air quality, is intended both to support traditional transportation control measures and to encourage innovation in developing new strategies and technologies for controlling emissions from transportation sources. While the program has indeed encouraged some innovative approaches to local transportation and air quality problems, critics see it as a diversion of funds that could more usefully be devoted to conventional highway improvement projects. The current debate in Congress over the reauthorization of ISTEA and, specifically, the CMAQ provisions, is hampered by the lack of detailed information about the achievements of previous CMAQ projects and a plan for evaluating future projects. Resolution of this debate could be aided by emphasizing the role of CMAQ projects as natural experiments and developing a plan to conduct them. The purpose of this paper is to outline a strategy of analysis and data collection that will facilitate evaluation of CMAQ projects. This paper argues that the lack of emphasis (in all but the largest projects) on project evaluation can be explained by the public goods nature of information. Because local implementing agencies bear the costs of evaluation, while the benefits are enjoyed primarily by other jurisdictions in planning their transportation and environment projects, too little evaluation is conducted. At present, much of the potential usefulness of CMAQ projects to planners is dissipated because there is little systematic learning. Indeed, a project could succeed as an experiment if learning took place, even if it failed to improve air quality. This paper examines the kinds of data collected now in CMAQ programs in comparison with the kinds of data that would permit more effective program evaluation, particularly ex post evaluation, i.e., analysis of what actually resulted from the implementation of the individual project. In many cases, data-gathering should concentrate on observable outcomes that can clearly be attributed to the project and yet bear some relationship to air quality or congestion, either established by previous empirical study or by model results. A method is proposed for collecting the requisite data for each of several important types of CMAQ projects. To assure that the data are collected and evaluated will also require changes in the way in which CMAQ is administered, including the dedication of some portion of CMAQ funds for evaluating completed projects. The biggest change may be the need to develop measures of "success" and identify "control cases" against which to judge the success of the experiment.