This report uses two methodological innovations to measure the impact of advocacy groups on education reform policy, Surveys with Placebo (SwP) and Critical Path Analysis (CPA). SwP is designed to measure the influence of advocacy groups and CPA to identify which tactics were successful in influencing reform. The report finds them to be effective methods for achieving these goals. It finds that coordination of advocacy groups strengthens their impact and that the perceived impact of advocacy groups tracks closely with policy outcomes. While we agree that the SwP and CPA may be useful in education policy research, these methods are more limited than the report acknowledges. Moreover, the research is a small case study of three states, with a low response rate for the SwP and CPA based on advocacy groups&rsquo; self-reported tactics. The report also fails to give sufficient information on the responses to the SwP and the selection of the advocacy groups studied to assess the usefulness of methods and validity of the report&rsquo;s conclusions. Finally, there is not a strong connection between the evidence presented in the report and its conclusions. We therefore caution against adoption of the methods or reliance on the conclusions presented in this report without significant further research.</p

Goldrick-Rab, Sara

Rogers, Robin

Undetermined

CU Scholar Institutional Repository

  REVIEW OF MEASURING AND UNDERSTANDING EDUCATION ADVOCACY   Reviewed By Robin Rogers, Queens College, CUNY Sara Goldrick-Rab, University of Wisconsin-Madison April 2015 Summary of Review This report uses two methodological innovations to measure the impact of advocacy groups on education reform policy, Surveys with Placebo (SwP) and Critical Path Analysis (CPA). SwP is designed to measure the influence of advocacy groups and CPA to identify which tactics were successful in influencing reform. The report finds them to be effective methods for achieving these goals. It finds that coordination of advocacy groups strengthens their impact and that the perceived impact of advocacy groups tracks closely with policy outcomes. While we agree that the SwP and CPA may be useful in education policy research, these methods are more limited than the report acknowledges. Moreover, the research is a small case study of three states, with a low response rate for the SwP and CPA based on advocacy groups’ self-reported tactics. The report also fails to give sufficient information on the responses to the SwP and the selection of the advocacy groups studied to assess the usefulness of methods and validity of the report’s conclusions. Finally, there is not a strong connection between the evidence presented in the report and its conclusions. We therefore caution against adoption of the methods or reliance on the conclusions presented in this report without significant further research.          Kevin Welner Project Director William Mathis Managing Director Don Weitzman Academic Editor Erik Gunn Managing Editor  National Education Policy Center School of Education, University of Colorado Boulder, CO 80309-0249 Telephone: (802) 383-0058 Email: NEPC@colorado.edu http://nepc.colorado.edu Publishing Director: Alex Molnar   This is one of a series of Think Twice think tank reviews made possible in part by funding from the Great Lakes Center for Education Research and Practice. It  is also available at http://greatlakescenter.org. This material is provided free of cost to NEPC's readers, who may make non-commercial use of the material as long as NEPC and its author(s) are credited as the source. For inquiries about commercial use, please contact NEPC at nepc@colorado.edu.   http://nepc.colorado.edu/thinktank/review-measuring-education-advocacy 1 of 6  REVIEW  O F MEAS URING  AND UNDERS TANDING  EDUC ATION ADVOC ACY  Robin Rogers, Queens College, CUNY Sara Goldrick-Rab, University of Wisconsin-Madison  I. Introduction The number of advocacy groups in education has expanded rapidly over the last decade, and there is substantial and growing interest in the role they play in shaping education policy. In a new report, Measuring and Understanding Education Advocacy , Grover J (Russ) Whitehurst and his former colleagues at the Brookings Institution’s Brown Center on Education Policy attempt to tease out the causal influence of these groups by examining advocacy organizations for and against education reform initiatives in Louis iana, Tennessee, and North Carolina.1 Their report describes a set of results, primarily focused on the influence of selected advocacy groups as perceived by lawmakers and key political actors in the three states studied and the tactics used by the advocacy groups to shape and support or to oppose legislation. The authors also advance the case for two methodological innovations, Surveys with Placebo (SwP) and Critical Path Analysis (CPA). The SwP uses a placebo (fictional) advocacy group to control for the influence that respondents are willing to attribute to any advocacy group in the survey. The CPA uses interviews and other qualitative data to identify tactics used by the advocacy groups to shape legislation and to get it passed or defeated.  II. Findings and Conclusions of the Report The report has findings on the methodologies used and implications for influencing the policymaking process and legislative outcomes.  Methods: The report asserts SwP can detect meaningful differences in the perceived influence of advocacy groups. Respondents separate their own position as a supporter or opponent of a bill from the perceived influence of groups. CPA gathers significant information that cannot be gathered for the SwP. The five-minute form of the SwP was highly correlated with the longer thirty-minute form. Implications for Influencing Policy: Advocacy organizations were recognized influences in the states studied. Coordination and role differentiation strengthened their  http://nepc.colorado.edu/thinktank/review-measuring-education-advocacy 2 of 6 impact. Perceived influence tracked closely with actual impact. Efforts to influence political outcomes are local. Advocates on the same side often had different line-item goals for the bill. Bipartisan support is enhanced by appealing to the motives of different groups.  III. The Report’s Rationale for Its Findings and Conclusions The report focuses on three case studies: Louisiana’s 2012 omnibus school choice legislation (House Bill 976), North Carolina’s 2013 Senate Bill 337, and Tennessee’s Senate Bill 196/House Bill 190. Each of the three bills had different content and different outcomes. Louisiana passed HB 976, which expanded eligibility for an existing statewide school voucher program, Student Scholarships for Educational Excellence. North Carolina passed SB 337 after considerable compromise, particularly over key provisions that created a process to authorize new charter schools. The proposed legislation in Tennessee would have permitted some low-income students in poorly performing schools to receive vouchers. According to the report, the Governor withdrew the bill in response to attempts to amend it. In Louisiana and Tennessee, the researchers studied four groups advocating for the proposed reform and two groups opposing it. In North Carolina, they studied three groups advocating for reform and two opposing it. What role did the advocacy organizations play in the outcomes in each state? To address this question, the authors first turn to a method they call Survey with Placebo (SwP), which was introduced in a prior paper.2 The method’s purpose is to “quantify the amount of influence that any advocacy organization exercises” (p. 1). The report focuses on the methodological innovation of a placebo group. The placebo group is a fictional advocacy group included in the survey in order to generate a known zero point. This zero point, it argues, can serve as a point of reference when measuring the reported influence of the other (real-life) advocacy groups. The research also employs Critical Path Analysis (CPA) to evaluate and verify the advocacy tactics used by the advocacy group. Researchers interviewed advocacy groups about their tactics and then selected a small number of tactics that they felt represented a major strategy used by those groups.  The report concludes that “role differentiation among advocacy organizations that have related policy goals strengthens total impact” (p. 23) and that “Achieving bipartisan support for education reform is enhanced by understanding and appealing to the motives of different camps of supporters” (p. 24). The report also states that the SwP “can detect meaningful differences in the perceived influence of advocacy groups and between the actual influence groups vs. the placebo organization” (p. 22).  IV. The Report’s Use of Research Literature  There is no research referenced in the report other than the authors’ prior report on Louisiana.   http://nepc.colorado.edu/thinktank/review-measuring-education-advocacy 3 of 6 V. Review of the Report’s Methods The researchers distributed the Survey with Placebo to all state legislators as well as selected “political insiders” whom the researchers expec ted to have first-hand knowledge of the factors that influenced the education reform proposals but who were otherwise undefined by the authors of the report. Recipients of the survey were asked to rank the influence of a set of advocacy groups, including a placebo group that did not exist in the state, using a scale of 1 (No Influence) to 7 (Major Influence). The survey was distributed in long (20-minute) and short (5-minute) forms, and the total number of respondents  was 72 in Louisiana, 77 in Tennessee, and 78 in North Carolina. Political insiders were over-represented in the responses (pp. 6-12). The use of the placebo is probably an improvement over typical influence surveys because it should account for respondents who are willing to attribute influence to any advocacy group. In this way it creates some ability to quantify (anything above the zero point) the actual influence of the advocacy groups. But while the SwP is an improvement over other methods used to measure the influence of advocacy groups, it does not permit the authors to “quantify the amount of influence any organization exercises,” as they claim. That is because what they measure simply the perceived influence of advocacy groups, not their actual influence. There are points in the report where the authors make this limitation clear, but the overall statement of findings implies that the impact of the advocacy groups can be measured. The SwP does not accomplish this. In the CPA, initial sketch diagrams of the tactics and outcomes (together comprising a “critical path diagram”) were sent to the advocacy groups for verification. Once the path was verified, researchers then attempted to independently verify that the tactic had been used and that it had worked as reported by the advocacy group, using data obtained from other sources including media accounts, state documents, and interviews with other involved parties. Certainly, diagrams and the multiple data sources are useful for providing insight into the strategies that the advocacy groups see as important. But the approach is limited by its focus on self-reported tactical paths. The CPA does not constitute an analysis of which advocacy tactics are in fact effective but rather examines retrospective reports of the perceived impacts of various tactics. By asking advocacy groups which tactics were successful and then letting them vet the diagram, the researchers gave too much power to the advocacy groups to describe their own tactics and did far too little independent investigating. They also assumed a sequence of influence that could be inaccurate. For example, a task force and the governor who appoints its members may have political ties to an advocacy group and therefore solicit information from that group for a hearing. This information may end up in the task force’s report and in the final legislation. The political Notably absent from the report is any mention of the highly contentious and politicized national context of education reform during the years under examination.  http://nepc.colorado.edu/thinktank/review-measuring-education-advocacy 4 of 6 influence of the advocacy group, therefore, may have been the cause of the information being provided to the task force rather than the result of it.  What is most concerning about the report is that even though its authors claim to offer methodological innovations, they provide very little information to help the reader examine how well those methods achieved their intended goals. For example, even though the SWP relies heavily on the correct identification of fake advocacy organizations having zero actual influence, the authors do not report the number of times those organizations were given a zero influence rating: that is, how many respondents correctly identified the placebo group as having no influence. Instead, means are presented without any other context.  Moreover, when describing the CPA the authors do not explain how specific categories of advocates were included or excluded. For example, they do not mention philanthropic foundations such as the Gates, Broad, and Walton foundations, which have spent millions of dollars on sustained reform efforts for over a decade. Instead, they compare and discuss individual state-level advocacy groups such as Louisiana’s Federation for Children and StudentsFirst in Tennessee without providing historical or national context or a discussion of the potential biases resulting from the omission of key groups and tactics. The reader is left wondering what else might be out there. Notably absent from the report is any mention of the highly contentious and politicized national context of education reform during the years under examination. Some of the groups examined, such as StudentsFirst, have a strong national presence. While this context could easily constitute a variable affecting policy reform through multiple avenues, the report treats each advocacy group in isolation, as if its impact on policy reform resulted solely from the actions and tactics of that state’s groups. In other words, national context constitutes a potentially critical omitted variable.  VI. Review of the Validity of the Findings and Conclusions The assumption that the reports of perceived influence measured by the SwP are tantamount to actual influence is very problematic. Similarly, the verification of reported advocacy group tactics done through the CPA is not sufficient grounds to make generalizations, as the report does, about the usefulness of particular advocacy tactics. The reader might be surprised to know that the sole basis for the report’s claim that “role differentiation among advocacy organizations that have related policy goals strengthens total impact” (p. 23) is the CPA analysis in Louisiana that suggested that one advocacy group in the state coordinated with other groups. But the existence of a single advocacy group verified as coordinating activities is insufficient evidence on which to base such a broad claim about role differentiation strengthening total impact. The report goes on to say, “The impact of advocacy organizations is conditional on powerful political actors taking positions that open the door to supportive advocacy activities” (p. 23). But it is impossible to defend this conclusion based solely on the evidence of a survey of perceived  http://nepc.colorado.edu/thinktank/review-measuring-education-advocacy 5 of 6 impact and a qualitative study of perceived tactical methods as described by the advocacy groups themselves. Overall, there is a weak connection between the report’s conclusions and its evidence. VII. Usefulness of the Report for Guidance of Policy and Practice Certainly, the report is right to note that the traditional delivery system for K-12 education is being “disrupted” by outside forces. And it should be commended for recognizing the limitations of influence surveys and attempting to get inside the “black box” of advocacy group activity. Both the SwP and the CPA are potentially useful methods. The former c an improve understanding of the perceived influence of various advocacy groups but it cannot quantify their actual influence. Similarly, the CPA can tell us what advocacy groups think (or are willing to tell researchers) are the strongest tactics, but it cannot tell explain which tactics are actually the most influential.  Given these limitations, this report is useful primarily for understanding the perceptions of education advocacy groups’ influence and tactics in the three cases studied. The sample and methods are too limited to be generalized much beyond that. Moreover, while both SwP and CPA are interesting methods that deserve further development, we caution against their adoption until they are better developed.    http://nepc.colorado.edu/thinktank/review-measuring-education-advocacy 6 of 6 Notes and References   1 Whitehurst, G. J., Stuit, D., Graves, C., & Shaw, L. (2015) Measuring and Understanding Education Advocacy. Washington, DC: Brookings Institution. Retrieved April 14, 2015, from http://www.brookings.edu/research/reports/2015/03/04-education-advocacy-whitehurst 2 Whitehurst, G. J., Stuit, D., Graves, C., & Doan, S. (2013) Measuring the Influence of Education Advocacy. Washington, DC: Brookings Institution.                     DOCUMENT REVIEWED: Measuring and Understanding Education Advocacy AUTHORS: Grover “Russ” Whitehurst, David Stuit, Claire Graves, and  Lauren Shaw PUBLISHER/THINK TANK: Brookings Institution DOCUMENT RELEASE DATE: March 4, 2015 REVIEW DATE: April 16, 2015 REVIEWERS: Robin Rogers, Queens College, CUNY, Sara Goldrick-Rab, University of Wisconsin-Madison E-MAIL ADDRESS: Robin.Rogers@qc.cuny.edu PHONE NUMBER: (347) 989-7869 SUGGESTED CITATION: Rogers, R. & Goldrick-Rab, S. (2015) Review of “Measuring and Understanding Education Advocacy.” Boulder, CO: National Education Policy Center. Retrieved [date] from http://nepc.colorado.edu/thinktank/review-measuring-education-advocacy 

NEPC Review: Measuring and Understanding Education Advocacy

https://scholar.colorado.edu/downloads/3484zh59c

NEPC Review: Measuring and Understanding Education Advocacy

Abstract

Similar works

Full text

Available Versions

CU Scholar Institutional Repository