"Reading Wars," "math wars," "voucher wars," "charter wars" and now Harvard's Paul Peterson has fired off a salvo in the "school contracting wars."  The basic feature of this conflict is the dueling evaluator, funded by those with a stake in the outcome of the study. The basic tactic is the press release, announcing a study that demonstrates results favorable to those who bankroll the hired gun's work, combined with attacks on studies that suggest a contrary assessment.

In his press release, Peterson, who is also a veteran of the voucher wars, announces a study -  funded in part by Edison, along with foundations that favor school contracting -  that finds the performance of contract schools - including Edison's - superior to other Philadephia public schools, and then adds that a RAND study with the opposite conclusions is "not... what it purports to be."

Students who attended privately-managed Philadelphia schools made larger test-score gains on the Pennsylvania State System of Assessment (PSSA) between 5th and 8th grade than did other students in these grades in the Philadelphia school district as a whole, according to a new study prepared by Paul E. Peterson, director of the Program of Education Policy and Governance (PEPG) at the Kennedy School of Government at Harvard University.  They also performed better than students at schools managed by the school district’s own Office of Restructured Schools (ORS)....

Results from the PEPG study differ from those reported in February 2007 by a team of scholars associated with the RAND Corporation and Research for Action, a policy-oriented research group in Philadelphia. That study found students at privately-managed schools learning no more than other students in the school district.

Using publicly available data on student test-score performance between 2002 and 2006, Peterson tracked the test performance of two groups or cohorts of 5th graders at elementary and middle schools to see whether, by the 8th grade, those attending schools under private management learned more than students in the district as a whole, as indicated by their performance on the PSSA, which is used for accountability purposes under the federal law, No Child Left Behind. Peterson also compared the students’ achievement with that of students in ORS-managed schools.
        
“Gains during the middle years of schooling are of special significance,” Peterson said, “because on nationwide tests, those are the very years when many students are seen to be losing ground.” ....

Peterson points out that the RAND study was not the quasi-experiment it purports to be, and it fails to adjust adequately for student background characteristics and peer-group effects.  Nor did the RAND study focus exclusively on the PSSA, says Peterson, the curriculum-aligned test for which schools were being held accountable. Instead, it included a mix of results from three different tests that differ from one another in important ways.

Whether or not competition stimulated a rise in district-wide performance could not be ascertained.  Philadelphia test scores have risen substantially, but the data are not available to ascertain whether those gains exceed gains achieved in other, comparable school districts.  RAND’s attempt to address this question compares schools that have strikingly different student populations, Peterson pointed out. It was also not possible to conduct a cost-benefit analysis, because the School District of Philadelphia does not make available information on per pupil expenditures by school.  Statements about cost-effectiveness in the RAND study lack the supporting evidence on expenditures per pupil, Peterson said....

Support for this research was provided by the Lynde and Harry Bradley Foundation, the John M. Olin Foundation, and Edison Schools.

Press Release, April 10.

The sad fact is that Peterson may be right, or that his results may be equally valid to RAND's. Unfortunately, both possibilities will be lost in the forthcoming duel between RAND and its defenders and Peterson and his. Sadly here  evaluation looks to have been enlisted to obscure understanding rather than shed light on an important question. 

RAND appears to have no particular axe to grind. In contrast, circumstances imply that the Peterson study was intended as an answer to RAND's, the findings (conveniently) coincide with the interests of his funders, and Peterson goes out of his way to attack RAND's methodology. Whatever the merit of his work, by setting the story up as "Peterson v. RAND,"  Peterson has assured that RAND's findings will win the battle for credibility with the vast middle ground of policymakers and people who simply want the best for kids. The case more favorable to a school improvement industry provider will be dismissed as self-serving.

There are some lessons here for the industry:

• First, it is never a good idea to hire a controversial political advocate to prove his point. The school improvement provider definitely raises the advocate's visibility, but whatever baggage the advocate carries gets attached to the provider. And the study lacks credibility per se with any but the provider's own choir. In short, is is worse than a waste of money. It actualy harms the provider's image.

• Second, the industry dies if its programs can't improve student performance, and if providers can't prove it.
Every firm in the industry is harmed by the lack of agreement on what constutites an adeaquate, fair evaluation of an educational program. The industry needs to reach out to the evaluation community, government policymakers and educators to come to some methodological standards that protect students, encourage innovation and help educators make decisions.

Jason Cascarino's column "Mission Smarts:  Definitiveness versus Probabilities in Education Evaluation" in this week's New Education Economy® offers readers a useful perspective on the role of evaluation.  To access New Education Economy® for free until June 1, you must  create a "User Account". Go to the upper left hand corner of this webpage to start the process.