Your editor did not plan on becoming a playing field for this contest, but Brian Gill, a Senior Social Scentist in RAND's Pittsburgh and lead author of the Philadelphia study, has provided the think tank's response to Paul Peterson's critique.

It's a reasonable, reasoned and professional response to the points raised by Peterson, and helps the rest of us understand how to make sense of the differences in the two studies, and the basis of RAND's methodological decisions.  It is worth reading, if only to see that issues of methos can be discussed in terms the informed layperson can digest without too much work, and this is a language that every employee of every provider should try to learn.

This link is to RAND's study.

–––––––––––––––––––––––––––––––––

RAND response to Paul Peterson’s report on student achievement in Philadelphia
9 April 2007

1. Peterson’s analysis looks at a limited number of grades (5th and 8th) and schools, excluding all schools for which 5th-to 8th-grade gains could not be examined.  He examined 19 of 45 privately managed schools and 8 of 21 restructured schools. The RAND/RFA analysis includes all of the privately managed schools, all of the restructured schools, and students in all of the tested elementary and middle-school grades.  It is not surprising that an examination limited to less than half of the relevant schools and a subset of the relevant grades would produce different results.  Average results are better assessed by including all the schools than only some of them.

2. We don’t agree that school performance is best assessed by examining only state test results to the exclusion of other, nationally normed tests.  Indeed, many education researchers argue that low-stakes tests produce results that are more valid than do high-stakes tests, because they are less susceptible to “teaching to the test.”  Results will be more robust if multiple assessments are used.  Indeed, the School Reform Commission would be wise to consider test results as only one component of a multi-faceted assessment of the performance of private managers.

3. Nonetheless, there is no evidence that the RAND/RFA results are sensitive to the particular tests used.  Two other reports—one conducted by the Accountability Review Council (ARC) and one by scholars at Johns Hopkins University (MacIver and MacIver) examined PSSA data and reached conclusions similar to ours.  The ARC report, unlike the Peterson report, incorporated PSSA results from all of the relevant schools.  Dr. Ken Wong or Dr. James Lyons can discuss the ARC report.  The Johns Hopkins report, like the Peterson report, examined gains from fifth grade to eighth grade, but it had the advantage of student-level data, permitting a true cohort analysis rather than a quasi-cohort analysis.

4. Demographic variables and other variables that do not change over time (e.g., a student’s race) are not explicitly included in our analysis because our method compares students to themselves.  Peterson fails to note, however, that we conducted secondary analyses that explicitly examined whether privately managed schools or restructured schools had differential effects for particular groups of disadvantaged students, such as African-Americans, students with limited English proficiency, and special education students.  We found no evidence of differential effects, and therefore no evidence that these factors would have affected the overall results.

5. Peterson’s suggestion that our analysis is not quasi-experimental is inaccurate.  A pre/post examination of changes in achievement is a prototypical quasi-experimental design.  Our report explains in detail why we included students with a minimum of two achievement test results rather than three: in essence, this allowed us to be more comprehensive in our analysis, with no evidence that it would bias the results.

6. We agree with the Peterson report that the district would benefit from greater transparency in the flow of resources to schools.  As Peterson notes, the RAND/RFA report did not include an in-depth analysis of the funding available in privately managed schools and other schools in the district; this was beyond the scope of our work.  We accepted the SRC’s explanation, made at the time contracts were issued to the providers, that the additional resources given to providers fully compensated for prior disparities and provided additional funding above that available to other schools in the district.  If resource flows subsequently changed due to changes in teacher or student populations, this initial comparison may not have remained accurate for later years.  We would encourage the district to support the work of Marguerite Roza of the University of Washington in conducting a rigorous analysis of resource flows.

7. The RAND/RFA report, the Peterson report, and the ARC report agree on one critical point: Student achievement across Philadelphia has increased substantially since 2002, in district-managed schools and privately-managed schools alike.  We agree with the Peterson report that the key question is the extent to which competition induced by private managers may have been responsible for producing district-wide gains. We found no evidence to attribute the district-wide gains to competition from private managers, given that the system was not set up to maximize competitive effects and given that other schools in PA had similar gains despite the absence of private competition. Nonetheless, this is not a question that can be resolved empirically, and reasonable observers may disagree about whether competition played a significant role in producing district-wide gains (as we acknowledged in our report).

8. RAND and RFA are not permitted to make student-level data available to other researchers, under our agreement with the School District of Philadelphia.  The district owns the data and the district must decide whether to make it available. We are willing to assist in making the data available if appropriate protections to confidentiality are in place and if the district desires our participation.

__________________________________

It would be helpful to decisionmakers great and small - teachers trying to decide whether they would like to use a particular program or legislators grappling with the question of whether to encourage private sector engagement in public schools  - if analysis were treated as a decision aid rather than persuasive writing. 

The state of the evaluation art is uncertain enough as a guide to decisions.

Employing it in pursuit of ideology or self interest - whether it is by government officials in support of phonics under Reading First, or their consultants who authored texts sold under the program; partisans for or against contracting, vouchers and charters; or providers who see evaluation as a marketing problem - only undermines the school improvement industry.

Its future depends not only on the promise of improved student performance, but the credibility of evaluations that show it.