Supplemental Education Services under NCLB: Emerging Evidence and Policy Issues, by professor Patricia Burch of the University of Wisconsin-Madison, is the leftish anti-market Education Policy Research Unit's latest attack on the school improvement industry.

It's a summary of the available research on the market that's worth reading. As an "academic weapon" employed by the left in the "SES wars" the study is a failure.  Burch's heart just doesn't seem to be in it. Your editor is surprised the paper was released by this route rather than some more centrist think tank, given the position of EPRU's Director, Alex Molnar, as the media's "go to guy" for anti-industry quotes.

Your editor would have to call it a victory for those in favor of SES. SES providers' Achilles heel is evidence that the test score of a student who takes the NCLB tutoring opportuntity will improve more than a student who does not.  Burch looks at the best available evidence of efficacy and decides it is inconclusive. Perhaps a better academic than an anti-market partisan, she calls not for the end of SES in NCLB, but federally mandated studies.

That's a win for SES providers, who will now hold up the study and say: "We have a consensus that the SES program should be continued. Even an organization like EPRU, which opposes all market solutions to education, has said SES needs to be studied more, not killed."

Still, that's only politics. There's another way to read the paper - at face value. From an investor's or buyer's perspective, the report suggests that most SES providers' management teams have been asleep at the wheel.  Think a bit more about the headline of Burch's report: "Key part of NCLB Lacks Research Support." The implications are obvious: quite a few providers with a lot of money behind them and more than a few years of work in the field have done little to provide solid evidence that they add value. That's a failure of management.  If programs really do work and managers haven't been collecting, analyzing and making their data and studies public - there is a big blind spot in management that needs to be filled. If programs don't work or management fears they don't work, there are a lot of buyers and maybe more important - investors - who have been snookered.  Whether you are an investor or a buyer, it appears that you need to find out whether your partner firm is poorly managed or just has a poor program.

If you are a provider - you want to prove you were in the first category, but are no longer.

You editor has heard all the reasons why it's hard to evaluate SES. The two leading excuses are: "It's hard to get districts to release the data" and "There are other things going on in the schools and the kids' lives - how do you tease our effect out?" With experience as a Senior Social Scientist with RAND Education, and New American Schools Chief Operating Officer responsible for the evalutions of eight nationally distributed school interventions, your editor can safely say that these are the same problems faced by literally every other provider of school improvement services. There are enough providers with enough solid evaluations to demonstrate that SES providers arguments are born of ignorance, a cop out, or obfuscation. None are satisfactory perceptions.

Moreover, The What Works Clearinghouse bar on program evaluation is really quite low. See the April 4 issue of New Education Economy® for an example.

Methodologically, there is nothing unique about the SES evaluation problem. An inconvenient truth perhaps, but still truth.

__________________________________

The data and analyses presented here highlight limitations in the current law and implementation of SES: low participation rates; limited services available for English Language Learners and special education students; and, state and district capacity to implement the law and monitor program quality. Even with improvement in such areas, however, it is unclear how
SES might affect academic achievement, because existing research leaves many questions unanswered. Similarly, existing research offers little information about specific conditions that support positive outcomes. To make well-informed decisions in the future, policy makers will require additional empirical evidence....

The SES reform reflects such marketplace values as outsourcing, limited government regulation, competition and choice. It also incorporates elements of government contracting, in that school districts contract with outside firms to provide the mandated tutoring. And it incorporates features of vouchers and school choice by providing a fixed per-pupil allocation for eligible students and then allowing parents the choice of providers....

For SES, the evidence is indeed quite limited.... The evidence that does exist tends to fall into one of three categories: evaluation studies by third parties (for example, nondistrict, non-provider entities); evaluation studies conducted by local school districts; and academic and scholarly research. To date, the U.S. Department of Education has not conducted or commissioned a national
evaluation of the program....

Each of these three categories generally offers insight into a different aspect of the SES enterprise. Third-party evaluations provide a broad overview of implementation patterns occurring in local school districts, and they yield information on district capacity and the providers’ role. School district studies, such as those conducted by Chicago and Minneapolis, evaluate the effect of SES on their students’ achievement and test-score gains. Scholarly research addresses questions of SES
program design and implementation, primarily as they relate to lowincome and minority students. These studies inform the following segments of the brief, which examine what is currently known about SES....

The research base on how SES may affect student performance is virtually non-existent. The only relevant studies were conducted by two urban school districts: Minneapolis and Chicago....

The Minneapolis Public Schools (MPS) conducted two paired studies to explore SES impact. The first study began by determining reading gains for SES students as indicated by the Northwest Achievement Levels Tests (NALT), a national norm-referenced test of 500,000 students. Researchers compared two test scores for 602 students who all took the same two NALT reading tests: one in the spring of 2004, when they were in grades 2-6, and one in 2005, when they were in grades 3-7.
After determining the growth rate for these SES students, the study compared it to the rate in national grade level norms.

The second study was based on the Minnesota Comprehensive Assessments (MCA), used to meet NCLB...  accountability requirements. This second study compared students who received SES from the district’s largest providers, with students who had similar demographic characteristics but who did not receive SES. This group included all SES students who took NALT
reading tests in the spring of 2004 and MCA 2005 reading tests in grades 3, 5 or 7 in 2005.

The MPS studies found that students receiving SES did not perform as well as the matched samples. Further, no significant difference appeared among SES providers as determined by NALT annual reading gains. No provider serving 10 or more students produced achievement gains averaging close to 100 percent of expected academic growth indicated by national norms. For example, 561 students receiving Education Station services averaged 71 percent of a year’s growth, while 92 students receiving Newton Learning services averaged only 67 percent of a year’s growth. Overall, the average growth for SES students was only 66 percent of the national norm....

The Chicago Public Schools (CPS) report also attempted, among other things, to determine any achievement score gains associated with SES. To assess achievement, researchers analyzed test score data from the 2003-04 and 2004-05 school years. For students in grades 4-8, gains in reading and mathematics scores on the Iowa Tests of Basic Skills were compared for eligible students who did, and did not, receive SES services; results were then analyzed in terms of expected gains. Across grades 4-8,
students receiving at least 40 SES tutoring hours in 2004-05 showed higher gains in both reading and mathematics than eligible students who did not receive SES services. Further, students who received at least 40 total hours of tutoring had higher math and reading gain scores than students who received fewer than 40 hours. The study also compared the benefits of tutoring across the 17 providers included in the study. Students receiving SES from seven providers (one of the seven was the CPS itself) had higher reading scores than the district average. The cost of the program appeared to have no relationship to the score gains....

Therefore, it is recommended that policy makers... Commission federally-funded, comprehensive evaluations to
determine.... to what degree SES may affect student achievement

Patricia Burch, Ph.D., University of Wisconsin-Madison Supplemental Education Services under NCLB: Emerging Evidence and Policy Issues, The Great Lakes Center for Education Research & Practice, May 2.