Nothing concentrates the mind like the prospect of a hanging.

Attributed to Samuel Johnson and Mark Twain

Never underestimate the human capacity for denial.

Roger Molander
______________________________

Student Achievement

1. What are the effects of provider services on student achievement in reading/language arts and mathematics?

• Student achievement results for 2004-2005 yielded no statistically reliable effects for any of the SES providers.
• Two providers had “below standards” outcomes for student achievement for the 2005-2006 school year, with the rest having “insufficient information” to determine outcomes…..

While the 2004-2005 student achievement results yielded no statistically reliable effects for any of the SES providers, the 2005-06 results were mostly inconclusive in the sense of failing to indicate statistically significant differences between tutored and non-tutored students.  The two more rigorous analytical models that controlled for teacher effects yielded the negative effects for two providers in 2005-2006.  Given that these effects are evidenced for one year only and are based on relatively small sample sizes, it would be premature to make confident inferences about the effectiveness of services by these providers.  Numerous confounding variables associated with school, district, and home experiences could well exert more powerful effects than the comparatively limited number of hours in which students participated in tutoring.  There is also the possibility that the obtained effects are attributable to unique characteristics of the student samples that happened to receive services from certain providers in given districts.  However, consistency of results across multiple years would strengthen conclusions regarding individual providers’ true impact on the students served.   

Supplemental Educational Services in the State of Tennessee: 2005 – 2006 (Download below)
_______________________________

THIS IS NOT JUST ANOTHER STUDY.

The authors. Two members of the study team - Steve Ross and Bill Sanders, are both highly respected researchers who cannot be characterized as enemies of the market, SES or private providers. Ross has an extensive career in the evaluation of educational interventions on a massive scale. Sanders is one of the developers of the Tennessee Value-Added Assesment System and now with SAS Education. They know what they are doing and they are known by key members of the congressional education authorization and appropriations committees to know what they are doing.

The study. Unlike the recent study of edtech programs sponsored by the Department of Education, this evaluation examines some 45 individual programs and providers. The list includes national providers Club Z!, Failure Free Reading, Newton Learning (Edison), PLATO, Princeton Review, Education Station, and Sylvan. To add insult to injury, the evaluators found only two programs with statistically significant evidence of (in)effectiveness - PLATO and Education Station.

The method. These are the experts in education statistics. Rather than using only one model, they employed three. Each with its strengths and weaknesses, but all pointing in one direction:

"The results across models are consistent in showing no statistically significant provider effects in either Math or R/LA (Reading/Language Arts)....  Although several of the effect sizes in Math are moderately large, none is statistically significant.  Therefore, the differences indicated cannot be considered reliably different from zero...."

"The three analytical models yielded no statistically reliable effects for any of the SES providers.  Thus, it cannot be interpreted with sufficient confidence that students receiving tutoring services from any of the 12 providers in Math or 9 providers in R/LA differed in end-of-year achievement from non-tutored students."

––––––––––––––––––––––––––––––
The results are preliminary but point in the wrong direction entirely. If this were a mid-term exam, the student would definitely be on a path towards flunking the course. But here, flunking the course may not mean repeating the course - it's more likely that the course will not be offered.

Were your editor a member of the board of directors of an SES provider, he would have one question of management: "What are you doing to assure that we can demonstrate a statistically significant effect on our students' academic performance? We don't have a company if we can't do that one simple thing. It is not for our anti-market opponents to prove we don't work, we have the burden of proof to show that we do. How are you planning to do it within twelve months?"

Were your editor an investor he would be shaking his head and muttering: "NCLB passed in 2001. It's 2007. These firms depend on one funding stream under constant attack from the anti-privatization crowd. They can't demonstrate that they work. They have to show results after one year of tutoring, but they don't seem to have invested in showing it. This is/was not a good investment."

Were your editor the CEO of an SES provider that had good results demonstrating efficacy, he would have a call into his board,  investors and communications firm with the same message. "We have results and what we need to do is differentate ourselves from the pack of providers covered in the Tennessee study."

Were your editor the head of a trade group representing SES providers, he would be sending an e-mail to his members saying: We can no longer respond to claims that our programs don't work with white papers and press releases. We need a proactive plan to demonstrate effectiveness. We need to bring in some evaluators to help us come up with quick but relevant and credible evaluations designed to demonstrate that SES programs can work. We need to do it now, or SES will be highly vulnerable in the reauthorization process."

Were your editor the account excecutive for an SES provider that had good results, he would have a call into that CEO with the message: 'You and firms like yours need to get together to show these programs can work, or you and the SES program will be dragged down by all the stuff that doesn't. We need some kind of new trade group of firms commited to ongoing evaluation that have 'research-proven' programs. I can't help you much if we can't show Congress a critical mass of firms with programs that work."

Were your editor the political director of an organization opposed to school improvement providers in general and the SES program in particular, he would be telling like-minded colleages. "I really think that if  NCLB reauthorization is delayed more than a year, we just might be able to kill SES.  I think the providers won't be able to prove they work. I'm not even sure they will try. And even if they try, it's awfully late in the game for them to pull it off."

In the real world, the only conversation your editor has any reason to believe will actually happen is the last. Why? Because the opponents of SES need only sit back and wait for bad things to happen. Every other group would need to do something that takes time. energy, money and some vision, and there is little evidence to suggest any have such inclinations.

Your editor's sense, based on interactions with the members of SES management teams since NCLB passed, is that if confronted with the Tennessee study the reaction of most would be "this is just another study;" followed by an unspoken feeling of anxiety mixed with a recognition that they lack the expertise to adresss the problem, expressed as a list of reasons why evaluation is hard; thankfully interrupted by the tyranny of the in box - something that needs to be done today.

It is extremely hard for your editor, whose professional experience has demonstrated the promise of research-based school improvement programs, to watch the SES industry self-destruct - especially in slow motion. But unless some vigorous actions are taken, that is an increasingly likely outcome.