The reaction of Education Industry Association Executive Director Steve Pines tracks what your editor suggested to expect from most SES providers.

Read Pines' response. Then let's deconstruct it. Then read one of the study's author's, University of Memphis professor Steve Ross, comments on the exchange.

But first, let's be clear about one thing. Your editor is not raising the alarm because he is opposed to a market, doesn't like SES providers, or doesn't think tutoring can work. He believes in the market, thinks the offer of SES is an appropriate response to students in schools in need of improvement, and expects that some SES programs will work very well. He is raising the alarm because the market will be undermined, SES will be thrown out of NCLB II, and tutoring programs that could really help kids will never reach them because most SES providers are neglecting their evaluation responsibiliies. It's bad business, bad public policy and bad for kids.  But there is time to turn things around.

–––––––––––––––––––––––––––––

Dean

Did we read the same report from TN?

It basically said there was no statistically significant (neutral) impact on achievement; largely due to insuffient data from providers at this time. Even the conclusion from Dr. Ross makes the balanced statement of an evolving and maturing system that shows parents are enthusiastic, monitoring is on the increase and as sample sizes increase, there will be more data on which to conduct statistical analysis of impacts on reading and math test scores.  

He also correctly observes that progress from a modest number of tutoring hours may only have a slight affect on student achievement as measured by state tests--thus the need to include parents, teachers, administrators, etc, in the evaluation rubric.

This type of finding is going to be the norm unfortunately for a while, but it not the damning type of research that is the death sentence your column seemed to indicate, unless I am missing something.

And yes, providers do have their own pre-post test data and even at least one (ED Station-KLC) has gotten  the cooperation of a couple of districts to correlate its gains to student progress as measured by the district testing. But few have this access to student data held by districts, and most outside observers may dismiss providers' pre-post data as unreliable because it is not independently verified (without the cooperation of LEAs)- thus we - they are trapped.

Thanks,

Steve

__________________________________


Pines: Did we read the same report from TN?.... It basically said there was no statistically significant (neutral) impact on
achievement; largely due to insuffient data from providers at this time.

Deconstruction: Either Pines does not understand statistics or he is deliberately obfuscating their meaning. I prefer the former interpretation.

In plain English, the lack of statistical significance identified by Ross et al or the "neutral" impact note by Pines means that as far anyone could see, taking the tutoring services offered by SES providers made no difference to a student's test scores (except in the two cases where it seemed to hurt them). The Tennessee study's authors are not saying they lack the data to say that SES providers had no impact on student test scores in the period studied. They are saying they do have it. What they want is another year's data to see whether this sorry result is repeated.

In any case, although Pines seems nonplussed with the finding, "our services are no worse than doing nothing at all" is not exactly a formula for convincing taxpayers or Congress that the way to help kids from schools in need of improvement is spending more money on SES in NCLB II. 

Pines: Even the conclusion from Dr. Ross makes the balanced statement of an evolving and maturing system that shows parents are enthusiastic, monitoring is on the increase and as sample sizes increase, there will be more data on which to conduct statistical analysis of impacts on reading and math test scores. 

Deconstruction:  Anyone involved in school reform for any length of time has read how much parents love their schools and heard educators use those statistics as a defense against changing how they do business. It was insufficent then and it is insufficent here. The great debate over accountability in NCLB is whether we are going to be driven by improved performance on test scores, or a host of other "feel good" factors like parent satisfaction, or the system's historic love for the measurement of such inputs as the extent of monitoring. The reason SES providers get a chance to offer their services is that schools have failed to meet adequate yearly progress targets on the percentage of students who are proficient on state tests. In many of those schools, parents still love their teachers and principal. If Pines wants to use parental satisfaction as an indicator, he has just cut the ground out from under the provision of SES altogether. Everyone will live or die by their contribution to student performance, or there will be no SES.

Like No Child Left Behind, your editor favors a market in school improvement services based on performance. The other stuff is nice to have, but if you can't improve student test scores first, the nation's taxpayers can't afford it.

Finally, your editor has yet to meet a researcher or read a research report that did not call for more data and further study. The "best available evidence" is what policymakers have to go on. And here the best available evidence isn't even close to promising. By the way, speaking as a former Senior Social Scientist with RAND Education, the researcher's code phrase for "I think this stuff might work, but I don't have the facts to prove it to my fellow evaluators" is "promising."  That phrase is not to be found in the Tennessee study.

Pines: He also correctly observes that progress from a modest number of tutoring hours may only have a slight affect on student achievement as measured by state tests--thus the need to include parents, teachers, administrators, etc, in the evaluation rubric.

Deconstruction:  First, "only a slight effect on student achievement as measured by state tests" is hardly the bumper sticker the folks in marketing would suggest Pines takes up to Capiol Hill when he argues the case for SES in NCLB reauthorization. 

Second, as Pines knows (because your editor has told him repeatedly), every single educational intervention ever studied suffers from the reality that other factors affect students. The point of analytical tools like the three models employed in this study is to separate out, to isolate, the effects of the intervention. And yes, the impact of tutoring might be slight, but it is still measurable. This is why the report refers to both effect size and statistical significance. What the three models show so far is no effect of any statistical significance (except in those two negative cases), which covers both concepts.

Pines: This type of finding is going to be the norm unfortunately for a while, but it not the damning type of research that is the death sentence your column seemed to indicate, unless I am missing something.

 Deconstruction: "Just another study" or "never underestimate the human capacity for denial." Pines takes your editor's characterization of the Tennessee study too seriously and not seriously enough.  Too seriously, in that the study was never called a death sentance, but a mid term exam pointing in the direction of flunking the course. Not seriously enough, in that anyone familiar with research, this research and these researchers can see that it is a clear warning sign, but there is time to turn things around if the student wakes up and works very, very hard.  Pines dismissive exxageration of your editor's characterization tells you he does not want to be disturbed from his slumber. Yes, EIA's Executive Director and the SES providers' chief spokesperson is missing something when it comes to the evaluation of education programs.

Pines: And yes, providers do have their own pre-post test data and even at least one (ED Station-KLC) has gotten the cooperation of a couple of districts to correlate its gains to student progress as measured by the district testing. But few have this access to student data held by districts, and most outside observers may dismiss providers' pre-post data as unreliable because it is not independently verified (without the cooperation of LEAs)--thus we -they are trapped.
Deconstruction: First, Pines should know that pre/post studies simply do not demonstrate a program's efficacy - no matter how many one does. It's a pretty flimsy defense against the charge of having done no research.

Second, "We've done a little research but districts won't cooperate so we are pretty much screwed" is not the kind of talk that builds confidence in management. This is not the kind of talk that will attract investment to SES providers , nor is it the kind of talk that should make SES providers eager to join EIA. It does strike your editor as a very good formula for a process leading a point where Congress is willing to trade away the continuation of SES in NCLB for something more important.

Pine's response to edbizbuzz is either a cop-out or a sign that SES providers have never taken the time to hire evaluators to help them come up with a serious plan for immediate, mid-term, and long-term evaluations - given whatever constraints they operate under. Other school improvement providers have done all kinds of credible research, with and without school districts cooperation.  Next week's issue of New Education Economy® will show a very creditable effort by Voyager to explain its plan of program evaluation to potential clients.  Scientific Learning has it. Success for All has it. RAND has done it for Edison. CPRE has done it for America's Choice. Your editor could go on and on and on with providers that have good research. 

The letter Executive Director Pines should have sent, might have has everything he put in above, but would have carried an entirely different message if he had laid out what providers and EIA are doing to demonstrate that their programs work. Instead, it reads as apology for doing as little as possible as late as possible, and playing the educators' age old contest - "the blame game." That is not in the best interests of SES providers, the school improvement industry, the taxpayer, or students.

If SES providers want to lose the support of hard-core, results-oriented school reformers in the public education space, they only need to keep writing letters like this. It's perfectly reasonable to have a less-than-perfect program - if you can demonstrate a committment to research and evaluation, and if you can show progress. Anyone who has tried to change schools knows how hard it is, but they also know it cannot be done without devoting real resources to research. The SES market segment has had years to demonstrate this committment. Some providers have to some extent. But most have adopted the Pine's position - addressing the troubling studies with words, making excuses for doing virtually no research, and kicking the can down the road. 

If there was ever a good example of why quality providers commited to student performance who back up talk with resources need to form their own trade group - this is it.

––––––––––––––––––––––––––––

May 4

From Steven M. Ross. Ph.D. Faudree Professor and Director
Center for Research in Educational Policy
The University of Memphis

Unfortunately, I’m on limited email in Turkey and Greece until the 16th.  I think that both you and Steve make valid points.  I do seriously question, on the one hand, the ability of providers to produce detectable effects early in the game on the state tests. Goodness, even the intensive curriculum-based programs, like SFA, DI, and Reading Recovery, often don’t show large effect sizes early or ever in some implementations.  Tutoring may not overcome the myriad of much more influential confounds, unless sample size is huge.  Still, I like your points very much  about the industry needing to take a more improvement oriented rather than defensive posture.  The tutoring, in my opinion, MUST be directly linked to the classroom teaching or it will fail.  Right now, it is intermittently or weakly so.
 
Take care, Dean.  I’m out of time for now.