First, the Education Industry Association's (EIA) email and press release; then your editor's reply; then comments from SES evaluator Prof. Steve Ross of the University of Memphis and Director of its Center for Research in Education Policy (CREP); and finally, EIA Executive Director Steve Pines' nonresponsive, but final, word.
-------------------------------------------------------

From EIA:

Dean –
 
Steve Pines and I figured you’d be interested in EIA’s position on the RAND’s SES evaluation, released earlier today by US DoE.
 
Let us know if you have questions.
 
Thanks,
 
Steve Drake

Steven Drake Associates, LLC
12912 Two Farm Drive
Silver Spring, MD 20904
TEL:  301 680 0585
e-mail:  sdrake@verizon.net
 
 
FOR IMMEDIATE RELEASE
CONTACT:        Steve Drake
                                TEL:  301 680 0585
                                Email:  sdrake@verizon.net
 
 
EIA Hails Results of U.S. Department of Education Study on Federally Funded Tutoring, Says Program Should be Strengthened in Reauthorization of
“No Child Left Behind”
 
Statement of Education Industry Association (EIA)
Executive Director Steve Pines
 
 
WASHINGTON, DC – June 27, 2007 – The Education Industry Association (EIA), which represents many private providers of supplemental educational services (SES) under “No Child Left Behind,” is very pleased with the results of the report conducted by the RAND Corporation (State and Local Implementation of NCLB: Volume 1 – Title 1 School Choice, Supplemental Educational Services, and Student Achievement), and released today by the U.S. Department of Education.  This scientifically rigorous study indicates that the federally funded tutoring program had a statistically significant, positive effect on student achievement in reading and math in five out of seven large urban school districts examined during the past several school years.
 
The report further found that SES helped the lowest-achieving students, and in a majority of districts studied, had a particularly beneficial impact on student achievement among key student subgroups, including African American and Hispanic students, as well as students with disabilities and limited English proficiency (LEP).  These students are often the farthest behind academically, so it was pleasing to see that SES tutoring was shown to be an effective life-line helping to close the achievement gap with their more advantaged peers. The study also took note of the cumulative, positive effect of multiple-year SES enrollment on student achievement.
 
EIA believes the report’s findings, in combination with the several other state- and district-sponsored SES evaluation studies released in the past year which indicate that majorities of low-income parents and school officials find SES to be beneficial, provide strong evidence for the continuation of the still-young initiative in a reauthorized “No Child Left Behind.”
 
When more wealthy families seek help for their children, they call a tutor.  SES marks the first time in public education policy that the playing field is leveled for lower-income families, by giving them access to the same high quality tutoring that their more fortunate peers have enjoyed for decades.  The independent research firm, RAND Corporation, has documented what parents from all backgrounds have known for years -- tutoring works!
 
EIA looks forward to participating in upcoming discussions on NCLB reauthorization, and will seek enhancements to the tutoring initiative.  These include more resources for States and school districts for evaluation, parent outreach and program administration, policies which reserve unspent tutoring funds for supplemental services in the following year, and the requirement that schools open their facilities to providers to ensure the safest, most  convenient location for service delivery.
 
About the Education Industry Association
The Education Industry Association works to expand educational opportunities and improve student achievement for learners of all ages by infusing American education with market-based drivers of service, innovation, and results. Founded in 1990, EIA is the leading trade association for private providers of education services, suppliers, and other private organizations in all sectors of education. EIA currently has more than 600 individual and corporate members.  For more information call EIA Executive Director Steven Pines at 800-252-3280 or visit www.educationindustry.org.
-------------------------------------------------------

Your Editor's Reply:

Two Steves (well three because I'm copying Steve Ross for comments):

Thanks for the input. There will be a lengthy discussion of this in Friday's edbizbuzzcom, including your press release.

If you look at the facts as we know them across all the studies so far - and more important as Steve Pines has portrayed them - here's were we stand:

1. (Unlike what he has argued in the past - e.g., "too blunt an instrument" in his letter to the Chicago Sun Times), by applauding the RAND study, Steve P. has admitted that the state of the art in education evaluation can measure whether SES has any effect on student performance. This is the point of RANDs finding of "statistical significance." However big or small the effect, RAND is saying it is attributable to tutoring programs.

2. "Statistical significance" alone says absolutely nothing about "educational significance."  RAND does tell us it is generally higher than zero.

3. The RAND study says almost nothing more about educational significance. But it is a meta study - a study of other studies - and we know that none of those studies showed improvements in tests scores  that would justify breaking out the champagne - at most a handful of points up on whatever test in whatever study you choose. Steve P. admits as much, again in his letter to the Sun Times ("SES... is likely to have small influences on state standardized test scores'") .

4. SES programs are pretty costly to the taxpayer - in the vicinity of $1000 per per child.

To summarize, here's what EIA has accepted by its statements to date:
• Whatever the size of the effect on student test scores, the impact of SES on test scores can be measured, and RAND has demonstrated effects that are attributable to SES.
• On average, SES programs can only be expected to offer students a small improvement on the tests that demonstrate their proficiency in math and reading.
• The cost of this average improvement is about 10 percent of the total amount the taxpayer spends on each student each year.

I am on record as a champion of a market in school improvement since the early 1990's. I would like SES programs to work, but I don't see the advantage of pretending we are drinking Krug when it's really stale ginger ale. Markets only work for society where there is good and complete information. The SES program was intended to help kids in schools that are not making AYP - that is, not helping enough kids demonstrate proficiency on state tests fast enough. Therefore the only measure of merit for SES providers under NCLB is whether and the extent to which they improve student test scores, and at what price. The price here is far too high for the results. Frankly, at the moment,  we can get better student test results at a lower cost from many other school improvement options.

I also think that, by its after-school nature, SES has a lot of potential for helping us learn how to design and deliver differentiated instruction at scale.  If the tutoring programs could show significant payoff on tests, I'd be happy for the taxpayer to subsidize poor parents' choice of tutoring programs at the present price. But the argument that poor parents should be permitted - even encouraged - to choose whatever tutoring program is out there - no matter how small the educational value-added, as long as the providers comply with other aspects of law and the parents like them, makes as much sense for SES as it does for charter schools. Aside from the fact that it's a sham transaction offering little benefit to the poor for a lot of cash (a familiar critique of products and services sold to the underprivileged), the taxpayer has an interest here, and that interest is student proficiency as measured by standardized tests. The rest is not much more than frills.

As I see it, EIA, acting in its role as the SES trade group, has three choices:

• The road you are going down now leaves your highest performing SES providers in the same boat as your lowest. In this regard, you are demonstrating the weakness of most trade groups, and the best providers will be dragged down with everybody else, as the points I've made above become obvious. This will be especially true if NCLB reauthorization takes place under a federal government controlled by Democrats.

• Separate out the best providers, and show they have educationally significant outcomes at decent prices - and base the case for SES as is in NCLB II on their performance. I don't think EIA's budget can stand that decision, because it would eventually lead the rest to leave the organization. I also suspect the new club would lack the critical mass required to sustain the current concept of the SES program.

• Admit SES is not ready to be treated as a consumer market for the economically disadvantaged, but argue that the trends are positive and hold promise. Figure out how much is spent on SES now and fence something like that amount off for a competitive R&D program where districts match several SES providers with schools in need of improvement, addressing the shortcomings of the current models of delivery/implementation and building on their strengths. Frankly, I think providers would find this much more profitable than the current market structure.

As Chris Whittle might say:

Something to think about,

Dean
---------------------------------------------------

Comments from Steve Ross:

As usual, I agree and disagree.  But I think we’re getting closer to understanding the SES strengths and limitations.
 
The RAND finding of .9/100 of a standard deviation gain makes sense to me.  That’s exactly what I’d expect from a 30-40 hr. supplementary program, given that the best whole-school reform programs produced only twice that (and the vast majority produced weaker effects!).
 
Statistical significance doesn’t mean much when you have gigantic numbers, so best not to over-emphasize that.
 
I’d like to think that there are “many other school improvement options” that reliably produce gains that are more economically justifiable, but what are they?  In 20 years of research, I’m still searching.  It doesn’t seem to be technology, Reading First, small class sizes (except for STAR) study, phonics or whole-language, or even Direct Instruction and SFA.  So, SES tutoring with its .09 effect size might well be justified for $1k per student, assuming we can show an effect of that magnitude to be reliable and replicable. (The jury is out, since the RAND methodology and sampling is adequate but not highly rigorous).
 
Dean raises good points at the end of his reaction.  The idea the 2,000+ providers can all deliver the goods to achieve a .09 effect size just doesn’t seem logical. Determining the best and most economical ways to deliver SES seems the best way to save it from elimination.  If the weaker and most expensive methodologies were weeded out and we could show a +.15 effect size for $800 per student, SES would seem a slam dunk to me.
 
On the other hand, Dean is putting all the eggs in the state-mandated AYP basket.  One-on-one tutoring can do many other beneficial things for kids. In Memphis, for example, we hear about kids who are gaining discipline and self-esteem at 3-4pm instead of hanging out with gangs.  The Rand study doesn’t capture those potential dividends of the $1000 cost.
 
Keep up the dialogue.  It’s healthy, I think.
 
Steve
------------------------------------------------------

Steve Pines' Last Word:

Dean

Thank you for invitation to respond.

While we are pleased with the news from the recent RAND study on SES’s impact on student achievement, we also know that it will not the last “word” on the dual questions of cost/effectiveness. On-going evaluations by States and others will help to fill in the growing body of research on this still-new education initiative. EIA’s members are also organizing a research study that will also examine some of the potential impacts from SES tutoring, while considering length of contact time, ratios, and perhaps costs, too. This study will be conducted by independent researchers as well with the participation of school districts and state education agencies.

Sincerely,

Steve Pines
EIA Executive Director