Looking back with our present knowledge of tutoring’s limited role in improving student proficiency in math and reading, SES providers made a great mistake allowing the definition of “research based” to drift at the federal level. The states are setting a higher standard. The whole school improvement industry will pay.

It is hardly surprising that state education agencies have become a lot more interested in the efficacy of tutoring programs offered by SES providers. With so many credible evaluations suggesting that few SES programs have educationally or statistically significant effects on student achievement, they would hardly be doing their jobs if they did not start taming the wild west. But it is also true that most state agencies are less than eager to make life difficult for their constituent districts by encouraging competition from the private sector. In any case, the result is that what SES providers intended to be a lower standard of proof when NCLB was negotiated in 2001 is morphing into “scientifically based research.”

Consider Louisiana’s requirement for evidence of program effectiveness in its most recent application for SES providers, due July 31 (download below).

“Provide descriptions of the evidence of effectiveness for the specific programs or services that you will be offering in Louisiana… Please cite all sources of evidence. The Department of Education expects that the evidence submitted will be that for the instructional program to be delivered to students at the costs cited in this application. (Emphasis in original.)

These indicators are listed in order of priority, with strongest consideration given to evidence of positive impact on student achievement on state, district, or other nationally available tests, particularly for low-income, underachieving students.  Evidence of positive impact on additional outcomes will also be considered (e.g., school grades, family/parent satisfaction, student discipline, student attendance, and/or retention/promotion rates), as well as provider-conducted studies, database information on student outcomes, and other sources of evidence.  However, please note that priority will be given to third-party, independent research.  Please ensure that data collected within the past three years is utilized.”

Indeed, for its standard of program inclusion under SES, Louisiana has explicitly adopted the definition of scientifically based reading research contained in U.S. Department of Education’s 2002 guidance for Reading First (which, ironically, the department went on to ignore in its implementation of that program)

“Your application in this area will be evaluated based on the extent to which you are able to clearly and specifically explain how the key instructional practices and major design elements of your program are (1) based on sound research, and (2) specifically designed to increase student academic achievement, preferably among students with the same demographic profile. A definition of scientific research can be found in the Reading First Guidance on the Department of Education website at www.louisianaschools.net under Student and School Performance;  Division of Student Standards, Accountability and Assistance; Reading Programs; Reading First.”

The definition reads as follows (download guidance below):

1. Research that applies rigorous, systematic and objective procedures to obtain valid knowledge relevant to reading development, reading instruction, and reading difficulties.  This includes research that:

a. Employs systematic, empirical methods that draw on observation or experiment;

b. Involves rigorous data analyses that are adequate to test the stated hypotheses and justify the general conclusions drawn;

c. Relies on measurements or observational methods that provide valid data across evaluators and observers and across multiple measurements and observations;

d. Has been accepted by a peer-reviewed journal or approved by a panel of independent experts through a comparably rigorous, objective and scientific review.

In 2001, the tutoring firms pushing for an SES program in NCLB had a compelling argument for the lower “research-based” criterion - they lacked much in the way of research that could pass the “scientifically based” standard. It is now 2007, and most still lack scientifically-based research that suggest program effectiveness. While other providers in other NCLB programs have made this their own responsibility - indeed a central part of their corporate cultures, most SES providers left the problem to government and nonprofit education researchers. So in one respect, the mess they are in today, of study after study showing no great impact on school performance, is their own fault.

If all that's at stake here were the fate of companies whose managers - and boards - lacked foresight, your editor would be perfectly happy to let them all go out of business and leave it there. That's how markets should work. The SES program exists for kids, not companies.

But much more is at stake.

First, the entire school improvement  industry – is tainted by the SES providers failure to demonstrate efficacy and fall back to arguing lawful operation and parent satisfaction as their raison d'etre (i.e. SES, by itself, is likely to have small influences on state standardized test scores, the tutoring program’s impact should also be measured in terms of parent satisfaction, principal and teacher opinions, and compliance issues related to program implementation). As a matter of public perception, it really doesn't matter that so many school improvement providers were founded on research and evaluation, have made it part of their core operations, and internalized it as a day-to-day value.

Reputations are hard to build and easily wreck, and the public doesn't know the difference between a given SES provider and, say, Scientific Learning Success for All or Carnegie Learning in the broader industry. The school improvement industry's reputation for quality is no higher than it's lowest quality provider.

Second, the definitions of “research based” for SES, “scientifically based reading research” for Reading First and “scientifically based research” in NCLB are linked; making each a matter of industry-wide concern. State decisions to elevate the definition of "research-based" to that of "scientifically-based research" will only make it more difficult to rationalize a neglected, confused - and crucial area of regulation. If "research based" is the floor, and we want some kind of higher standard for “scientifically-based research," what then? A static gold standard that suggests the state of the evaluation art can tell us more than it really does?

The nation is headed towards a compliance model that encourages providers to do the minimum level of evaluation required to satisfy an arbitrary standard of sufficiency in evaluation. We need something much simpler, yet more sophisticated - to encourage both innovation and efficacy - a regulatory regime focused on a provider's capacity, record and invesment in research and evaluation, and its use of that information to improve its products and services. The SES situation is making this harder to accomplish.

Third, related to the above, and far more important as a matter of public policy, students need individualized attention. Students learn differently and have different needs. The broader that range of differences turns out to be, the more difficult it will be to address the challenge in the normal school day, in the typical classroom. Today, the after-school SES program offers the best opportunity to get at the problem. In short, the potential of SES as a research and development activity (see here and here) is far more important (and justifiable) than the prototypical voucher-driven consumer market for “off the shelf” programs it has been advertised to be.

What we’ve got here is one god-awful mess, and hardly the kind of clarity and consistency school improvement industry needs to attract investment. Another reason it is time to strat thinking about an industrial policy for k-12 (listen here).