Investors and k-12 educators prepared to bet against the old education industry of multinational publishers with great marketing but untested programs should look for three things in a school improvement provider, and Scientific Learning has them all:

• Members of the management team with professional credentials in education research, development and evaluation, and experience in program development in a business setting. There is a vast difference in value between a modest firm that has licensed a set of educational programs (most of which are still far from off-the-shelf) from a developer, and one that has the capacity to develop and modify programs based on real world experience.

Two members of Scientific Learning’s nine-member board, Michael Merzenich, Ph.D. and Paula Tallal, Ph.D., developed the firm’s Fast ForWord reading program through their academic research in neuroscience. A third member, Dr. Joseph Boyd Martin, Dean of the Harvard Faculty of Medicine, established the W.M. Keck Foundation Center for Integrative Neurosciences at UCSF. The firm has a Ph.D in the role Director of Research who coordinates evaluation work, a position that might sound obvious, but often does not exist.

• Regular releases of program implementation and results data for review by professional peers. The best way to know if you can believe a provider's claims is if they’ve given the research over to peers who are only too happy to help them find flaws. Above all, research that is subject to peer review indicates that the provider intends to use it for program improvement rather than simply marketing - because faults will be found.  Scientific Learning has successfully submitted Fast ForWord's underlying research to such peer-reviewed publications as Science and the National Academy of Science Proceedings. (Below, download the firm's response to your editors request for information on their research.)

• Constant evaluation of program implementation and outcomes from a variety of sources, and a process for feeding back that information for program improvement.

Firms with one study that has passed muster at the Department of Education’s What Works Clearinghouse (WWC) wave their "Intervention Report" like a victory flag at the Indy 500. In fact, it’s more like qualifying for the race. WWC review criteria are not very demanding; an experimental design covering a program version implemented by the original development team years ago in a small number of schools that passes a dissertation committee essentially certifies any provider who licenses program version 4.0 using entirely different staff operating in thousands of schools.

The fact that few firms have any WWC review says more about the average provider than the house rules. By the same token, the fact that few firms with a WWC review pass without straight A's, says more about the emerging state of the intervention and evaluation arts than it does about the quality of that handful of providers. The best indicator that evaluation implies a corporate value to improve their programs over time, rather than an act of regulatory compliance or marketing expectations, is constant program research.

Scientific Learning’s Fast ForWord program "passed" WWC review as a program for English Language Learners in elementary grades. WWC researchers reviewed six relevant studies. Two met their expectations for evaluation methods. One of those studies was conducted by Scientific Learning, the other was independent. The findings were no “slam dunk.” Frankly, few programs reviewd by the Clearinghouse are. In this case, the bottom line was "potentially positive effects on English language development and no discernible effects on the reading achievement of elementary school English language learners.”

To your editor, the interpretation that, at best, the WWC findings suggest Fast ForWord shows promise in work with English Language Learners, is far far less important than the fact that the firm has had hundreds of research activities going since the firm was formed in 1996. This broad program of research, encompassing work by its own staff, client school districts, consulting academics, independent researchers and doctoral candidates' successful dissertations is an investment that will lead to program improvement over time. Providers that don't truly care about research don't spend so much time and effort on it.

If this discussion is disappointing to some, consider that most of the major publishers have done no program research on their products and services, and that many of the school improvement industry’s new entrants' evaluation efforts are nowhere nearly as extensive as those of Scientific Learning. It is no mistake that your editor chose to discuss this firm after Success for All, Carnegie Learning and Voyager Expanded Learning. It is part of a small club, in a k-12 education environment that will only come to value outcomes more in the coming years.

Educators with a greater interest in the role of evaluation in purchasing decision should listen to our podcast series, starting here. Some information related to your editor's firm is out of date, but the substance remains on point.

Investors who want to pursue the importance of evalation to the sustainable competitive advantage of school improvement providers should listen the series starting here.

(Note: Your editor has followed Scientific Learning since 1996, but has no invesment interest. Scientific Learning was a client/subscriber to an early web-based information service of your editor's own firm with a fee of around $1000 per year, but is no longer.)