The recently released Department of Education Study on technology driven math programs may have helped to advance the state of the evaluation art, but it confused any administrator or teacher considering any particular  tech-based math program.

Today's announcement by RAND of a $6 million grant from the Department of Education for a "gold standard" review of Cangegie Learning's Cognitive Tutor is a welcome antidote, but industry leaders should not see it as a chance to buy time or kick the political can of evaluation down the road. The study is welcome, but it should also be mildly worrisome.

It is welcome in that there should be adequate resources for a first-rate five year review of the algebra software program.  With similarly well-financed studies of New American Schools' "whole school designs" and Edison (both legacies of the late Tom Glennan), RAND has perhaps the most extensive experience in the review of large-scale k-12 program implementations offfered on a fee-for-service basis.  Because RAND will be comparing one math program (Cognitive Tutor) to doing nothing rather than the whole category, we will all get actionable information. Carnegie Learning epitomizes the research-driven firm every school improvement provider should strive to equal - it is as good a proxy of the "industry" as we could hope .

What is worrisome is the cost.

If RAND and others associated with the $6 million study have added bells and whistles to extend the state of the art, and believe that evaluation costs will be driven down over time, o.k.  And RAND is notorious for its overhead and other internal "taxes" not going directly to review the program in the classroom. But if this approximates the fees providers' face, the school improvement industry has a very serious problem on its hands. Carnegie may be getting a "free" review, but the Department of Education is not going to bear the costs of evaluation for the whole industry.

If this proves to be the standard cost of proving programmatic efficacy, the economics of k-12 programs begin to resemble those of pharmeceuticals.  The costs may be lower than drug tests, but the potential revenues are vastly smaller. There may be "blockbuster" drugs, but there are not likely to be blockbuster k-12 interventions.

A handful of school improvement providers could underwrite the RAND evaluation today. Not one firm could have afforded it when it was formed, except perhaps Edison and Education Alternatives.  Such a hurdle would have not have killed the industry at birth, it would have prevented its conception.

An FDA-like program approval process (before "Fast Track") spells the end of investment capital for most providers.  Should the government come to see this study as a template for the definition of Scientifically Based Research (SBR), the school improvement industry will only survive if it is absorbed by the publishing industry. But publishers want to slow innovation down rather than speed it up. And there are many opponents of high standards for all children who will use the conumdrum as another excuse for dropping the whole school improvement enterprise. It would be ironic if unreasonably high expectations of evaluation proved to be the status quo's salvation.

Industry leaders need to take an active role in the design of research and evaluation standards for the industry. Evaluations must provide objective help to decisionmakers, and to set some kind of a floor on what will be permitted in the market. But the regime should encourage innovators in a k-12 environment desperately short of innovation and  dominated by research-free programs that have shown no signs of efficacy.

New Education Economy® has examined a challenge that tends to be treated as if costs are irrelevant.

–––––––––––––––––––––––––––––––––

The U.S. Department of Education has awarded the RAND Corporation a $6 million grant to conduct a five-year study of the effectiveness of a technology-based mathematics curriculum created by Carnegie Learning, Inc., of Pittsburgh.
 
RAND researchers will examine the impact of Carnegie Learning’s Cognitive Tutor Algebra I curriculum. The curriculum supplements classroom instruction with a software program that adapts to individual students’ understanding of algebraic concepts to improve their problem-solving skills.... The grant to RAND is the largest available under the Department of Education grant program for scientific assessment of academic courses....
 
The company's Cognitive Tutor programs are currently used by more than 475,000 students in 1,300 school districts across the United States.... The math programs of Carnegie Learning are based on cognitive science research at Carnegie Mellon University in Pittsburgh, where researchers study how students think, learn and apply new knowledge in mathematics.... Carnegie Learning was selected for the study because its Algebra I program is one of a few math courses that meet the Department of Education’s grant requirement for strong prior evidence of effectiveness.
 
RAND researchers will examine how students using the Carnegie Learning curriculum fare on a standardized algebra assessment as compared with peers who receive traditional classroom instruction..... Classrooms in participating school districts will be randomly assigned to use either Carnegie Learning Algebra I or the school’s existing Algebra I course. The new study will show whether the Cognitive Tutor curriculum is effective for a wide range of students and environments.
 
“We are tremendously proud to be selected as the curriculum for this important study,” said Dennis Ciccone, chief executive officer of Carnegie Learning. “Our company was founded on the guiding principle that research-based programs are the best tools to equip our teachers and students for success in the classroom.... No Child Left Behind was established to measure effective teaching and learning, and this study is a model for the accountability that we believe will improve education in this country... We welcome an honest and objective analysis of the effectiveness of our curriculum as a means to better understand what we are doing well, and how we can serve our students better.”

Press Release, April 9.

––––––––––––––––––––––––––––––––

One point for the folks at Carnegie Learning. RAND is a "fair and balanced" organization. Neverthless, many choices are made in the course of study design and implementation that can affect study outcomes. And note that RAND's client is the Department of Education, not Carnegie Learning - and this can lead to awkward discussions around pivotal issues. Even a $6 million study has resource limitations and a team with particular stengths and weaknesses. These efffects only play on the margins, but this editor will predict that the study will not produce a "slam dunk" finding either way for reasons that have more to do with client selection than anything else, so the margins will count. Carnegie's management has research experience, but overreliance on it can be dangerous too. The firm would be very well-advised to make a serious effort to interview leaders in the dozen or so organizations that have gone through similar RAND evaluations for "lessons learned" about the subject's role in "high stakes" third party evaluations.

New Education Economy® Addressed RAND's review of Edison in October, 2005.

The outcome is one Carnegie wants to avoid.