School improvement program providers need to stop and think about whether the direction evaluation is taking is setting them up for mediocrity - or even asking the right question.

Two reasons why a range of programs have been developed to improve student learning are the variety of theories about how people, especially children, learn and the range of approaches one can take to communicate information. The new array is a welcome departure from the one-size-fits-all combination of a district-wide textbook and teacher training from former district employees. Moreover, they are consistent with a growing body of scientific evidence that different people learn in different ways, and school reform research that shows teachers get the most teaching benefit from programs they choose.

Ironically, when it comes to evaluation of the new programs, school improvement providers have accepted the idea that they should be graded on the basis of the one-size-fits-all model. In the forthcoming RAND study of Carnegie Learning's Cognitive Tutor "students will be randomly assigned to a course using Cognitive Tutor and one using traditional methods." This implies that the program will also be randomly assigned to teachers. 

Frankly, the approach suggests a renewed search for the Holy Grail - in this case, the "student-proof," "teacher-proof" school improvement program.  It also suggests that there is "one best" theory of learning, one best means of communicating information, and one best approach to teaching. Indeed the reporter says as much: "The new study will show whether the Cognitive Tutor curriculum is effective for a wide range of students and environments." But wasn't the "one best way" of public education discredited some time ago? Is there one best way in any other sphere of human activity?

––––––––––––––––––––––––––––––––––

The RAND Corp. will study the effectiveness of Carnegie Learning's Cognitive Tutor algebra curriculum with a $6 million grant from the U.S. Department of Education.... Cognitive Tutor, which is based in Pittsburgh where RAND has an office, uses a software program that adapts to each student's understanding of the subject.

John Pane, a research scientist at RAND who will lead the study, said the curriculum will be studied at more than 130 schools in six states: Texas, Tennessee, Kentucky, Michigan, Connecticut and New Jersey....

He said students will be randomly assigned to a course using Cognitive Tutor and one using traditional methods.

The math programs of Carnegie Learning are based on cognitive science research done at Carnegie Mellon University, where researchers study how students think, learn and apply new knowledge in mathematics....

Mr. Pane said RAND researchers will use rigorous experimental methods to examine what he called a "very promising technology-based math curriculum" and how well it works in a diverse set of public schools around the country.
Classrooms in participating school districts will be randomly assigned to use either Carnegie Learning Algebra I or the school's existing Algebra I course. The new study will show whether the Cognitive Tutor curriculum is effective for a wide range of students and environments.

Eleanor Chute, Pittsburgh Post-Gazzette, April 10.

__________________________________

A decade of program evaluation - most prominently RAND's - surely demonstrates that if teachers don't use the program, their classroom students don't get the results. RAND should at least be able to assign Cognitive Tutor to teachers who actually want to use the program. It's not a fair or useful test if results from classrooms with teachers who don't want the program and don't use it as designed are treated the same as results from the students of teachers who buy into the program and implement it with fidelity.

Frankly, if we randomly assign the program to teachers, we are not including Carnegie's "client selection" in the evaluation, and surely that is an important part of the overall "program." It is highly likely that their early client selection process overvalued sales and lacked good screening mechanism to filter out clients who would not really use the program. Chances are the process has improved over time, precisely because management is aware of how "bad" clients pull down the model's overall performance in today's evaluation regime.

Regardless, understanding the problem of client selection, evaluators ought not to be setting out on a course they know will tend to prove a program won't work. We ought to be taking its overall design at face value and examining the probability that it will work. See this week's New Education Economy for Jason Casarino's guest column on this idea.

High stakes evaluations must be viewed by management and investors with at least the same level of  concern and scruting as any financing deal. And the stakes don't get any higher than a RAND evaluation. It is probably the case that RAND's overall evaluation of Edison Schools in 2005 had a negative impact on the firm's school contracting business. Judging from Edison's funding of the Peterson study and his efforts to discredit RAND's recent review of contract schools in Philadelphia, at a time when the firm's contracts are up for renewal, it is reasonable to surmise that management fears an impact on revenues. 

The fiduciary duties of management are not widely recognized as extending to evaluation, but if your editor were an investor in Carnegie he would surely expect management to be on top of these matters. The firm is very likely to see a material impact on revenues if this study turns out badly. Decisions on study design made today will be almost impossible to change. We can only assume that program performance will become more important to sales and that performance expectations will rise over the study's six year life. Management will look worse to investors if less than stellar findings are the result of a clear shortcoming in methodology that was known at the outset. 

If your editor were a member of Carnegie's Board of directors, he would expect a briefing on the decisions RAND has made to date, the impact of those decisions and the alternatives that might have been chosen, what might be changed, and regular updates thereafter. Evaluation is an art, but it is not a black art and it is not beyond the comprehension of anyone who al;ready has some facity with numbers. And making it comphrensible to the board is management's job.

There's a bigger question here as well, and that is when the industry is going to understand the importance of evaluation to its future, the inability of single firms to shape the evaluation system, and the need to put up the resources required for the trade groups to build the capacity required to engage on matters of real technical substance.

Your editor believes that research will eventually support the conclusion that different programs work best for different students and teachers. When that happens, we will get back to the ideas of comprehensive school improvement, and the value of autononous, right sized schools where students and teachers can choose to join like-minded communities of practice, rather than schiools that are sized, organized and provided for the benefit of central direction and the average student. Until then, we are doomed to study after study showing no great difference between today's one-size-fits all combination of textbooks and cronies in school districts and the new programs. 

"Eventually" can be a very long time. It is not obvious how many programs and firms will survive this transition, if it is allowed to occur by default rather than with the industry's assistance.