In the words of Voltaire, "the best is the enemy of the good."
_________________________________

A buzzed-about U.S. Department of Education study released this month found that some popular software programs schools use to teach math and reading are pretty worthless….

The study, ordered by Congress back when it approved the federal No Child Left Behind Act, tracked 15 reading and math software programs as used in first-, fourth- and sixth-grade classes at 132 schools nationwide.

Phoebe Cottingham, the Department of Education's commissioner of evaluation, said the study used "gold standard" scientific methods to gauge the software's effectiveness in raising student academic performance…. [T]he inescapable conclusion, she added, is that despite all the time students spent sitting before computers, the data didn't show more than a blip of increased learning…."I don't think anyone expected it would come out so flat," she said....

To increase my understanding, I visited USC's Information Sciences Institute in Marina del Rey. Carole Beal, a former professor of child development and education at the University of Massachusetts in Amherst, has been working with brainy grad students to develop educational software that transcends the rote "drill-and-kill."

She is familiar with many of the programs in the software study and says she wasn't surprised that they didn't produce big results. The problem, she said, is that only one of those tested uses the sort of artificial intelligence technology that encourages high-level interactivity.

USC's programmers, for example, are developing questions to assess students' learning styles and eagerness to improve their grasp of material. They are tweaking the software to predict where the student's acquisition of information will lead, tossing up new challenges at a pace that the student will find motivating.

Bob Sipchen, Los Angeles Times, April 16.
__________________________________

No product or service is as good as it could be. The first books were too expensive. The first patent medicines were often poison. The first cars were slower, less efficient, and less reliable than a horse. The first double-hulled passenger liner became a disaster movie. The first commercial jets killed quite a few passengers. The first televisions and radios hardly had shows to carry - and they were pretty hard to make out. Only the truest of geeks could use of the first commercial internet. The first mobile phones were the size of a brick. And if no one had bought these first versions, that's pretty much where the state of the art would remain.

What's the point? Education technology can be improved. The methodology used to evaluate it can be improved. No doubt the programs recently reviewed by Department of Education contractors are far from perfect. But there is also little doubt that the methodology employed did not prove much beyond the fact that it was not a very good methodology for demonstrating programmatic efficacy. The article hints at as much when it notes that "only one of (the programs) tested uses the sort of artificial intelligence technology that encourages high-level interactivity" that Dr. Beal endorsed. Presuming that particular program works, its effects were lost in a study that threw all the programs together in a common experimental group.  (All that glitters is not gold.)

But you have to start somewhere, and the state of the art in every field only advances with use approaching scale. If schools don't purchase less than perfect programs, evaluators will never learn how to judge efficacy, and society will never get programs that are better. If there is no market for the less than perfect, we'll never get the perfect. Sure, we'd all like the other guy to buy the first new gadget, and wait for the price too fall, and the bugs to get worked out. But today, no matter what you buy, within six months there will be something better, faster, cheaper.

Schools that are not purchasing educational technology programs for students - not every program, but some, are simply delaying their inevitable charge up the learning curve. They should not go on a spending spree, but they should not be waiting until all the kinks are worked out and the price is right either, because the second coming is a long time off.