The attached March 20 email from Software and Information Industry Education Division VP Karen Billings is the first evidence of any k-12 trade group ("old industry" or "new industry") adressing federal evaluation of k-12 programs as more than a communications challenge.  The substance of its premptive comments on the Department of Education's forthcoming study of the value added by technology to student achievement is less important than the decision to comment.

What it suggests:

First, that the "technology provider" wing of the school improvement industry understands that the details of federal evaluation policies, standards and methods has a material impact on every members' revenues. It is a problem that neededs to ne adressed industry-wide rather than by individual firms. So far this is a point lost on Education Management Organizations, charter schools operators as a whole, as well as Supplementary Educational Service and Comprehensive School Improvement providers.

Second, that it will have to address these issues on the technical and regulatory levels, instead of rhetoric, anectodal evidence or their own counter-reports. The last are important, but must stand up to methological scrutiny as well.

Third, that it must get out ahead of these issues, rather than wait for reporters to call asking why their members programs don't work.  Trade groups should not only be a shield for their industry; to be effective they must be able to wield the sword.

Fourth, that it has grasped the methodological challenge of program evaluation - district and teacher support for the "intervention." This places the burden of client selection on the provider and will lead to its own conflicts between any providers sales reps and those in the firm responsible for efficacy, and that's a conflict providers have failed to intyernalize to date. But it also points to a need to think about program evaluation as a process rather than an event, and to focus regulation on indicators of a firm's commitment to quality - like making the data necessary for evaluation availble to qualified researchers and the designation of a corporate officer responsible for certifying that data - rather than any single study.

These steps are all vital to the development of a strong trade group, and they put SIIA ahead of its competitors for school improvement industry firms.

–––––––––––––––––––––––––––

Dean,

As one who understands the opportunities and challenges of technology in K-12 education, we write to share our views on an issue of growing concern -- that the implementation of educational technologies and software is too often taking place without the proper conditions and supports needed for success.
 
Admittedly, this is nothing new, but we see it coming to a head as evaluation studies, especially that employing random control trials, may not sufficiently account for variations in these underlying conditions and practices.  As you may know, the U.S. Department of Education evidently plans to release its study on the effectiveness of technology in bolstering student achievement sometime in the next month.  Originally scheduled for release nearly a year ago, the study is rumored to indicate that technology's impact on achievement is minimal or non-existent.  If true, this outcome would be surprising to us and our members as we have studied and found results to the contrary over the last few decades, many of which we have published in several volumes titled "Effectiveness of Technology in Schools."
 
Based on confidential discourse, we understand that the implementations of technology in many of the sample schools and classrooms were not executed properly.  SIIA research, and that of many others, illustrates that the fidelity of software implementation is crucial to the success of any program. In other words, the process of implementation can be as significant to the results of student success as the technology program itself.
 
As you know, these conditions include adequate technology infrastructure and access, teacher training in both the technology as well as in revising curriculum/instruction to  leverage the technology, school leadership and technical support, time on task, and a matching of the technology to the educational goals.

In one pending study, SIIA learned of the following implementation concerns:
* Poor quality of teachers, i.e. not engaged or motivated, not experienced, cynical
* Students did not spend enough time on task
* It was hard to get started in some schools, i.e. technology issues (bandwidth, connectivity, dated hardware)
* Whole class design is not ideal if intervention solution is for at-risk students, not for everyone
* Unresponsive administrators; little or no buy-in from many administrators, principals, and teachers. They were told what to do. So there is immediate resistance.  spent a lot of Year One getting buy-in.
* push back from some schools in terms of training.  even refused access to one building; they would not give us 'teacher release time'....

Where conditions are in place and there is appropriate match between the software and the needs/goals, success is taking place, and this success comes through in carefully crafted research studies.
But bringing software into a school or classroom without proper implementation and then studying the impact, especially with a blunt research design, will most likely create a self-fulfilling prophecy of inadequate impact.

We hope this information and perspective will prove useful to you as you continue to examine these issues for your readers.  Please let us know if we can provide any further information as you research and write about these and related issues.

Thanks,

Karen Billings
VP, Education Division
SIIA
 
Mark Schneiderman
Director, Education Policy
SIIA