Regulating Software
The marketing and
procurement practices Todd Oppenheimer describes (“Selling
Software,” features, Spring 2007) were in place decades before NCLB. Even 10 years
ago, the largest school contracts were with multinational publishers for
textbooks. States either made approval a political process or delegated the
function to districts. The discretion that legislation granted to districts
for purchasing reflected beliefs that administrators would honor their
fiduciary obligations to students and that educators were the real experts
on curriculum. Student outcomes did not matter quite enough to districts
that providers had to demonstrate results.
Now that educational outcomes matter, procurement decisions are critical to school success. Unfortunately, the market is unprepared. Market leaders’ programs lack evidence of efficacy. Administrators lack the capacity to deal with the vast array of providers, products, and services now available. State and local agency heads have not kept up with the state of the art of education science. Evaluation methodology has not advanced far beyond considering whether an intervention will have some effect on some students. Above all, the federal government has neither defined nor enforced NCLB’s requirements that federal funds only be used to purchase programs proven effective through scientifically based research.
But there is good news. Several hundred firms and nonprofits grew up in the last decade because entrepreneurs and venture investors believed that the standards and accountability movement would apply to products and services as well as teachers and administrators. Research and evaluation is built into their offerings. But if they are to compete with entrenched multinational publishers, these emerging school improvement providers will need institutional capital. These funds will flow only when investors see movement to a regulatory environment that rewards program efficacy.
When educators demand better products the market will respond, but government must help. Federal policymakers need to establish a regulatory regime that sets a standard for quality, encourages innovative firms to compete against the historic market leaders, and gives administrators leeway to determine programmatic fit. The secretary of education has the authority to approve the clear, workable definition for scientifically based research that is needed. The real question raised by the Oppenheimer article is why she hasn’t already acted.
Marc Dean Millot
Editor
New Education Economy
Comments on Oppenheimer’s "Selling Software" (February 1)
< style="font-family: arial,helvetica,sans-serif;">Todd Oppenheimer does a fine job of explaining the current state of marketing, purchasing, regulation and research in educational software. Indeed, the same could be said for the entire range of products and services purporting to improve teaching and learning in our public schools:
• While the new accountability is pushing educators to focus on the needs of individual students, evaluation methodology has not advanced much beyond exploring the likelihood that an intervention will have some effect on members of a given race or socio-economic status.
• State and local education agency heads have neither kept up with the state of the art and nor given education science a meaningful seat at the table on important program procurement decisions.
• Whether they run nonprofits or for-profits, the CEOs and boards of most organizations working to improve schools have not acted as if program evaluation is terribly important
• Marketing budgets and personal relationships are a better indicator of program adoption than demonstrations of efficacy.
• The Secretary of Education has allowed the definition of No Child Left Behind’s (NCLB) “Scientifically Based Research” (SBR) provisions to become a source of confusion in the market rather than guidance.
This is bad news, but it is also old news to those who use, buy, sell, give away or observe the emerging market in school improvement services. The ugly news to a broader audience is that quality didn’t matter much until NCLB set the goal of 100 percent student proficiency in key subjects by 2014, held states to the path of Adequate Yearly Progress they set to achieve that goal, and made life very uncomfortable for people who work in local education agencies and schools that get behind plan. The good news is that the law is not only brings the problem into focus, but gives the Secretary of Education the power to make significant changes for the better.
To determine how government can get the market for school improvement created by NCLB headed in a better direction, we should understand a bit more about where we are and how we got here.
Most operating agencies in government have developed extensive long-range planning, programming and budget activities to assure the highest and best use of taxpayers’ money. Likewise, legal restraints are in place to avoid corruption and the appearance of favoritism in contracting. The marketing and procurement practices Oppenheimer describes are the legacy of the rather different conventions public education agencies accumulated around teaching and learning over several decades before NCLB.
Even ten years ago, the primary contracts related to this core function concerned textbooks. States either made approval a political process or delegated the function to districts. But in both cases, districts were granted enormous discretion in decisions about other products and services related to the classroom. The underlying legislation reflected a belief that public education differed from, say, public works in that administrators would be driven by their fiduciary obligations to students (this was the era of in loco parentis), educators were the real experts on matters of curriculum and pedagogy, and experts could be expected to keep up with the state of their art.
Decades passed. In the absence of any meaningful personal or institutional accountability for student failure, factors other than a demonstrated contribution to student outcomes came to dominate the procurement process in what have been essentially monopoly providers of public education services. Multinational publishing corporations gradually assumed control of the huge and profitable textbook market. The relationship became very cozy.
Now that outcomes do matter, inputs like program performance have become visible and important. Rather suddenly, once ancillary procurement decisions are recognized as essential ingredients of school success. Moreover, these purchases have been diverting funds from the previously comfortable growth of textbook sales.
Traditional institutions of public education are not all that ready for the change. Organizations that benefited under the old regime are happy to slow things down, wait, and see if the new approach sticks. School districts that relied on a handful of companies lack the capacity to deal with the vast array of providers, products and services now becoming available, and many in positions of authority lack an interest in the new approach. Those who dominate today’s k-12 market generally haven’t developed the right products for the emerging market segment.
In the midst of this, the federal government has failed to bring order and direction to the forces unleashed by NCLB’s legal requirements that schools and districts 1) meet AYP and 2) use federal funds to purchase only products and programs that embody scientifically-based research. The Department of Education and the relevant committees of Congress are still coping with the challenge of regulating school performance with Adequate Yearly Progress. Moreover, since its inception, the Department has practiced the “regulation” of public education as a series of informal and secret negotiations with state agencies where exceptions are the rule. And the Department has given education science and the k-12 industry no meaningful role in any Secretary’s standard decision-making process. Today, the institution is utterly lacking in the expertise and experience required to regulate k-12 markets under a law that has deliberately made private enterprise a key stakeholder.
Several hundred new firms and nonprofits grew up in the last decade because entrepreneurs believed that the standards and accountability movement would apply to products and services as well as teachers and administrators. Rather than serving as a post hoc justification for sales, evaluation is built into the development of their offerings. They found venture capitalists and philanthropists who shared their vision. What they lack is the capital required to grow. These levels are well beyond what venture investors are organized to provide. Funding must come from the far more risk-averse managers of institutional capital.
To provide this capital, institutional investors must conclude that the innovative firms won’t be crushed in a market held by some of the world’s largest companies. The key to unlocking the bank is regulation that translates the new firms’ superior offerings into a real market advantage. Capital will not flow to emerging school improvement providers until institutional investors see movement to a stable, consistent - and reasonably demanding - regulatory environment around standards of program performance. A weak regime offers the new firms no defensible beachhead on the turf held by today’s market leaders.
Institutional investors’ initial impression of government interest in SBR regulation can only be a delegation of policy to a group of academics with abstruse perspectives, who can’t come to a conclusion or compromise, and who exhibit no empathy for the practical problems faced by firms or capital considering investment in an emerging market. Understanding the mix of institutional politics, cultural division, pedagogical ideology, the limitations of the state of the evaluation art, and reasonable differences among experts is just too complicated for investors who are constantly approached to provide capital and simply look for a decent risk-adjusted return.
It must also be ironic to investors that a Bush Administration so committed to the dim prospect of a market in public education fueled by vouchers has proved quite uninterested in an existing market to improve public schools. As with SBR, the Administration and Secretary seem to have delegated market regulation – this time to bureaucrats, ideologues and university-based program developers with a variety of personal interests in a particular approach. The result, exemplified by several Department of Education Inspector General reports on the implementation of Reading First, can only leave investors with the impression of arbitrary and capricious regulation typically associated with the new market economies of the Soviet Empire. Rather than spending the time trying to figure out the game, rules, terms and players - and then making bets on winners - institutional capital is going with investments they understand. The net effect is to leave the historical market leaders without must in the way of challengers to their comfortable positions of dominance, and little reason to pick up the pace on innovation.
If administrators demand better products, the market will respond. If Reid Lyon had been an administrator rather than a research scientist, after picking apart a slew of marketing reps, he would have bought something he considered better than what his teachers were using at the time. He would be informed by evaluation science, but he would not demand perfection given the pressures to do something about his failing students. If policymakers want to avoid the choice of no regulation of quality or over-regulation the Oppenheimer article implies, government needs to help k-12 administrators.
In part, this means continuing to hold their feet to the fire on AYP. But it also means that policymakers - generalist, political officials - need to step in to the clean up a technocratic mess. They need to establish a regulatory regime that sets a floor on quality, encourages innovation and innovative firms to compete against the historic market leaders, leaves administrators with constrained yet ample room to determine programmatic fit, and a mechanism for advancing agreement on the state of the art rather than incentivizing chaos.
In k-12 education we face a challenge no less complex, politicized and lacking in science as the debate over the environment 40 years ago. The reason that today’s arguments on clean air, forest management, the use of federal lands, and even global warming now tend to result in improvements is the decision of all parties to direct their energies and arguments through regulatory systems based on adaptive management. This is really just a fancy term for formalizing the historical progress of technology applied to human problems by placing scientific inquiry in a context where the need to trade off multiple societal interests is recognized. An adaptive management approach would have the Secretary of Education make those tradeoffs based on the best available evidence and input from all stakeholders. These decisions, their rationale, and the evidence would be recorded in writing. The record would be subject to judicial review against a “rational basis” standard to protect k-12 program buyers and sellers from arbitrary decisions – as it is with every other market under federal regulation.
Something like this method offers us a way out of the mess school improvement has become. The nation needs an approach to SBR regulation where the participation of innovative firms and their investors are encouraged, perfection doesn’t drive out improvement, standards are based on a consensus of the best available evidence, open questions of science are identified and deliberately subjected to additional research, the benefit of a rise in standards is measured against its costs, all the stakeholders have a voice, and what are in the end political decisions are made by policymakers rather than activists or bureaucrats.
It is completely within the Secretary of Education’s authority to implement NCLB in this way. The real question raised by Oppenheimer is why she hasn’t pursued it.