Value is results at a price.

Mark Strickland, Schoolhouse Partners
_______________________________

In a May 6 posting, your editor asked how much we ought to be prepared to pay for increases in student performance on the state tests that judge student proficiency. The Chicago Public School system's evaluation of SES providers during the 2005-6 school year begins to get at the answer.  (Download below.)

As with the Tennessee study discussed here some weeks ago, the overall results are disappointing. Scale scores in the Illinois Standardized Achievement Test (ISAT) range from a possible low of 120 to a potential high of 411. That means a student can fall somewhere on a spectrum that equals some 300 points. The CPS SES study’s findings cover a universe of potential effects on students that encompass a range of about eight points on that spectrum.

Figure 5, on reading scores, illustrates the impact. It "breaks down the changes in achievement of the three groups (students eligible for SES who did not participate, students who did participate, and students who were not eligible) by grade level... with the achievement of SES participants set as the baseline: This figure indicates that SES students in 3rd grade scored 1.4 adjusted reading scale score points higher than did eligible students that did not participate. The educational significance of this is debatable, at best.



Taking a magnifying glass to the results, Figure 9 reveals that about one third of the providers actually managed to produce declines in students' reading performance. Compared to a kid who could take SES but does not, the best SES providers add a bit more than four points to the ISAT reading score, the worst take a bit less than four points away. In percentage terms, we are talking about moving a kid in the vicinity of one percent forwards or backwards along the ISAT's 300 point spectrum.

In math, the difference between taking the tutoring or not is both statistically and educationally insignificant. In terms of the ISAT, students aren't affected by tutoring at all.




According to CPS (f)orty-two programs tutored students during the 2005-2006 academic year…. Table 3 summarizes the number of students offered tutoring, the number data were available for, the number of hours each provider tutored students, the grade levels providers were approved to serve, the costs to provide services to each student, and the cost per hour of tutoring. The A.I.M. High program offered by CPS was the most widely provided program, serving over 16,000 (40%) students.

The SES offerings were not homogenous. Table 3 shows a considerable variation in price and service. Some providers did all their work face to face, some were online, some mixed the two. The hours of tutoring offered range from 30 to 120, the price charged to the taxpayer from $360 to over $1800. The least expensive program, Babbage online, offered 80 hours of service. Four providers held the high price of $1866.94. Online provider Catapult offered 30 hours; SL@Home, 50. Progressive offered an online-onsite mix of 45 hours. Off-site provider Knowledgepoint offered 60 hours face-to-face.  (Because CPS sells A.I.M High to itself, the transaction is not arms length. Therefore, its "transfer" price of $375 can't be relied upon as a reflection of the true costs of its 80 hours of onsite services.)



Figure 12 suggests that higher prices, mode of delivery (online or face-to-face) and total hours of service are not great predictors of program performance. 



To your editor, this suggests that tutoring under SES is closer to an R&D activity than something "off the shelf" as implied by the way NCLB structured the program as a consumer market.

Indeed, the CPS study found that [a]fter adjusting for differences due to 2004-2005 ITBS scores, race, gender, grade level, and disability status, SES participants demonstrated a small but significant improvement in reading achievement compared to students eligible that did not receive SES. Translated to ISAT reading scale score points, eligible students scored an average of 0.8 adjusted scale score points lower than SES participants, which translates into a small effect considering the average student scored +/-15.60 adjusted scale score points different from 0. Prior achievement was a much better predictor, uniquely accounting for 8 scale score points in the model, while group membership uniquely accounted for only 1.6 scale score points….

The bottom line here is fairly clear.  Overall and Individually, the impact of SES providers on student performance is statistically significant but educationally minimal.

So What?

For $1600 per student and 80 hours of on-site tutoring, the best overall program - Unparalled Solutions - is delivering 4 adjusted points on the ISAT reading test - a bit more than a one percent improvement over the SES-eligible student who choses not to take advantage of SES. The least expensive program showing positive results other than the district's own A.I.M. High, SCORE!, is delivering two points - a bit under one percent, for $1100 and 96 hours on-site.

The Chicago Daily News says this has cost us some $50 million to date. The basic policy question is whether moving average individual reading scores one percent at a price of $1100-1600 per pupil is a good deal, or at least a promising opportunity worthy of staying the course. Your editor doesn't know if that's the right thing, but is skeptical.

SES is for students in schools that are not making AYP, presumably to help them achieve proficiency in basic subjects as measured by state tests. It seems unlikely that moving a few points up the scale is the difference between pass and fail  for many. And paying an average of $1400 per student for a few points seems very high, when the Bureau of the Census most recent report tells us that the total average per pupil expenditure in Illinois is around $9,000, and ranges from $5,000 in Utah to $14,000 in New York. (Download below.)

 To get some sense of SES providers' value - "results at a price," consider the reading program offered by the Success for All (SFA) Foundation - a program with impressive results. The arguments about SFA's outcomes are not about educational or statistical significance - no one disputes the programs' ability to improve student performance dramatically, they are about persistence.

According to the 2005 CSRQ Center Report on Elementary School Comprehensive School Reform Models, SFA's total average annual cost over three years is about $367 per pupil.

The general explanation of SES providers' "less than hoped for" performance comes in two parts: not enough time with the kids, and not enough of a connection to what's happening in class. SES trade group representive and Education Industry Association President Steve Pines makes the first point. SES and program evaluator Steve Ross makes the second. Both make their points fairly often.

Both are structural problems that SES providers can't solve. But it's not the kind of defense your editor would employ on behalf of SES providers. It's actually a good case for ending the program. Something like 80 hours is the ceiling for most private sector providers (and most SES providers in EIA.)  And there are simply too many providers for school teachers to accomodate the tight classroom connection each program seems to need (and the most likely to connect are local providers with personal relationships throughout in the teaching force - again probably not most of EIA's SES members.) So these explanations become just another way of saying that the SES provisions in NCLB can't  produce the outcomes Congress expected. Which essentially hands the opponents of SES the bludgeon with which to kill the providers.

Steve Pines' May 18 letter to the editor of the Chicago Sun Times, on behalf of SES providers, an effort to avoid responsibility for improving student test scores, is an absolute nonstarter in the era of accountability ushered in by NCLB.

[W]hile SES, by itself, is likely to have small influences on state standardized test scores, the tutoring program’s impact should also be measured in terms of parent satisfaction, principal and teacher opinions, and compliance issues related to program implementation.... It would be a pity if the terrific job CPS and our members have accomplished in delivering high-quality tutoring services to so many of Chicago’s deserving students were reduced to a simplistic “pass/fail” score on a single standardized exam score, which is too blunt of an instrument to truly detect student progress from tutoring.  Indeed, we urge the Sun-Times to speak to the tens of thousands of Chicago parents who see federally funded tutoring as helpful....

To any SES provider with any history in school reform, and especially urban school reform, and especially any EIA member trying to establish a reputation for performance, this statement has to be an embarassment.

At least as important, Pines may not realize that his words constitute the argument made by most traditional k-12 interest groups for weakening NCLB's determination of AYP. It's doubtful that EIA would formally join them in dismantaling the provision that actually gives school improvement providers - including SES providers - their market.  But Pines' argument is offering a de facto alliance leading to preciesly that outcome, under the general heading of "what's good for the goose is good for the gander." This partnership can't possibly be in the interest of EIA's non-SES members, even if its SES members have convinced themselves that this is their best possible strategy for staying in business.

Were the several years of SES activities in the field a competiton sponsored by a private firm, there is every reason to believe management would pull the plug, or at least restructure the program radically to focus on the best of the bunch. But this isn't an R&D program, it's an open market  where providers can charge a wide range of prices and offer what they want in the way of services, subject only to parental choice. 

If this were private tutoring, where parents pay providers from their own earnings, that would be fine. But this is a market where parents are spending federal taxpayers' money, and we have a right to demand that parental preferences be constrained by some demonstration of program efficacy.

Your editor thinks we ought to start talking about changing the SES program before others build much momentum for killing it.

• Because the results to date show public school tutoring under  SES is an R&D activity more than a consumer good, let's make SES something more like the Comprehensive School Reform Demonstration Program where districts and one or more providers would apply for grants to work in schools and get these programs through the product development stage they are in today.

• And because we can't get more hours out of today's SES strategy, or tight classroom connections in its current consumer-market approach, let's think about encouraging curriculum providers like SFA to offer tutoring programs that reinforce their proven programs in schools targeted for improvement.

Without giving up private sector engagement or after-hours tutoring as a way of helping students in failing schools, let's stop treating SES as an "extracurricular activity." Let's start integrating SES time to leverage classroom programs that have some kind of a record of effectiveness.