My school's better than your school

An article in this morning's NY Times caught my attention. It relates a large-scale study of math achievement among children in public and private schools. According to the Times, the study concludes:

...that when it comes to math, students in regular public schools do as well as or significantly better than comparable students in private schools.

This conclusion was reached after adjusting the students' raw scores for differences in their backgrounds:

Though private school students have long scored higher on the national assessment, commonly referred to as "the nation's report card," the new study used advanced statistical techniques to adjust for the effects of income, school and home circumstances. The researchers said they compared math scores, not reading ones, because math was considered a clearer measure of a school's overall effectiveness.

The study found that while the raw scores of fourth graders in Roman Catholic schools, for example, were 14.3 points higher than those in public schools, when adjustments were made for student backgrounds, those in Catholic schools scored 3.4 points lower than those in public schools. A spokeswoman for the National Catholic Education Association did not respond to requests for comment.

I'm finding it very difficult to come to grips with what this really means, if it is true at all. I know that it does NOT mean that even though little Johnnie can't add two and two all is well. His public school is doing just fine! It's just the fact that little Johnnie is poor and black that holds him back.

Perhaps it means that little Johnnie shouldn't be moved to a private school when he can't add, but I'm not at all sure of that either. In fact, without looking into the methodology used to make these adjustments for background, I can't conclude much of anything, but I'm very skeptical. Take this paragraph, for instance:

The report found that among the private schools, Lutheran schools did better than other private schools. Nevertheless, at the fourth-grade level, a 10.7 point lead in math scores evaporated into a 4.2 point lag behind public schools. At the eighth-grade level, a 21 point lead, roughly the equivalent of two grade levels, disappeared after adjusting for differences in student backgrounds.

Here we have the average kid in a Lutheran school two years ahead of his counterpart in a public school. What does the study mean? If that same child were poor and black instead of rich and white, he would be more than two years behind his rich, white classmate in the Lutheran school, and worse off than his counterpart in the public school? Somehow, I very much doubt that.

The study is being used to argue against school vouchers and charter schooling, a thrust I'm inclined to agree with (though for different reasons). But, I suspect we really need to examine the methodology behind this study before we begin to use it to argue about policies.

One side issue in the study, which I find it less difficult to question, was this finding:

The current study found that self-described conservative Christian schools, the fastest-growing sector of private schools, fared poorest, with their students falling as much as one year behind their counterparts in public schools, once socioeconomic factors like income, ethnicity and access to books and computers at home were considered.