Lies, Damn Lies, and Statistics

State School Report Cards have been released.
When Khrushchev was voted out of power 7-4 by the Presidium in the Soviet Union in 1957, he dismissed the vote saying,"Politics are not arithmetic. They are something different." The Central Committee promptly overturned the Presidium vote. The arithmetic of "School Accountability Reports" in Colorado is a model lesson in politics.

A four-year-old child could understand this report. Run out and find me a four-year-old child. I can't make head or tail out of it.
Groucho Marx, Duck Soup

First, the numbers presented in the Colorado "School Accountability" reports are numbers! Only numbers. What I mean to say is that they are not accompanied by reliability, error range, validity, or much else that give numbers meaning. So, a 3.2% increase (or decrease) can be minimal, encouraging, significant, or fantastic, depending on your perspective.

Adding a decimal point increases the truth of any statistic. 82.34% of people say so.
Anonymous

A total of 76 percent of 47 schools were rated "low" or "unsatisfactory." Last year, 28 of the district's 46 schools - or 61 percent - were rated "low." None was unsatisfactory. "We have schools that are missing average by literally hundredths of points. . . . If we keep our forward momentum going, we will have a large number of schools moving into the average category next year," he said. [Wait a minute. Let me get this straight. Going from 61 percent rated low last year to 76 percent rated low or unsatisfactory this year is forward momentum?]

Statistics: The only science that enables different experts using the same figures to draw different conclusions.
Evan Esar (1899 - 1995)

Even the purpose of measuring only reading, writing, and math is unclear. Education is much broader and deeper than any simple snapshot of basic skills. For example, most of my students score proficient on the CSAP reading exams, but few of them are "readers." That is, they do not habitually read (or even occasionally for many). The tests tell only very simplified stories and what those stories are aren't exactly clear. Just because a student scores in the partially proficient range doesn't tell much about exactly what skills that student has and what they need to improve upon.

An undefined problem has an infinite number of solutions.
Robert A. Humphrey

Even the scores themselves and the categories they spawn are often spurious. For example, I have a student this year, George, who has scored Unsatisfactory across the board for the last 2 years. It saved me a great deal of time by finding out that the student simply didn't want to take the exams and gave random answers.

Facts are stubborn things, but statistics are more pliable.
Mark Twain (1835 - 1910)

A school this year in the district, very similarly was rated as "Unsatisfactory" on the exams, but not because of the test scores, but rather because about 20% of parents at the school boycotted the exam to protest the end of a popular program. So what does "Unsatisfactory" tell us about that school and its teaching? Almost nothing. But even if those scores hadn't been included, so many variables are unaccounted for that the purpose of a "report" card becomes blatantly political.

The first lesson that you must learn is, when I call for statistics about the rate of infant mortality, what I want is proof that fewer babies died when I was Prime Minister than when anyone else was Prime Minister. That is a political statistic.
Sir Winston Churchill (1874 - 1965)

Partially Proficient on the CSAP tests mean just that in Colorado. Not proficient, but partially so. However, the State has decided that for Federal statistics required for the No Child Left Behind Act, Partially Proficient is considered Proficient. Where you stand, I guess, depends on where you sit. On balance the idea of simply improving Reading or any particular test scores is no great vision. It is mediocrity itself.

I could prove God statistically.
George Gallup (1901 - 1984)

Finally, just downright sloppy error is hidden so well in numbers and statistics, that we must be extremely vigilant. As an example, from the Rocky Mountain News, note that the "Note" is completely wrong. The ACT figures are not reported in percentages. ACT doesn't report figures in percentages. Let us please start getting out of the numbers game, and begin a real dialogue about what education means, and what it should look like. A vision, not bean counting.


Numbers are the refuge of the insecure. The more incompetent, the more precise and certain the numbers.