American education's use of "value added measures" is statistically bankrupt

Famous, probably irrelevant, story about statistics error. Study of kidney disease found best kidney health in rural counties. Some conclusions were drawn and publicized and a well known statistician contacted the organization that did the study and asked them if they had looked to see where kidney health was worst. Oops, kidney health is worst in rural counties.

Turns out small counties have insufficient population to draw good statistical results and most of your high and low scores will be there because temporary spikes in the data will have a greater influence on averages in a smaller sample. Pushing the results out of shape.

This is a well known problem in statistics, but still an issue in the social sciences, where mathematical models are less common that “intuitive” ones.

2 Likes