A checklist for figuring out whether your algorithm is a "weapon of math destruction"

[doctorow]

Originally published at: http://boingboing.net/2016/12/08/a-checklist-for-figuring-out-w.html

…

[lloydcogliandro]

@doctorow

Systems that are not available for examination cannot be trusted to operate without error. Either the software code must be available for inspection or test data must be available. Otherwise, there is no way to know whether the sins of science are present.

That would mean removing one of the most popular features of using “unbiased” algorithms!

[Liam1]

What’s the sign in the photo about?

[renke]

Annett showing the checklist for the composting

[Liam1]

Thanks. So is it supposed to be a “badly designed algorithmic decision-making system”? What’s wrong with a composting recipe?

[knappa]

I think it is just supposed to be a checklist.

[jhbadger]

1. You need to check that you have not trained on one population and then run the system on a different population, otherwise you may get enormous errors. Etc.

Easier said than done. The whole idea behind any statistical method is that you can make subsets that accurately predict the results on a larger or different group. While this sometimes fails (as in “Dewey defeats Truman” and in the recent US election), the reason isn’t that the researchers don’t get that the groups are going to behave differently-- it’s just the differences weren’t obvious until after the fact.

[RickMycroft]

Maybe they were right to classify cryptography as munitions for export purposes?

[nojaboja]

@jhbadger

the reason isn’t that the researchers don’t get that the groups are going to behave differently-- it’s just the differences weren’t obvious until after the fact.

Or to put it another way, the researchers happened to ask the wrong question.