Originally published at: http://boingboing.net/2016/09/15/rules-for-trusting-black-box.html
…
1 Like
Boxes are great, but “black boxes” condition people to hide complexity rather than strive to master it.
Why I trust algorithms is that they lack a self-serving bias. So my trust of them requires that they be designed to reject the self-serving biases of both their creators and users. They function well, but not to the benefit of anybody in particular.
5 Likes
This is conflating security through obscurity with preventing ‘gaming’ an algorithm (although the line is blurry).
Going back to the actual security example, Security Through Obscurity would be a router manufacturer relying on a proprietary, and secret, crypto algorithm. This is bad for all the reasons that the OP brings up.
However, while it is best if a router uses only publicly vetted crypto algorithms, there is no advantage for them to disclose which such algorithm they are using, as that is giving an attacker additional information for free.
Furthermore, it is actually a good idea to obscure which algorithms are being used, if it is possible to do so without compromising the strength of the implementation (for example, adding some delay jitter to response times). This is also not security through obscurity, it is defense in depth.
There are many scenarios where this sort of thing makes sense, and many places where misaligned incentives create the unlikeliest attackers (eg. teachers, or even whole school districts, ‘teaching to the test’).
With email spam detection, we indeed have the proper set of circumstances to treat the algorithms as if they were crypto, so Bayesian filters and other criteria can be (relatively) openly disclosed, even though we have no idea exactly which algorithms are being used by GMail for this purpose.
With content recommendation systems, while we should in theory have assurances that the black boxes are only using algorithms that don’t insert biases, there is absolutely no reason for disclosing which known-good recommendation algorithms are being used. And that is even sidestepping the issue of recommender systems that are too good, leading to filter-bubbles and epistemic closure.
Further, given the increased use of machine learning for these purposes, even fully disclosing the exact algorithm wouldn’t help (spammers or the public alike) very much as you also would need a copy of the training corpora and all hyperparameters for evaluation.
For human content curation, we have various standards like disclosure of potential conflicts of interest, and “Chinese walls”, and I believe that is the direction we are going to have to pursue for many of these algorithmic systems, rather than uselessly insisting that all details of the deployed systems be open to examination.
Of course, deciding which systems need what sort of disclosures is itself a rich area for policy discussion and disagreement, but the point is that painting any and all hiding of implementation details as “security through obscurity” isn’t helping.
3 Likes
Would someone get the Internet back to Big Ben. The elders of the Internet have been asking about it.
3 Likes
The problem is, algorithm programmers have self-serving bias.
3 Likes
I strongly disagree with O’Reilly on 3. Take the “unintentionally” racist hiring algorithm. The algorithm’s consumers want the algorithm to be “unintentionally” racist or they would do something about their existing hiring process. The only thing they’re trying to do is speed up the process and make human beings less accountable for it.
“Don’t blame us, it’s the algorithm. Who knows why it only shows us resumes of white guys or women whose names could be mistaken for a man’s. In fact, I bet it has nothing to do with your name or anything on your resume. Do you have a political opinion on Facebook?”
People who make these algorithms have a lot higher responsibility to humanity than the low bar Tim set on #3.
1 Like
This topic was automatically closed after 5 days. New replies are no longer allowed.
