Messy: When automated anti-disaster systems make things worse, and what to do about it

Humans certainly aren’t immune to this(see every ‘bubble’ ever, lethal stampedes where a nice orderly queue could have gotten everyone out in plenty of time, etc.); but one thing that computers’ “Do exactly what they were told to do, very fast” style is not good with is when the orders given produce unpleasant and unexpected emergent behavior.

People can, and all too frequently do, keep merrily marching toward disaster in these cases; but they do sometimes recognize that This Should Not Be Happening and either freeze up or start improvising, which can head off whatever was going to cascade out of control.

Under the same circumstances, computers just cascade out of control. Sometimes this is merely humorous(reseller bots on Amazon getting into little bidding wars with one another), sometimes it goes genuinely badly; but myopically continuing to do the wrong thing is pretty much assured.

That said, the great thing about computer systems (of at least reasonable quality) is their very good resistance to ‘dumb mistakes’. Humans get tired, get distracted, try to multitask, and suddenly they’ve slipped a decimal place, omitted a step, or transposed a couple of characters. Computers are much better about avoiding that. They’ll never do the right thing on their own initiative; but they don’t produce a modest but steady stream of idiosyncratic mistakes.

2 Likes