That big Amazon S3 outage was caused by a typo, company admits


#1

Originally published at: http://boingboing.net/2017/03/02/oops.html


#2

Well, that explains the tinder Netflix, err, outage.


#3

Well, sounds like they fixed it. We should, as a society, put more things on the Amazon servers!


#4

“Wups.” ~ Darrel, the IT guy


#5

RIP that tech’s job at Amazon. Unless the playbook was actually wrong, then RIP the guy that wrote it. And possibly RIP the developer that created one such command to take down so many servers at once, even though he probably created it correctly based on requirements. Maybe RIP the requirements writers or IT architect that allowed such a tool to exist. But all management overseers will be fine. Accountability lies with the little guy. We’ve all heard what it’s like to work there.


#6

http://img.pandawhale.com/post-40444-Johnny-Airplane-Just-Kidding-g-T1Q3.gif


#7

I got escorted out the door after a single digit mistake on a DNS update. Freaked me out at the time, I now wear it as a badge of honor (I may have uttered, “by the power of greyskull!!”)


#8

With great power comes great responsibility.


#9

I’m actually impressed that they revealed so much detail to the world at large. I guess they needed to make a giant mea culpa to ensure that their clients using Amazon’s cloud knew Amazon was taking it seriously. Still at least it sounds like they’re doing the right thing (figuring out how their system allowed a single tech to create a massive DOS with a single typo, and fixing it so it no longer works that way) instead of doing the typical thing (figuring out how to find the right scapegoat to blame it on so they don’t have to admit that their system currently requires humans to be perfect operators in order to guarantee uptime).


#10

Well, they’re telling us about the technical fix. Maybe not so much about the human resources fix.


#11

"I brought down S3"
Achievement unlocked


#12

Probably doing both.


#13


#14

That guy is going to have a failure mode named after him.

“Oh, cripes, I McKenzied the command line!”


#15

‘Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems.’


#16

Just one wrong bit and you’re in the shit


#17

And this is why I always use a “?” Instead of typing out “PRINT.”


#18

that’s a c64 reference, is it not?


#19

C64? NEVER!


#20

$_ =~ s/./ohcrap/bg;