That big Amazon S3 outage was caused by a typo, company admits

boing
1

Originally published at: http://boingboing.net/2017/03/02/oops.html

…

5 Likes
2

Well, that explains the tinder Netflix, err, outage.

9 Likes
3

Well, sounds like they fixed it. We should, as a society, put more things on the Amazon servers!

4 Likes
4

“Wups.” ~ Darrel, the IT guy

9 Likes
5

RIP that tech’s job at Amazon. Unless the playbook was actually wrong, then RIP the guy that wrote it. And possibly RIP the developer that created one such command to take down so many servers at once, even though he probably created it correctly based on requirements. Maybe RIP the requirements writers or IT architect that allowed such a tool to exist. But all management overseers will be fine. Accountability lies with the little guy. We’ve all heard what it’s like to work there.

7 Likes
6

http://img.pandawhale.com/post-40444-Johnny-Airplane-Just-Kidding-g-T1Q3.gif

19 Likes
7

I got escorted out the door after a single digit mistake on a DNS update. Freaked me out at the time, I now wear it as a badge of honor (I may have uttered, “by the power of greyskull!!”)

24 Likes
8

With great power comes great responsibility.

1 Like
9

I’m actually impressed that they revealed so much detail to the world at large. I guess they needed to make a giant mea culpa to ensure that their clients using Amazon’s cloud knew Amazon was taking it seriously. Still at least it sounds like they’re doing the right thing (figuring out how their system allowed a single tech to create a massive DOS with a single typo, and fixing it so it no longer works that way) instead of doing the typical thing (figuring out how to find the right scapegoat to blame it on so they don’t have to admit that their system currently requires humans to be perfect operators in order to guarantee uptime).

7 Likes
10

Well, they’re telling us about the technical fix. Maybe not so much about the human resources fix.

3 Likes
11

"I brought down S3"
Achievement unlocked

27 Likes
12

Probably doing both.

13

http://gifyu.com/images/108-not-something-ill-forget.gif(image larger than 4 MB)

9 Likes
14

That guy is going to have a failure mode named after him.

“Oh, cripes, I McKenzied the command line!”

9 Likes
15

‘Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems.’

12 Likes
16

Just one wrong bit and you’re in the shit

17

And this is why I always use a “?” Instead of typing out “PRINT.”

4 Likes
18

that’s a c64 reference, is it not?

1 Like
19

C64? NEVER!

4 Likes
20

$_ =~ s/./ohcrap/bg;

5 Likes