IT as a whole is much like the janitorial staff: a lot more important in standing between you and deep shit than their lack of prestige would suggest.
4 Likes
Yes. I think most of us who have actually worked for a living could probably write a book (or at least good sized pamphlet) on the shit we’ve seen.
We had a VP at one company where I worked and his slogan was “IT like air” as in he wanted the company as a whole - and as an extension customers - to just have the services they needed without having to even think about it. “It just exists”, he would say. Which is fucking wrong. The business needs to understand clearly what it takes to provide them with the necessary tools to make the company successful, keep the the information safe, etc… They don’t need to know the full inside and out of the entire infrastructure stack from the edge router to servers and storage to the port they plug into - but at the least, the business leadership across the organization should be informed and should be held responsible along with IT leadership when they push back on critical infrastructure recommendations. Further, proper cross-charging for projects and direct business requested expansions of infrastructure should also be in place.
4 Likes
I think that’s a crock. Sure, she could have been incompetent, but not because of her college major. I’ve worked with competent security professionals who had degrees in biology, chemistry and Russian literature, and with others who had no degree at all.
We all know the problem is more likely to be with the money and resources Equifax management invested in security.
1 Like
Considering the magnitude of the breach i’m not going to give this person any courtesy. They did not have the proper credentials for a position of such importance.
2 Likes
Equifax considers the problem so important that they’ve hired two PR firms!
5 Likes
Where’s the clawback?!
1 Like
Without it, you die in about 3 minutes?
5 Likes
Does anyone know WTF they are doing in a position where they are supposed to know what they are doing and why should they feel like they can tell me how to do anything? “Oh, nice title to go along with that fake diploma on the wall.” When the FBI busted a fake diploma ring, further investigation was of the realization that one percent of all diplomas are fake.
2 Likes
Fixed that &tc.
11 Likes
That would require thinking!
Depends how long you can hold your breath in the bathroom at the data center. Which, believe me, is necessary sometimes.
1 Like
I’m a somewhat senior SRE. In healthy IT/Ops cultures, you do not fire the guy who screws up. In fact, one of the most entertaining interview questions I’ve ever run across is “tell me about a time you fucked up production”. Because we’re all human, and we all have stories. Willingness to tell those stories in an interview context shows a lot of self awareness and ability to learn from those situations.
At the same time, this situation is pretty extreme. Firing someone might be merited, but I still don’t support firing the front line IT techs. If anything, this isn’t a failure of the guy at the bottom of the chain, it’s a failure of management. Getting rid of the C-levels may actually make sense. But the rot is probably deeper than that. Their IT department may not have enough budget, bad priorities, too few staff and bad processes. This is likely a systemic problem and the rot is top down. Forcing the C-levels into retirement may or may not have been the right move. Laying off the junior staff is definitely not the right move. And generally, making the junior staff who pushed the wrong button clean up the mess under close mentorship/supervision of a senior tech is the right way to advance the junior tech’s career. No need to be mad, just get it fixed and running. System recovery is a valuable skill and there’s really only one way to learn it…
7 Likes
I agree with everything you said.
Also - a friend of mine got a job one time working for an ecommerce company. He took their website down for like a half hour one time his first week there. Did not get fired. Shit happens. I took down the print server at a company one time - a couple thousand local users - thankfully kind of early in the morning. And it was a VM so I bounced it and it came right back online. Co-worker of mine did a data transfer to a new NetApp for all the department shares of a FDA regulated facility that we managed and he screwed up the permissions leaving it Full Control for “Everyone”. Didn’t double check his robo copy script.
2 Likes
More than once on discussion forums, I’ve seen people complain about the IT people being dicks over security. Here’s Exhibit A on why the IT security people need to be dicks.
4 Likes
Sounds like financial engineer’d failure.
sell high, buy back in low… its nice to have insight information 
1 Like
Actually, I find the rub is that the customers do not want the inconvenience and expense of real security. It’s pretty easy to default to minimal industry standards when the customer, if they get any choice at all, will always opt for the most convenient and cheap solution because they assume that all competitors are equally secure (and how would they know differently?)
Since security is often a competitive disadvantage, the incentives to secure your systems more than minimal standards are not exactly high. (You don’t want to be the worst security, just the 40-70th percentile.)
Now of course, this means you are at greater risk of an incident (everyone is at risk of an incident, all you can control is the size of the risk), but that’s a risk you are forced to take if you don’t want your customers to abandon you when your extra security costs them money, or worse still, convenience.
Now those minimal standards are improving year after year. But then so are the criminal’s cunning.
Of course, in this instance, it does look like Equifax may have been down in the 80-90th percentile of security for their industry, although I wouldn’t be particularly surprised if this was motivated more by uptime (and risk to that uptime) than cost.
3 Likes
Remember how Cory pointed out all the problems with the site that was set up to check if your SSN was compromised? My theory is that it was set up by people outside the IT dept because there was no confidence in the IT people. Now maybe I think they set it up too!
1 Like
Good security doesn’t have to be about money. But you do need to have reasonably competent people working all through your organisation, and good practices enforced from the top down. Where I work the wordpress sites are constantly getting hacked. There are several reasons for that. They aren’t really expensive to fix, but the people in charge of keeping them secure only really think of closing the stable door after the horse has take off.
1 Like
You’d fire a security guard for leaving a door unlocked. I can be fired for screwing up big time as well.
2 Likes
True, however, the level and nature of it varies widely by industry. When I worked for chemical companies, their main focus in IT security was keeping data out of competitor’s hands. That was also the case with pharma firms. In healthcare, it was not a priority, which is why we read so much about breaches there. Banking firms seemed to be the most secure and on top of keeping up with federal regulations & requirements. It was treated with respect, because they knew how it could affect the bottom line.
Given the nature of the data (and how their customers access it), Equifax had no excuse for treating IT as a low priority.
3 Likes
