A detailed technical rebuttal of Bloomberg's "backdoored servers" article


#1

Originally published at: https://boingboing.net/2018/10/27/someones-lying.html


#2

Although at this point we can conclude the Bloomberg article is full of shit (I expected confirmation in 2 weeks), reading between the lines the attack described in the first one (but not the second “ethernet jack” one) is actually plausible and effective…

Namely, you have a 6 pin “SPI interceptor”, a Man-in-the-middle chip between the SPI flash used for the BMC’s program and the BMC itself. Unless the BMC chip only runs signed code, the SPI interceptor could patch the BMC’s OS with a malicious additional payload, which would patch the host OS when it is installed under the direction of the BMC.

This would only take modifying two traces on the board and adding a power and ground, and the hardware chip used for this (which would be mostly the same as a serial EEPROM) could be reprogrammed for all sorts of targets since “load ROM from serial Flash” is a common motif in hardware design).

And the BMC, by being able to speak on the PCI bus, is a “god mode” chip: access to the PCI bus means, with only minor exceptions, ability to fully control everything.


#3

So there’s means, motive and opportunity, but…it didn’t happen.
Ok then.


#4

Well, I certainly don’t buy this…

And while I buy this level of malicious ass-covering, I don’t buy this level of organized cover-up (there would be squealers, probably not all anonymous and certainly not all only talking to Bloomberg)…

That leaves two other alternatives. Either Bloomberg, including the staff and editors of a nine billion dollar media company, risked their entire reputation weaving a fiction out of whole cloth and made-up sources knowing full well it would be debunked and they’d go down for it.

Or…

The latter is by far the more reasonable explanation for what was undoubtedly at the very least a sloppy fuck up.


#5

I’m not quite sure what could be more sensitive than the code that determines what the server is going to do when you turn it back on again; which is what BMC access conveniently provides.

Most sensitive data; no, you’d have to leave that up to some other aspect of the system; but owning the boot process and having a persistent peripheral on your side is about as sensitive as it gets.

It looks like Serve The Home followed a potentially poor turn of phrase toward an arguably true but not terribly helpful conclusion there.


#6

tl;dr summary of the quoted part:

“You can’t access drives and storage when they’re powered down.”

Did the Bloomberg article insist that the Chinese were doing that?


#7

Bloomberg’s article is State-sanctioned propaganda implying that China is attacking us. This is distraction from the Russia 2016 election investigation.


#8

But Bloomberg double checked the article with Ian Restil of Jukt Micronics!


#9

I’m going to be intentionally vague here, but I was at an event last week one of the companies mentioned in the Bloomberg article put on. I had an opportunity to bring the article up with a member of their FED team, even, and we chatted about it for a bit.

My main takeaway: Internally, this firms’ employees, even those who interface with sensitive customers, do not seem to think there is truth to this. This wasn’t a C-level employee by any means, however. This person said they have received a ton of calls related to the article, but it hasn’t actually affected anything on the ground.

I was told that Bloomberg did go to this company with the claims prior to publication, but no actual evidence was offered to them other than the statements of anonymous sources.

I suggested that they ought to seriously consider suing Bloomberg, because the publication of the article, to me, seems like a malicious act. I said that this would show people like me that they are not afraid of what a discovery process in court would uncover, and that even if they lost the case, it would likely cement the falseness of the story, for those who remain open to the possibility that indeed, hardware was altered. This person didn’t seem to totally disagree, but they in no way represented legal.

This really doesn’t clarify anything, but it does offer some color of what it has been like “on the ground” for some of these companies, post-Bloomberg article.

This continues to be one of the oddest stories I’ve ever seen in the media about the tech industry, having watched it very closely for 25 years.


#10

Agreed. And if none of these companies do sue Bloomberg, it will only serve to leave a giant question mark hanging over the article’s claims.

Personally, I find it really improbable that Bloomberg just straight up made the whole thing up. Getting caught was inevitable. The evidence either way is in everyone else’s hands.

But I also find the claims incredible in the original sense of the word, mostly based on the patient technical analyses of the skeptical hardware experts.

I absolutely believe Bloomberg reporters misunderstood what their sources told them - sloppy tech and science journalism is endemic - and I wouldn’t be surprised if that misunderstanding combined with some ill-considered sensationalist hyperbole to result in a story that’s more wrong and right.


#11

This doesn’t feel right to me. They had 17 sources. Let’s say it was corrupted firmware, far more likely, and really not all that effectively different, other than how blatant it is. You’re telling me the entirety of Bloomberg’s editorial staff confused corrupted firmware, for embedded chips? That makes no sense. Good chunks of the article went into detail about the location of the bad chips on the boards, what they did, different generations of the hardware, etc. This was absolutely not a misunderstanding.


#12

There are two problems with the analysis.

  1. Although one issues commands only on the baseboard management LAN channel, that does not mean that the BMC cannot read from the primary LAN network port. Firewalls can protect against inbound access to the BMC, but can provide only ordinary protection against outbound access from the BMC via the primary LAN interface.

  2. What a BMC can or cannot do when the main processor is powered off is only interesting if you never plan to turn the main CPU on.


#13

I found myself thinking the same thing. This was one of the poorer analyses of the Bloomberg article I’ve seen. MANY others actually suggest that what Bloomberg describes is not only possible, it’s in fact been a vector of concern for a while now.


#14

You’re missing the obvious. This article grew out of a year-long investigation, in which Bloomberg’s journalists talked to 17 sources. If any of those sources had actually handed Bloomberg the full picture as it’s presented in the article, then it wouldn’t have taken them a year to put the story together. This is a case where the journalists did a lot of detective work, talking to bunches of people who only knew a few bits, and then assembled the full narrative in the course of writing the article. The danger of such reporting is that if the journalists get a wrong idea about what the nature of the story is, then their reporting is complete bullshit because they’ve put the pieces together wrongly. After spending so long working on it, they are invested in the narrative they’ve put together and so they ignore all the signals that maybe that narrative is off base.

Also, please remember that Apple and Amazon know better than to flat out lie about shit. If they lie about something that has affected their stock price (which this Bloomberg article has done), then they open themselves to SEC investigations and lawsuits from shareholders. That is why corporate lies always take the form of non-denial denials or masterworks of weasel wording. For both companies to so completely, categorically, and unambiguously say there’s no truth whatsoever to Bloomberg’s reporting, suggests that they are, in fact, telling the truth.


#15

I don’t recall any claims about BMC or SPI in the Bloomberg article. Nor did they discuss the back channel, other than to suggest some unusual network traffic in at least one case.
Spook stuff resembles magic: asymmetric force forward, side channels backward.
These rebuttals cover the obvious issues. They’re useful but ultimately likely to miss the real story.


#16

They had 17 sources out of over 100 they interviewed who agreed that something might have happened - after consistent pestering by these two reporters who have done sloppy work before.

People who had been contacted by them and said ‘no no no’ have talked about how these reporters insistently pursued the ‘yes yes yes’ line and ignored all evidence to the contrary. They wanted there to be a story, and they got 17 people who had heard from other people that something might have happened and ballooned it into something huge. And maybe a couple who were deliberately feeding them a story. This is something the Trump administration would love for support for their trade war with China.

As for Bloomberg editorial, I don’t know. It’s a business magazine, do they have the expertise?

The most obvious problem with the whole story is that supermicro motherboards are so insecure you can do this much simpler, easier, and cleaner by just messing with the UEFI and other firmware. Maybe some cases of this actually happened and it ballooned into this story.


#17

What baffles me is how they expected to get away with it. I mean, it’s not like InfoWars or Fox News where the audience is happy to be fed a load of bullcrap. Bloomberg is a serious business and technology media company and they had to know this would get torn to pieces and cost them credibility. Even if the reporters were being sloppy, where the hell was the editorial oversight on this thing?


#18

Some of the unpacking articles I’ve run into have suggested that the Bloomberg piece’s description of investigations and meetings between the companies involved and US intelligence roughly match an actual casual conference that took place, in both attendees and topic. With the proviso that that conference was about speculative looks at potential supply chain based hardware attacks and how they would work. A couple of the companies in their response suggested that Bloomberg’s reporters might be confused by actual, known, published, investigated, and fixed hacks and security holes that were superficially similar. And this rebuttal article points out that a few of the features of the supposed attack resemble shit like Spectre and Meltdown. And that their description of the magic chip matches pretty exactly with the real world features of the BMC. As in the list of actual connections and components it has is nearly identical to the connections and components the BMC complex uses to do its job.

It seems pretty plausible that these Reporters could have caught 2nd or 3rd hand accounts of such a meeting Took the descriptions they were given as a real attack. Combine that with some bad misunderstanding about the hardware (BMC becomes super chip), and sources talking about actual, less concerning attacks, or internal company rumors. The kind of “Bring me the invisible suit from predator” credulity we see in some Government department heads. And they arrived at the story we have now by trying to string a bunch of conflicting shit they don’t really understand into a cogent narrative.

But nearly every one I’ve seen take that line points out that it wouldn’t be possible as Bloomberg described. In fact its a bit hard to tell what Bloomberg is describing because they offer multiple, conflicting descriptions of how the attack works and what the chip is even doing.

Exactly what I’m thinking.

Bloomberg itself is a business mag yes. But they’ve also been running one of the major new wire services for a while now. The Business mag is… Business maggy. Lots of news by press release, puff pieces. Some good reporting. The newswire started as a business newswire. But as the AP got, well, weird in the 00’s and some of the other US wire services struggled. They shifted a bit to a more general news wire service, and for a while there took the default/of record status from the AP. They have a pretty good editorial reputation over all. Though they seem to have pulled back to a more business focus the last 5 years or so.


#19

I essentially agree. However what you fail to address is the role of the editors and even ownership of the magazine. The writers of articles don’t just hit “submit” and the article goes to press. Especially so when it is a major, major accusation against large corporations, with geopolitical implications. And editors actually do generally do an awful lot, at major news organizations, when reviewing the supporting materials for a major story. So it goes way beyond the reporters, and, if it is really quite false and off-base, represents a top to bottom failure of a major news organization. One that is generally fairly respected. I won’t even get into the brand Bloomberg, and the man behind the brand. He has political ambitions, last time I checked. Which just makes the whole thing weirder.


#20

It wouldn’t be the first time its happened. Remember that Rolling Stone campus rape story? And Rolling Stone is really, really well known for investigative reporting. Stephen Glass? Jayson Blair at The New York God Damn Times? Those structures are in place so such things don’t happen. Not because they preclude it entirely. And its entirely contingent on editorial (and legal) caring, and understanding the topic well enough to vet it. If the reporters don’t understand the topic (and they don’t appear to), and the editors don’t. Then it could look entirely kosher. All legal is going to do is double check that asses are covered, its not their job to vet veracity.

Bloomberg isn’t known for investigative work or tech reporting (or just technical reporting on anything but finance). They’re a respectable enough organization. But there’s a difference between their Business operations, which aside from news does things like real estate investment lunches where you can learn to get rich quick. And their full news side which is mostly focused on a breaking news basis. They’ve got a small history with publishing shit that turns out to be wrong, and a tendency not to fix it or publish corrections when that does happen, especially on the Business side.

These two reporters apparently have a bad record on this topic as well. And I think what’s telling is nothing more has come from Bloomberg to back this up in 24 days. Coming up on a month that their credibility has been the major topic to come out of the story. And all we’ve seen about how this was reported is their few named sources walk it back. And complaints from people who refused to participate about how the reporters conducted themselves.

I’m not particularly acquainted with this level of hardware. But I’m extremely well informed about how journalism works, and critically unpacking questionable claims in the media. And this doesn’t look properly reported. Beyond that you have a report of a specific event happening. With rough dates and parties and details. And no 3rd party has been able to verify that it even took place. That’s a very, very bad sign.