By “us” I do not mean those of us who are wily to the ways of deception. I mean the general public and the media at large. Sure this is in the conspiracy theory camp, but we are talking about National Security stuff here. Before the various leaks regarding what the government was up to, those leaked ideas were conspiracy theories. The media needs to be more skeptical of what the government says, because it’s track history on this sucks.
As shaddack has said, the devil is in the implementation and there is more to implementation than just the parts of the code we would normally interact with. I am less interested in the crypto algorithm and more interested in everything else about the crypto subsystem. The crypto is probably fine, the subsystem is where the problems will lie.
First off there is the programming. Unless we can read the code the secure subsystem is running, we will never know if it has been backdoored. To continue in this vain unless we do xray microscopy, we will never know if the hardware has been backdoored (extra undocumented instructions or instructions that behavior differently when fed specific values). Given that a number of the NSA projects involve replacing surface mount components on circuit boards, it is not far fetched that they might substitute their own chips in anywhere along the supply chain. The same goes for the programming of said chips (many of which are their own embedded systems). And this is all before we have even talked about side channel attacks. Side channel attacks are hard to predict and hard to defend against.
What I have read suggests that at least for Apple, the crypto subsystem as it is programmed by default cannot be read, but Apple can update that programming. Which basically means the previous statement about not being able to read the contents goes out the window. You just have to trust Apple not sell you out, for them to have secured their private crypto keys they use to sign updates, and to pray that nobody has brute forced said keys.
Given the complexity of chips these days (billions of transistors) I do wonder if it would be possible to sneak extra circuitry into the designs and have the foundry unwittingly produce it.