ICE hacked its algorithmic risk-assessment tool so it recommended detention for everyone

Once again, while we were bringing out our best scifi-futurist minds to imagine and foresee all the insidious and subtle ways that algorithms could subvert truth and democracy, the fucking brute nazis just went ahead and did the most blatant, sledgehammer version. And will get away with it. Are they wearing little skulls on their hats yet?

2 Likes

Well, it’s how the system works.

I helped build a system for static testing mid-sized and small solid rocket motors back in the Reagan era, and a lot of those motors were used by the military.

So, this guy from Lockheed shows up, and he says “I need you to checksum the system disks on the computer” to which I reply, only more politely, “wtf madness are you on about now?”

The Lockheed guy explains that he has a slip of paper in his wallet, that contains the checksum value of the OS and data acquisition system the Navy uses when they’re testing Lockheed products. Before every test, the Navy guys run the checksum program, and give him the output to compare with his slip of paper. If it’s different, they know the filthy Russians have sabotaged the system, they’re probably hiding in the drop ceiling right now!

I ask him a few questions about the Navy’s system, and on determining that it does have on-disk OS managed data structures, and that the system isn’t necessarily freshly booted when the checksum is run, tell him that it isn’t testing the whole system disk, just the pertinent programs and that we can certainly do that for him, but it’d be more meaningful for us to buy a nice mosler safe and put the media used for Lockheed tests in it. He is outraged. He has the assurance of the US Navy that the number in his wallet applies to everything on the system and no technical mumbo jumbo from some young buck with an earring is going to tell him different! Rashly, I mention that the Navy’s program probably does nothing but print a fixed string, and is merely a pacifier for Lockheed to suck on, and that I can do that if he wants.

The Lockheed guy goes to the plant manager and insists that I’m being obstructionist and should be harshly sanctioned, perhaps by skinning over a low fire. The head of Testing comes to me and yells at me for a while (he has gone down in history as my “screaming boss” since he did it almost every time I saw him) and says “give the damn guy what he wants!”. I explain that it’s technically impossible with our system, that I could either checksum the static parts of the disk, or make a fake program that printed out the same string every time. He wants me to do the latter. I refuse on ethical grounds. So they start hiring until they find a guy who will say he has written a program that checks the disk, and write a program that just prints out a fixed string.

9 Likes

Yup, “calibrated”.
By a contractor, for $$$.

1 Like

The TOS Kirk was a bookish nerd in the Academy, if you pay attention to the original series dialogues. The writers wanted him to use his wits, not his muscles. Rather, just enough action for the NBC execs. Nick Meyer used that motif when writing the script for Star Trek II.

Besides, the way he cheated was most likely done through “social engineering” — that is, getting a friend to grant access to the simulator controls and replacing the scenario. The reboot movie version was, welll, dumb.

2 Likes

Agreed. Social engineering makes sense.

This only reinforces my suspicion that Spock Done It (perhaps unwittingly?).

Mind you – full disclosure – Spock is my childhood hero – not Kirk.

1 Like

I doubt that Jim Kirk even knew Spock back then, as realistically the longer-lived, less ambitious Vulcan would already be serving as a junior science officer possibly freshly transferred to the USS Enterprise under Captain Pike. (My new head-canon is that Spock’s career was also harmed by the fact that his adopted sister was the only mutineer in Star Fleet history up to then!) I also suspect that the bridge simulator was something all officers had to go through even after leaving the Academy. So multiple tests over the years in an officer’s career, the first as an undergrad cadet, later as a candidate for command position is, well, logical.

My favourite idea of how he reprogrammed the simulator was that he stuck around after failing, volunteering to help with the clean-up after destroying the bridge, then asked for another run to try out a new tactic. His cheat was also probably subtle, adding tweaks to engine speeds so that he was able to barely get in and out without having to fire a shot. Thinking in three dimensions instead of planar thinking would have been a nice Chekov’s Gun (ha!).

Er, we have drifted pretty far off topic, as the ICS hack is more like Catch-22 than it is like the Kobayashi Maru. And I would not say the algorithm was hacked so much as it was tweaked to get a desired outcome.

3 Likes

This topic was automatically closed after 5 days. New replies are no longer allowed.