Originally published at: https://boingboing.net/2024/06/05/recall-recall-microsofts-ai-feature-a-hackers-wet-dream.html
…
4 Likes
As a Healthcare IT person I can see HIPAA violations galore with this. Only saving grace is we usually are behind on OS versions so gonna squeeze Windows 10 till beyond it’s expiration but my lord in plain text is just so awful. And if not in plain text, the little hackers will probably use AI to crack whatever they did have in place so essentially use it against itself. So tired of hearing about AI.
19 Likes
https://www.antipope.org/charlie/blog-static/2024/06/is-microsoft-trying-to-commit-.html
Is Microsoft trying to commit suicide?
By Charlie Stross
The breaking tech news this year has been the pervasive spread of “AI” (or rather, statistical modeling based on hidden layer neural networks) into everything. It’s the latest hype bubble now that Cryptocurrencies are no longer the freshest sucker-bait in town, and the media (who these days are mostly stenographers recycling press releases) are screaming at every business in tech to add AI to their product.
16 Likes
It seems to me that this is the sort of very optional, but possibly useful to some people, feature that very much must be highly configurable by the user, with a wide range of parameters (on/off, frequency of capture, text or images, encrypted or not, retained locally or clouded, and so on and so forth) and not something embedded into the OS and switched on and operating from the moment the OS is installed. It should also come with plain language risk warnings about the possible consequences of switching it on and configuring it in certain ways.
But it’s Microsoft, so it doesn’t.
(Also, “rather large gaffs” would be big hooks for landing large fish or a type of sail-rigging, whereas an egregious error is a “gaffe”.)
5 Likes
And without Microsoft continually switching reporting options back on.
16 Likes
Yep - configurable by the user means just that - but of course MS thinks that it is the user, here!
9 Likes
I’d be morbidly curious to know how much of this is down to the product being in the hands of ‘AI’ true believers and how much is down to people with strong and reasonable objections being told by The Business that it has to be ready for the launch of glorious Copilot + PCs, or else; because the security model described is utter lunacy; and in ways that include threat model failures that other Microsoft products and security guidance do not possess.
It’s not clear that there would be anything you could do to keep malware running in the user’s context from having access to the same ‘recall’ search interface that the user does; but failure to protect the database from anything in the user’s context or anything running in the context of the admin user opens a significantly larger hole(access to the ‘recall’ searches would still be powerful; but could be rate limited and requires knowledge of suitable prompts to get some of the more…nonstandard…items of interest); since you can just snag the entire thing.
Keeping the grubby fingers of even the local administrator out of things is something that their DRM side has been attempting since more or less forever; and concerns about specific sensitive data and processes being tampered with have driven the introduction of things like HVCI/VBS, UEFI-locked LSA process protection, Hyper-V Shielded VMs, and the like; most of them not all that new.
Security guidance has also treated the difficulty of keeping malware from at least being able to execute in the user’s context as great enough that it basically has to be treated as an assumption.
Seeing something where the level of protection is “well, it’s on a bitlocker volume” for information of that sensitivity gives an overwhelming impression of zero fucks having been given. Not even ‘tried but embarassingly failed’; just nothing.
7 Likes
Just imagine what kind of embarrassing Microsoft corporate documents attackers are going to be able to steal from the Recall databases on their employees’ computers.
And if Microsoft turns off this feature on their machines, that lack of confidence in their security should indicate no one else should even think of enabling it.
8 Likes
North Korean tablets have a similar system, only without the AI woo.
7 Likes
Thankfully; the handy attack tool has made it to general availability even faster than the feature. Progress!
9 Likes
I don’t think I’m going to rush out to buy a new PC with the added cost of an NPU that meets their specs, plus lots of extra storage for Recall to play with, and then hand control of it to Microsoft.
6 Likes
I find Time Machine occasionally useful, but at least it can be encrypted (I think. Pretty sure.).
4 Likes
This “feature” is spyware, plain and simple. There’s no use case outside of some very, very specific edge cases, and software already exists for that. (including the already present “Steps Recorder” tool, which is used to capture actions for troubleshooting purposes.)
The only windows that you can configure it to ignore are edge browser sessions to the site level (i.e., you can tell it to not capture the contents of an edge browser window going to mybankingsite . com, for example. All other browsers? nope. So you Incognito windows are also captured.
And since what’s on the screen is what’s captured in the image, it’s a potential HIPAA and PCI-DSS violation waiting to happen, and those will be on the business first, which can then choose to try and sue Microsoft. (good luck!)
Now, corporates do have some legit use cases for this type of spyware- [RedactedCo] uses a third party program to monitor vendors that are accessing our systems by remote in the event they do something stupid and break a production app.
There’s no legit reason I can think of for this being installed on Jane Random’s personal computer. I can see an abusive partner using this to monitor/stalk their victims, or abusive / overbearing parents tracking/stalking their kids with this, and other such violations of personal privacy. (Along with a hacker leveraging it to extort money from their victims: “I can see that you have visited gandalfbignaturals . com and paid them lots of money, despite it being in a private window. pay me xx bitcoin and no one will ever know- except the two of us.”)
I can also imagine a resurgence of such spam mails if this ‘feature’ is made part of the next iteration of windows, because Surprise, Surprise! You can turn that feature on without having an NPU or Copilot + PC device. (multiple people have published how to do this)
I don’t know what Microsoft was thinking for this feature, if they were thinking at all.
I haven’t seen enough information to determine if it’s an attempt at duplicating Apple’s Time Machine feature, or some other application that will let you pull up saved snapshots of a system state from a point in time in the past.
I don’t like it, and I don’t appreciate Microsoft trying to force it down people’s throats.
13 Likes
[Image of Tux the penguin, winking]
5 Likes
I think that this assessment was on to something when the author says:
At a surface level, it is great if you are a manager at a company with too much to do and too little time as you can instantly search what you were doing about a subject a month ago.
In practice, that audience’s needs are a very small (tiny, in fact) portion of Windows userbase — and frankly talking about screenshotting the things people in the real world, not executive world, is basically like punching customers in the face. The echo chamber effect inside Microsoft is real here, and oh boy… just oh boy. It’s a rare misfire, I think.
It’s a capability that has some narrowly interesting use cases(and is certainly more powerful than something like good old Problem Steps Recorder; though less powerful than the more dev-focused tools that actually get their fingers into the actions of programs directly; rather than just machine-vision-ing screenshots; but presumably enjoys broader compatibility); but because ‘Copilot’ is currently Microsoft’s baby a horrifically underbaked version of it is being shoved out the door as a consumer on-by-default rather than being the special-purpose-for-already-heavily-retention-policy-and-no-expectation-of-privacy-on-company-hardware corpo feature your admin can enable on Win11 Enterprise E5 with upsells edition if and only if a bunch of fiddly virtualization based isolation features are operational.
9 Likes
9 Likes
You certainly won’t find me saying nice things about Microsoft’s judgement, especially around anything related to windows 11(except for a couple of things that they don’t preen about; like HVCI/VBS and Credential Guard now being on by default; and SMBv1 deprecation); and I’m still spitting nails about the absolute madness on display(metaphorically, they specifically tried to hush it up to the degree possible until that became impossible) during the Storm-0585 EO intrusion; or the AAD user Oauth grant defaults, or the “people love slack bots; and we are trying to fucking bury slack, so teams is getting a swiss-cheese ‘app’ security model” issue, or a bunch of other things I probably shouldn’t be getting into the weeds on.
That said, the disconnect between MS’ delivery of “buy a new computer so that you can run glorious Copilot + spyware!” and the reception was markedly more dramatic than seems typical.
They’ve been having trouble getting anything more than sullen disinterest with the various things they are doing to win11; but visceral fear and loathing for a headline-of-the-entire-event feature was well above the norm for failure to read the room.
6 Likes
I use it on my MacBook, and it can be encrypted.
3 Likes
Are you a coworker? 
I just sat through a meeting about IT things and the person demoing a product kept calling that product “AI Adjacent”. That’s not AI bro.
1 Like
Meanwhile you still can’t do a slide show in Microsoft’s stupid photo software.
I may have to give up using Windows.
5 Likes
