I was going to write this but you basically nailed it.
Becoming a comedy nerd and understanding stuff like that does basically ruin the magic, but you then appreciate the craft and skill and you still laugh. Like understanding how magic works, the best magic (like the best improvisational comedians) you can't see working and not only do you not understand it, you often don't want to.
When it comes to ad-libbing you rarely see nearly as much off the cuff stuff as you expect, much of it has been thought about or considered or done before or they use certain techniques to draw out laughter. Some will be genuine ad libs of course, the thing is when they go well The real skill with comics who thrive off of improvisation is not conintual off the cuff remarks but the ability to draw and link new ideas, stuff from the audience or situation and existing material together so you can get to what is funny and make it seem natural.
Ross Noble is one of the masters of that - he starts improvising on stage and keeps improvisation going throughout his shows to the extent that generally you will have no idea where the improvised material ended/started and the scripted started/ended,
It's hardly surprising Robin Williams was repeatedly accused of plagiarism, he went so fast I doubt his internal filters would have been fast enough to notice the idea he was dragging up in his memory wasn't his own.
I saw the phenomenally talented Nish Kumarat the Edinburgh Fringe last week and while he's a clearly a stand up who has his material absolutely down and honed he did a fantastic ad-libbed line, Then he stopped and said, actually that was a genuine ad-lib on the 4th show of the run but now 15 shows in I've kept it in and pretended to ad-lib it every single night since (getting further laughter).
Also making stuff seem off the cuff when it isn't is also a good skill (as Stewart Lee describes Eddie Izzard, making it seem like ou are improvising by saying um a lot) people like Williams use. The Mighty Boosh go to lengths to seem ad-libbed and conversational but they agonise over basically every word in ever line.
While I appreciate Robin William's skill in terms of traditional 'improv' (as opposed to the more broader concept of improvisation in general) I actually think his routines and improv style and material is actually very uninteresting and standard at best, his real talent was to some extent his motormouth and just how fast he got stuff out but it was really in the use of his voice to control the audience and draw out the laughs from not very good material.
I find most American style 'improv' (I say that because it is far less common outside of the US) pretty tedious and it conforms to too many rules and styles and is rarely unexpected or unique the only time it's really worth watching a comedian who is brilliant at improvisational and off the cuff comedy are when you get people who can do it but use a completely different and often unexpected frame of reference.
I saw god/man Daniel Kitson do a show with Andy Zaltzman and Alun Chochrane earlier this week called Fuckstorm 3001 that was mostly improvisational just shouting back and forth at each other across the room of The Stand One (although it was commented on by Kitson that Andy Zaltzman just has so many puns and odd jokes on so many subjects he wasn't improvising so much as just searching his memory for one liners, puns and jokes) .One of my favourite parts was him when given a chance to speak during an improvisation line Chochrane had taken he went 'Alun, I know the rules of improvisation say I should say yes and keep it going but my comedic senses are telling me I should say no and shut you the fuck down'.
I feel like that about most 'improv' comedy.