Depends, but mostly I agree with you.
Old Daily Show and Colbert Show? The interviews were amazing. Colbert’s character would make the interview always go in unexpected ways and Stewart is a brilliant interviewer. Heck, I liked the interviews over the bits.
Nowadays I’ve not seen anyone really that replaces it. Colbert is ok, more so when it is something that hits in his personal ballpark. Fallon, again, is unwatchable for interviews. Seth’s were passable. Conan’s were good on average but not amazing.
Again we get back to Ferguson. There’s hundreds of videos of his interviews. They were a conversation. That we were watching was a minor detail. He openly heavily flirted with the pretty woman (and men), was a pop-culture junkie, and just lived for the Uncomfortable Pause. The show was a live-action Animaniacs presentation. He even swore funny. Nowhere else could you find out about how someone got kicked in the ooh-la-la! It was even well loved by the people he insulted, like Paul McCartney. Do we have a picture of Paul McCartney? Great, thanks.