I think you nailed it.
I was an avid marvel fan as a youngling, collecting a number of books starting in the 70s. Fantastic 4 was never part of my collection unless a specific arc pushed my favs and that group together.
That doesn’t mean I was unawares, as it was very necessary to digest older Fantastic 4 yarns to truly understand how/why certain books/characters did what they did.
But it never interested me to buy or buy into the F4. As you said, I’ve never stumbled upon a die-hard F4 fan back in the day, whereas X-Men, Spidey, Avengers… had huge fan base.