The huge trouble with these discussions about the evolution of intelligent alien life across the universe, is that we are unable to grasp the astronomical scale of time that has passed already.
Consider for a second that another alien intelligent life has evolved elsewhere - but just, a couple of millions years before us.
What do you think would be their level of technology? Of consciousness? Of invasive strategies?
With their level of sophistication; maybe they have already invaded/seeded/colonized earth a million years back! And we could have easily forgotten with our limited lifespan of 100 years.
With these kind of timescales; even your “fact”/basic premise that we haven’t been invaded is speculative…
