I think you’re right about that, but it’s important to recognize the privilege in it. For most minorities, not fighting is not an option. The system treats us like dirt every time we leave the house. You can’t leave your skin color or different ableness at home and hiding your sexual orientation and other traits is exhausting.
You say you don’t judge people for what amounts to coasting on that privilege, and while I am also guilty of doing so (we all have at least some privilege and I have certainly coasted on it) we should judge that behavior at least a little. Because, well, it’s kinda the problem.