To many middle-class-and-up people, being visibly poor is nefarious.
Sometimes I’m working in remote bushland, but I’m often also on the fringes of public reserves, usually in wealthy neighbourhoods. There are no dining facilities provided in the bush; at lunchtime, we just sit on the ground. If there’s a park bench nearby, we might sit there instead. Our staff wear a clearly labeled uniform, and the gender split is roughly 50/50.
We routinely get complaints relayed via the council about “scruffy men sitting outside my house / lurking near the playground”. The only way to avoid that is to make sure we eat in places where rich people can’t see us.