It seems that many drone operators feel entitled to do whatever the fuck they want without regard to others’ safety or privacy:
Guy Refuses to Stop Drone-Spying on Seattle Woman
Drones disrupt efforts to fight California wildfire
Drone operator arrested for interfering with helicopter rescue mission
Drone spotted ‘few hundred feet’ from Southwest flight approaching Love Field Friday
FAA records detail hundreds of close calls between airplanes and drones
Rogue toy drones are interfering with military operations
If these problems cannot be addressed without regulation – social norms might help a bit, but you reject that – then regulation is coming, the only question is what form it will take.