Noah Yuval Harari makes a great case for this in 'Sapiens' (the following quote is not his argument, just a pithy unrelated blurb from an NPR interview).
" It's today, I think, quite common in the scientific community to acknowledge that the Agricultural Revolution was maybe not such a good idea. On the collective level, it's obvious that agriculture made humankind far more powerful. But the individual human being probably had a worse life after the revolution than before.
The average peasant, let's say, he or she had to work much harder and in exchange for all this hard work people actually got a much worse diet. Most of the population got maybe 90 percent of their calories from a single source of food, like wheat in the Middle East or rice in East Asia. On top of that, you had much worse social hierarchies and social exploitation. Very small elites exploit masses of people for their own needs."