Speaking as someone who has peas coming up in the garden: I have to pull weeds and fence out deer and encourage mantids and crush other insects if I want to have any crop at all.
If I do not aggressively manage my property, I will lose a great deal of species diversity, because the uneven introduction of non-native species without the complex predator/prey/disease/etc. relationships that normally allow them to exist in a stable equilibrium means species like lesser celandine, Japanese stilt-grass, Japanese honeysuckle, Australian mile-a-minute vine, Japanese hops, Chinese bittersweet, and European loosestrife will explode due to absence of predators and diseases, completely wiping out the native ecosystem. I have seen what happens when devil’s tear-thumb goes unchecked! Species diversity plummets.
But I’ve never found it necessary to use herbicide, fertilizer, or imported soils in my garden or property. Normal composting with food waste, leaves and grass clippings (I mow to keep down stiltgrass and ticks carrying Lyne’s disease and Rocky Mountain Spotted Fever) generates enough good fertile soil for me.
Also I have my own definition of weeds which is pretty idiosyncratic. The highly aggressive native trailing bindweed I treat like kudzu, and for good reason, and I pull about 80% of the native wingstem, but I let naturalized non-invasive foreign species alone for the most part.