As always, classism plays a role here.
Go back to the times of aristocracy, and it was a sign of wealth and status to be able to have perfectly crafted loaves of white bread available for your next extravagent display of waste and excess that was your average party. But then, the Pullman loaf and modern what milling made white flour trivially easy to acquire, so what was the upper-crust class supposed to do? Talk down the very thing they used to consider a status symbol, of course.
