I’d agree with that, which is the main reason I described the picture as incomplete; and in real-world cases it seems like the attractive arbitrage opportunities between money and influence and back again always seem to overwhelm people’s force of habit, or noblesse oblige or pragmatic concern that inspiring a proletarian uprising would be bad for business; so I definitely wouldn’t dispute that, as a matter of history, as well as definition, an oligarchy involves the economic rules being distorted in a variety of ways for the advantage of the already rich.
That said, I’d say that the issue of the perception of oligarchy and its connection to the trend in overall utility within a society likely exists independent of(though obviously closely connected in practice to, since oligarchy is both an efficient means of moving wealth to low marginal utility parties and what people call the outcome when wealth is flowing to low marginal utility parties) an operational oligarchy:
If a hypothetical oligarchy decided that a clean, civilized, welfare state with favorable human development indices and such was really the best way to keep the heat off while they made real money; I strongly suspect that the result would, except to all but the most principled anti-corruption campaigners, generally be read as “unequal but prosperous”(I suspect that this is not particularly hypothetical, unfortunately, in the case of some of the world’s dodgier tax shelters: if you spread just a smidgeon of the good fortune around locally you can enable monstrous oligarchical abuses on a vastly greater scale abroad without facing popular condemnation for them, since your policy doesn’t read as oligarchical at home).
On the other side, you could likely have a perception of oligarchy with a functional oligarchy if, say, an agrarian society suffered a few bad harvests. Regardless of their existence, or ability to control commodity prices in the face of the weather, I’d expect that a sinister cabal of commodity hoarders would soon come to exist in popular imagination.
Since people seem to have difficulty keeping their hands out of the cookie jar forever; I’d agree that the distinction between societies where utility is trending down despite absolute wealth remaining constant or increasing and societies where oligarchical tendencies are distorting the flow of wealth is mostly a matter of theoretical elucidation rather than something you’d generally find in the wild.