The actual truth, though, is that you’ve misunderstood the word. Irregardless isn’t based on the prefix at all, it’s a portmanteau of regardless with irrespective. Not my favorite way to generate words either because it makes them confusing, but not the mistake you think it is.
Not that double negatives are actually wrong in English either; they’ve been used for emphasis for a very long time, however they might seem to logicians. And there’s the problem with prescriptivism. English never came down to us with a full rule book in place. You have to take some aspects of how people use it and deem them the acceptable ones, and others and deem them not.
And, unfortunately, deciding which are which is arbitrary at best; as the_borderer said it tends to be classist or racist at worst. You mistaking the nature of irregardless is a trivial case, it’s a redundant word we can do without. But how do you know you won’t trip over something more important the same way?