I get that there may really be some deep psychological reason for disliking clowns... but how about the very simple reality that they most often represent pretty unlikeable personalities? They're 'joyful' but their delivery is obnoxious, loud, manic and chaotic; they act stupid, clumsy and oblivious.
I personally don't find clowns scary. I tend to dread them the same way one dreads the acquaintance who always gets boisterously and destructively drunk at their party, tells the same off-colour joke 40 times and falls over the buffet table.
That said, the more quiet, less trollish 'sad hobo clown', I typically like.