You're making unwarranted assumptions.
My experience is actually "biased" towards people who routinely use psychedelics recreationally. Not "a few times" but hundreds of times over the course of decades.
From my personal experience, prolonged regular use of weed doesn't make you at least slightly crazier than you were before. In my case, it seems to make me less crazy. My emotional responses to stimuli seem much more appropriate to me when I'm under the influence of marijuana than when I'm not. Then again, marijuana intoxication is usually too mild to qualify as what I'd call a "psychedelic" although some strains are potent enough to trigger psychedelic experiences even in people with a bit of a tolerance.
I would happily grant that sustained acute use of psychedelics (dropping every day for a week, for example) can cause some degree of dissociation from "reality" (however you want to define that). However, I suspect occasional use of these substances tend to make people "less crazy" (again depending on your definition).
I'm also happy to grant that you experienced what you claim to have experienced and that those experiences are entirely valid. Can you grant as much to me?