Nope. It’s a mixture of cultural and biological factors which leads to geophagy in humans. There’s interesting scientific literature about this. Fun fact: eating clay-rich soil can be detrimental, even, and despite the contrary popular belief deprive your body of minerals by chelatisation and surface-exchange processes (clay minerals being ion exchangers).
There are also some studies which have shown, albeit with relatively small sample size (if you ask me), that eaters are more likely to catch parasites by the practice of geophagy if the product is not processed.
In East Africa, you can buy the stuff over the counter. Children and pregnant women are culturally inclined to eat it. In Europe, medical clay (often, in fact, silt) can be bought in drug stores. I guess the US situation might be similar.
Animals, on the other hand, vary extremely in the nature and purpose of geophagy by taxon. Salt licks are naturally to take up minerals, so can be geophagy of, e.g. elephants. However, clay-rich soil is sometimes documented to be taken up as potential remedy against plant secondary compounds, like alkaloids. Same chemical story here: surface binding and chelatisation are likely to help digestion of toxic or irritating plant material.
Some of this might be conjectures from early naturalists surviving in scientific writing - but then, those naturalists often are dead right if you have a closer look and a controlled experiment. It’s definitely more than a myth.
Back on topic: this, dear friends, is why science - and especially botany - matters. You can kill yourself with plants, and it’s usually not a nice death.
Oh, and did anyone mention that Aloe vera itself is poisonous, and that you should not eat the green parts of the leaf? I didn’t even read the whole threat yet. 
scrolls
Ah. Aloin mentioned. Don’t forget the whole bunch of anthraquinones.
