Hate to say it, but I have to go with the “it’s what we’re used to”.
If it was really field-of-vision, we’d see more “landscape” windows. They certainly exist… but “portrait” format is as common to more common, and while there may be historical reasons driving that nobody gets bent out of shape about it. Similarly, nobody objects to portrait-format stills when they’re composed with that orientation in mind.
No, I really think this is the result of multiple factors.
Peripheral vision is certainly one of those, as anyone who remembers REAL widescreen (Cinemascope et all) can attest. IF the screen is large enough, a wider view gives a greater sense of presence. Omnimax demonstrates that a similar effect can be achieved by extending the image farther vertically… but I do think there’s a legitimate bias toward watching for action at, or not far above, our own level. There aren’t many predators for a creature our size who come from directly under or above us.
But another is imprinting – we’ve been exposed to landscape-mode video far more often, and as importantly we have developed a movie/video vocabulary based on that format. In fact, stages have always tended more toward landscape mode, so our whole dramatic structure has probably been pushed in that direction. (Though multilevel staging does exist, as do effects that push the action upward or downward.)
I think the letterboxing issue is overrated. Some displays and mounts are built so that they can be rotated into portrait mode and have the rendering immediately adjust, but that was hard to do with CRTs and even on LCDs most folks are not going to be interested enough to make the effort. (Mine does, which should be a huge win for text editing… but I rarely bother; the resolution’s high enough and the screen’s large enough that I don’t mind wasting some pixels). But… Frankly most folks watch amateur video in a small window anyway. Resolution clearly is not an issue for that audience. Nor is waste of screen real estate. To me, those arguments sound like audiophiles boasting about the quality of their amps and speakers while playing MP3s and cassette tapes; it may be true, but if you think you can tell the difference between that and an average-quality rendering you’re kidding yourself.
No, I think the real answer is that most vertical videos suck because most videos suck, period. (Sturgeon’s Law: 90% of anything is crap.) Folks may have a tiny bit more of a clue about how to make horizontal videos not suck – or folks who make horizontal videos may have about that much more clue about how to make videos not suck – but I strongly suspect that the viewer’s expectations dominate over any other effect.
Of course that’s just opinion. Odds are that someone has done actual research on this. If not, that sounds like a thesis waiting to be written, kids…