The intent of HTML was that it would be semantic markup, which the browser could render in whatever way made sense for that reader. That would let it adapt to varying screen sizes, and indeed to voice synth and braille displays and other radically different ways of presenting the context.
Instead, people insisted on using it as a visual markup tool.
CSS, and XHTML with XSLT, were attempts to escape that trap and bring the web back to separate semantic content and presentation style, on a mix-and-match basis. That, too, has largely failed.
So… Yes, you’re absolutely right. The page author shouldn’t be setting the font size; that should be under your control. But every time we push in that direction, the folks who insist that they want visitors to see THEIR design push back.