It isn’t. Adverbs modify verbs, not the sentences they are contained in.
I don’t have to read, “I quickly walked down the street” quickly.
If I say, “I literally waltzed down the street,” would you insist that this is true only if I was actually dancing in 3/4 time as I moved along the street? Or that it was true only if I was actually dancing in 3/4 time while moving along a street straight towards the centre of the Earth?
And if Tom Sawyer were to say “I literally waltzed down the street” would we say that cannot be true because he is a fictional character and thus could not have done so in actual real life? No, the entire sentence is contained with the fiction and describes something happening within the fiction. Something that literally happened fictionally.
In that sentence literally modifies “waltzing”, but probably not “down” or “street” (which is taken to be literal without any modifier), and certainly not “I” (who may, literally, not exist).
If I described someone arriving in a hospital with a hole through their foot made by stepping on a sharp object, I could say, “It looked as if he literally shot himself in the foot”. Here “literally” modifies “shot himself in the foot” to indicate it is a description of the actual act of shooting oneself in an appendage rather than the figure of speech. However, I don’t mean that he literally did that, the entire clause is figurative, as indicated by “as if”.
Literally can be used figuratively, it can be used literally while speaking figuratively or fictionally. It can be used to modify a single word or a whole phrase but not to modify the entire sentence it is in (without reference to the sentence).
