With some awareness and responsibility of course, bridge abutments can be a problem too! 
As a long-time rider myself I’ll say that one’s trickier! At very slow speed differential I tend to stay close to the right as you say, but that’s when I’m slow enough that a hard brake or swerve is sufficient to avoid any idiots.
Then as the speed difference becomes greater I’ll start weaving to the left as I pass, then back to the right for the gap between. That maximizes visibility while maintaining the ability to avoid anybody that jumps out simply by drifting further left if needed, (shoulder conditions permitting of course). This also minimizes the blind-spot duration.
Then at higher speeds I just stick to the left. However, there is still a limit to how much faster I’m willing to go as there’s always the chance of somebody actually trying to make it all the way over to the shoulder which would require a last-second change-up to pass them on the inside.
