What drives me nuts is when someone presses the button and then crosses without waiting for the light to actually change. Either leave the button alone or ignore the light, don’t make it change and hold up traffic for no reason after you’ve crossed.
Personally, I don’t take orders from street lights, and I don’t interrupt traffic flow unless it’s real busy and a natural gap is not forthcoming. The only two rules of pedestrianism: 1) Don’t endanger anyone, including yourself, and 2) Don’t inconvenience anyone, except yourself.