There used to be more than one referee in the 1860s, each team had their own and no-one would abuse that because it was bad sportsmanship and the players were gentlemen.
You can probably guess how well that worked in reality.
Other than that, the linesmen/women and fourth official are assistant referees and will take over if the referee is injured. I think there is also a standby referee now, but there are pictures of Jimmy Hill being a linesman in a league match because he was the only qualified person in the crowd.
