I once encountered a mode when even the hardware reset signal did not reboot the pi.
A good solution I made as a part of a bigger in-house project is controlling power to Pi with a P-FET transistor, controlled by an ATmega (or so) microcontroller, connected to the Pi via UART (or even I2C or SPI if you need said UART). It runs a timer, reset by messages from Pi. If the timer is 40 seconds to timeout, it tells Pi that it is about to power down; if it is still alive, it can do a clean shutdown in that time. Then, power down, few seconds off, power up. There can be more than one watchdog, and it can be zeroed from cron (once per minute or two or so) and/or remotely via e.g. a remote HTTP request. That covers most of the scenarios when something goes wrong. (Avoid a continuously running daemon. Sometimes on small embedded systems kerbel gets into a pinch and keeps running existing processes but refuses spawning new ones. Make these things a part of the watchdog zeroing routine, so if they fail, restart happens.)
(Edit: you actually don't need a full-blown microcontroller. A rudimentary version of the same can be done with a pair of 555 chips, communicating via GPIOs - one for keepalives, one for the upcoming-reboot message. Be sure to feed the keepalive pulse through a capacitor, so if it gets stuck at the "OK" value its lack would still get through.)
I also have a thought, though not tested, how to repair a damaged SD card. Make an adapter with 74HCsomething (or a little CPLD), switching between two cards - one to the Pi, one to a card reader on its USB port. The main card is the one from which the Raspi runs, the second one is a service one that gets automatically switched in if booting from the main one fails (a job of the watchdog microcontroller). (And then either do an automated reimage of the main card, or send a message to the sysadmin who can do it manually, remotely.)
SD cards use SPI protocol, just a few wires, all of them unidirectional. I saw some vaguely similar hack on hackaday.com some time ago. 1-to-2 digital multiplexers/selectors with all the A/B selector inputs wired together could do the job. Another option is using something like 74HC244 or 74HC245 (8x bus driver), wire them in a way that one set connects main card to raspi and service one to reader, the other vice versa, and tristate the outputs of either of them as needed.