If it’s gnarly, toss it. They sell refills for a reason. If it’s mostly hair, you can just pull it off and set it to dry (perhaps outside?) and then stuff it next to the toilet bowl cleaner wand.
I picked a refill pack up and didn’t realize it needed the handle. I was happy to find that it doesn’t really need the handle!
