I think you’ve hit the nail on the head, of course.
Accusations of witchcraft were rarely about specific extraordinary misfortunes, they were about quotidian regular misfortunes, plus existing social and/or interpersonal stresses and resentments.
If Sven blames Magnus for spoiling his beer with witchcraft, but Magnus is popular and easy-going, and Sven is known to have a beef with him over that cow that strayed that time, then everyone is likely to say “yeah, Sven, and it was trollies who hid your tunic after the goat got into your clothes chest.” But if Sven was to accuse Gunner, who nobody really likes, and keeps to himself, and is a mean drunk, and has been looking (pleased with himself/especially worried about something) recently, and a whole bunch of things have been going wrong around the place as well, and there’s worry about political tensions in the kingdom, and there are mutterings of “unnatural practices” going around anyway, then Sven’s accusations might be met with “you know, I never trusted that bastard, but he couldn’t have done it alone: we have to find out if there was anyone working with him.”, and things go to shit.