Machine learning does this too, and is also very bad at breaking out of those local maximas.
You see this when you’re working on something, and you start it going. Gen1 is meh. Gen 5 shows great progress, and gets you a good distance towards your goal. Gen 40 is looking a lot better. Gen 80 has you almost convinced you’re about to win some computing prize… and then Gen 81 - Gen 14,000 basically don’t get you any closer to solving your goal. Gen 19,000 is a regression a little bit, and your results were in the early Gen 100s.
So you tear open the model inside Gen 100 only to understand that you don’t have a clue what it’s doing. You didn’t really make this thing, you have no idea why it’s chosen what it’s chosen. It’s good enough for what it does, but not good enough for more than that and there’s no real path forward.
