FWIW, fivethirtyeight actually publishes calibration plots from time to time. A calibration plot is a graph where the x axis is “Percent likelihood we assigned some occurrence,” and the y axis is “If you count up all the times we assigned the same likelihood, what percentage of them actually happened.” Like, if you give 10 events a 70% likelihood and 7 of them happen, you’re well calibrated, so if your calibration plot is just the line y=x it means you are accurately estimating your own level of knowledge and certainty. It doesn’t mean “no one could have done better and given more accurate predictions,” it means “to do better we need more data or a better model but we’re not making any obvious mistakes with the ones we’ve got.”
I really wish everyone who made public forecasts regularly had to publish plots like this. Anyone poorly calibrated you should ignore, or at least apply an appropriate correction to any likelihood numbers based on past data.
