Update: Here’s Nate Silver’s thoughtful analysis of accuracy of his projections from last week. Nate also points out that the errors aren’t necessary normally or even symmetrically distributed as I assumed in the analysis below.
On his fivethirtyeight blog, Nate Silver has made 55 projections for 2012 Republican Primaries, each one including a projected mean and a 90% confidence interval.
Earlier in the primary season, Tom Tango pointed out that Nate was too accurate, meaning that his error bars had been wider than needed to that point.
Now that there’s more data, I’ll revisit the question: Are Nate’s confidence intervals too wide? Too narrow?
It turns out that 50 of his 55 projections (darn close to 90%) are within his 90% confidence intervals. Of course, the errors are correlated. For instance, if you were too bullish on Ron Paul in Virginia it implies that you were too pessimistic about Mitt Romney. To get a better sense of the distribution of Nate Silver’s misses I followed Tango’s lead and calculated a z-score for each of his predictions. Here’s a histogram showing the distribution of the z-scores:
This looks fairly reasonable. You can see the two largest misses from Massachusetts and Virginia at the extremes.
So, how accurate is Nate Silver? Quite possibly, just as accurate as he says he is.
Download the data set used in the above analysis:
Nate Silver 2012 Primary Forecasts (208).