As we get ready to revisit Steamer’s 2012 playing time projections, I thought I should take a look at which systems had the most success projecting playing time last year. Many thanks to Rudy Gamble of Razzball and Mike Spiher of Rotochamp who made this possible by supplying the data.
For all of the comparisons below, I adjusted each system’s projections so that the average projection of the system was the same as the average actual playing time of the group so that the overly optimistic system’s weren’t penalized for wearing rose-colored glasses.
RMSE in 2011 plate appearance projections:
The first two columns show the root mean square error in projections for the set of players who were projected by all of the systems and the column labelled “Starters” is limited further down to those players projected to have 400+ plate appearances by Fantasy411 (which is an average of 14 systems). I added this criteria in order to look at the group of players (230 of them) that fantasy players really care about.
The third and fourth columns include players who were only projected by some of the systems. Here systems without a projection for a given player were assigned a forecast of 0 PA for that player. As before, the Starters column only includes players expected to receive more than 400 PA by Fantasy411.
The Community forecasts excelled across the board, only getting narrowly edged out by Rotochamp among Starters who were projected by all of the systems. I can’t really brag about Steamer’s success because these projections were done right before the start of the season with the terrific benefit of being able to make comparisons with all of the other forecasts and tweak accordingly. We could have simply used the Community forecasts but we thought we could do even better. We didn’t.
I’ve highlighted two systems, “Fantistics 03/05″ and “Fangraphs 03/05″, because these projections came from earlier in the off season — March 5th to be exact. The Fangraph fan projections take a big hit when the study is opened up to all players since many players didn’t have Fangraph projections at this point, however, if we limit our focus to those who were projected by all of the systems, the fans look very impressive. Looking at Fantistics improvement between the 5th and 22nd of March, I suspect that Fangraph fan projections from closer to the start of the season may be comparable in accuracy to the Community projections.
Would it be possible to improve on the Community projections?
One possibility I explored was whether these projections would be improved by factoring in injury expert Will Carroll’s Traffic Lights. The following table shows the number of days spent on the disabled list in 2011 by players who were given green, yellow and red lights in his Team Health Reports:
Green players were certainly healthier than red players but as it turns out the Community knew better than to project high playing to to injury risks:
The players who were given the highest playing time projections from the fans are ALL green and yellow. Fans properly adjusted hitter projections for injury risk. Similarly, while older players got to the plate less, this wasn’t lost on the fans whose projetions could not be improved by factoring in age.
Projecting Innings Pitched
Is there’s no such thing as a durable reliever?
Let’s start by looking at days on the disabled list by traffic light.
Now, let’s see which systems had success projecting reliever innings.
Notes: Here “relievers” are defined as players that Steamer expected to pitch in relief prior to the season since I didn’t want to give these systems the benefit of hindsight. The player pool is limited to those players who received a health report.
That’s right, the Traffic Lights did a better job of forecasting relief innings than any of the projection systems. Put another way, none of the fantasy experts did very well. Most of them did roughly as well as “Mean”, a system that projects the same number of innings for every relief pitcher in this group. Why is this?
Notes: In the interest of fairness (fairness in terms of degrees of freedom, that is) I insisted that the Traffic Lights make the same deduction, 7 innings as it turns out, from green to yellow as from yellow to red. The Lights project 58 IP for a green pitcher, 51 for a yellow and 44 for a red. Also, I put an asterisk next to the Rotochamp line because they were unduly hurt by missing projections for a number of these players.
Let’s look at Community projected IP and actual IP while grouping pitchers by projected IP rounded to the nearest 10 innings:
The relievers who were expected to throw the most innings fell well short. This was true for relievers who were projected to be elite (based on Steamer ERA projections) and true for relievers given expected to be healthy (green lights). It begs the question: Are there any relievers who should be projected to throw more than about 60 IP? Here’s my challenge to anyone reading this: make a list of the 20 pitchers that you expect to throw the most relief innings in 2012 and post it in the comments section. Whoever gets the most innings wins.
If the Community had made such a list prior to 2011, they would have projected an average of 77 IP from a group that would go on to average 55 (25%ile: 33, 50%ile: 56, 75%ile: 66). Only three players in this group of 20 exceeded their IP projection. One was Alexi Ogando who was moved into the rotation and did what he could for the group average. The others were Craig Kimbrel and Jonny Venters. The Community top 10 and top 40 underachieved by similar amounts and this shortfall wasn’t unique to the Community projections, check other systems and you’ll see roughly the same thing. Forecasters overestimated their ability to determine which relievers would throw the most innings.
Starting Pitchers
Here the systems did much better and the Community did the best. All of the systems did much better than than the Traffic Lights alone and better than a simple system based on Traffic Lights and an ERA projection. The later Fantistics projections did much better than the early Fantistics projections and the early Fangraph Fan projections didn’t project enough pitchers to compete. It looks like there’s more to learn about starting pitchers than hitters during spring training. I looked for ways to improve upon the Community projections and, initially, thought that I’d found something when I saw this:
The relationship between the Community forecast’s residuals and Steamer’s projected ERA was certainly statistically significant (a two-tailed p-value of of .0045) and significant in size (knock off 20 innings for each point of ERA) but the relationship wasn’t robust when I fiddled with the player pool and it looks like this relationship could be explained by a non-linear relationship between projected IP and actual IP. I’ll need to revisit this with more data in hand.
There was also too little data to say for certain whether pitcher playing time forecasts could be improved by using either Will Carroll’s traffic lights or Jeff Zimmerman’s DL projections. The Traffic Lights had the more compelling statistical case and the larger effect size (add 8 innings for a green light and subtract 8 innings for a red light) and I suspect that both systems carry some information that the fans and forecasters aren’t fully incorporating.
So, there’s more work to be done here but I think I’ve seen enough to adjust Steamer’s 2012 playing time projections. We should have new downloads with updated playing time projections along with a handful of new players (including Yu Darvish!) ready late tonight or tomorrow.
