A quick post to keep things moving along during the holiday. Brian Burke at Advanced NFL Stats has recently been trumpeting success rate as a useful measure that he’s used for a while without knowing quite how useful it is. For example, it may be more predictive of future success than per-play efficiency, which he (and I) have been using with some success. Given its apparent consistency, I was curious if this is finally the measure we’ve been looking for that would give us a measure of performance that we can use from year to year. As readers of the Wages of Wins should know, quarterbacks are not very consistent from year to year. The linked article says that 70% of an NBA player’s win score can be predicted by their win score the previous season. That is, the R square should be .7 and thus the correlation around .83. In contrast, the WoW QB score values are only 15% and .39. So quarterbacks appear to vary wildly from one season to the next. The numbers are similar if you use the stats created by Football Outsiders. What about the Advanced NFL Stats stats?
I got the quarterback data from Brian’s site for quarterbacks that played the past five years (2005 to 2009). I aligned the data so that one column has the quarterback’s statistics from year n and the next column has the same quarterback’s stats from year n-1. Thus I eliminated any seasons that were not consecutive, and obviously any quarterback who only played one year. I believe that Brian limits the data to only show quarterbacks who attempted a certain number of throws; the smallest number of attempts in the sample is 144. Finally, I limited the stats of interest to win probability added (WPA) per game, expected points added (EPA) per play, and success rate (SR%). I did this so that quarterbacks could be compared regardless of the number of passes thrown or games played in a season. Then I simply correlated each measure in year n with the same measure for the same quarterback in year n-1 (I also threw them into a scatterplot in Excel and fit a linear trend so I could get the equation/R squared). The final sample is 102 season pairs from about 50 quarterbacks.
WPA/game had the lowest predictive power, with a correlation of .29 for an R squared of only .085. Thus you can only explain 8.5% of a quarterback’s WPA/game with their performance from the previous season. The equation is .241*(WPA last year) + .0459. Moving to EPA things get a little better; the correlation is .496 for an R squared of .246, so we’re up to 24.6% explained. The equation is .403*(EPA last year) + .0517. And finally, success rate does indeed prove more predictive. It has a correlation of .558 for an R squared of .311. So of all the measures I’m aware of, success rate proves to be the most predictive for future quarterback performance, and it explains 31.1% of what a quarterback is doing, or less than half of what win score does for NBA players.
I think there are a few important things to note here. First, as Dave Berri said,
“What does this mean? In a given season we see some quarterbacks are good, others are not so good. But the next season we can look at these same quarterbacks and see a very different ordering. Why does this happen? We have argued that how a quarterback’s teammates performs impact the quarterback’s value. In other words, quarterbacks may make their teammates better, or their teammates may make the quarterback better. The stats, though, can’t tell us which story is correct.
Consequently, the player statistics tracked in the NFL do not serve the purpose statistics serve in basketball and baseball. It is not clear that a rigorous analysis of the player statistics in the NFL would ever allow us to separate the player from the team.”
It’s possible that some of Brian’s (or other people’s) stats are more predictive from year to year for positions other than quarterback. But I doubt they would be too much better, for the same reason as the quarterbacks: performance depends too much on teammates (I would argue, mostly the offensive line). This is still true even with success rate; a correlation of .558 is fairly impressive, but we still aren’t describing a lot of what’s going on.
Second, this inconsistency isn’t to say that you can’t tell which quarterbacks are good and which are bad (although it certainly muddies the water). Peyton Manning has been above average in this sample every year, as has Drew Brees. What it means is that you can’t tell how good your good quarterback will be, or how bad your bad quarterback will be. And if your quarterback isn’t Peyton or Drew, then it is very possible that you’ll have no idea if he’ll be good or bad next year. Looking at WPA, Eli Manning has been pretty good, average, bad, pretty good, and great. Kurt Warner was ok, bad, ok, great, then ok. And of course Brett Favre is known for going back and forth from year to year; he’s been terrible, average, great, bad, and great.
Third, since we’re talking about WPA, this is further evidence for the non-existence of clutch performance. WPA is a measure of how much a player has affected his team’s chances of winning. It is particularly susceptible to high-leverage moments: in a close game a team will have a win probability of around 50% (by definition), and a touchdown to seal the game or an interception that gives it away will move WPA by a large amount (toward 100% or 0%). Since WPA/game appears to jump around from season to season, it’s hard to claim that any quarterback (at least in this sample, but of course I would say ever) is ‘clutch’ as a personality characteristic.
Finally, this doesn’t mean that predicting games is impossible. What it means is that it is probably next to useless to predict one season from the previous one. When player performance varies so much, it’s hard to say who will be good next year. And that is why hope springs eternal, apparently more in the NFL than the NBA.