NBA Retrodiction Contest Part 2: The Goods

In part 1 I tantalized you with the ability of different NBA productivity metrics to explain what happened.  This time we get to the important part, which is their ability to predict the next season.

No wasting time today; here are the results.

You can ignore 2000; since no one was in the database the previous season, everyone was given their actual production (i.e., there is no predicting involved).  The winner looks like ASPM with an average error of 2.24.  Win Shares is next at 2.37, then new RAPM at 2.62, new WP at 2.63, old WP at 2.74, PER at 3.07, old RAPM at 3.23, and APM at 4.24.  The one year of ezPM came in after ASPM, new RAPM, and Win Shares.  To put this in a bit more of perspective, we can convert these point differential errors to wins.  With a very simple prediction method, ASPM is off by 5.7 games on average.  Win Shares is only off by 6, so less than half a game across 9 years.  New RAPM, which uses the players’ previous rating as a prior, is back by about a game at 6.7, and virtually tied with new Wins Produced (although the averages are over different years; RAPM is better in three of the four years in the database).  Old WP comes in at just under 7 wins error then you have a jump to PER at 7.8, old RAPM at 8.2, and APM at 10.8.

So I should note right now, these results are different from what I got before.  RAPM was better than WP previously, and now it’s worse.  So now would be the time to double-check that database and let me know; otherwise I’m assuming that I was incorrect before.

Let’s start at the back.  APM and PER do not do a very good job of predicting future performance.  I don’t have the sense that anyone really uses these to evaluate players besides ESPN, so this isn’t really a big deal beyond the fact that the public will continue to have a poor idea of who the good players are.  PER may retain some value as a general measure of public perception, even though you will get highly rated non-superstars, but I think that’s about it.

Next you have Wins Produced and new RAPM.  Including a player’s previous season as a prior is helpful for RAPM, which is itself an improvement on APM.  So things are heading in the right direction.  However, you might hope that RAPM would do better given that it effectively has two previous seasons of information whereas the other metrics only have one.  And that’s being generous; 2011’s rating has 2010 as a prior, which has 2009 as a prior, which has 2008 as a prior, etc.  For players who have been in the league long enough, 2011’s RAPM rating is influenced by what happened back in 2002.

At the top of the heap (again, barring errors in the data) are ASPM and Win Shares.  These are probably the two most complicated metrics, but they’ve apparently earned it.  While Win Shares is widely available at basketball-reference.com and they have a whole page dedicated to describing how to calculate it, the actual equations are not present.  Nor have I ever actually seen the equations from Basketball on Paper replicated anywhere.  Similarly, I haven’t been able to get my hands on the formula for ASPM.  I think it was tracked at least somewhat on the APBR site, but it’s become difficult to find things there since the site crashed a little while back.  But ASPM uses ‘advanced’ box score metrics and is non-linear (i.e. includes interactions between predictors and/or squared terms and the like).  ezPM could be included in this group, since it uses play-by-play level data and would thus be pretty tough for your average person to ever compute on his own.  But it appears to be worth it; they all make predictions pretty well.

So you might think that’s the end of the story, but not quite!  All good stories come in trilogies.  Click here to check out part 3 (coming tomorrow).