Retrodiction Contest Update Part 1: The Method

In my last couple posts, I looked at the ability of a few NBA player productivity metrics to predict seasons that have already occurred based on data from the prior season: using 2010 to predict 2011, 2009 to predict 2010, and so on.  However, things have been very unsatisfying due to the limits of the publicly available data.  I decided to lower my standards a bit in order to get more usable numbers and run things again.  I also made a few changes, so in this post I’m going to describe the method and data as thoroughly as possible.

I already noted that the APM data you can download from basketballvalue.com seem to disagree with the numbers posted on the website.  Take Kurt Thomas, for example.  According to the website, his one-year APM for the past few regular seasons (starting at 2011 and going backwards) has been -3.77, -4.44, 3.19, and 2.67.  The data I downloaded, however, have him at -2.14, -2.33, 4.02, and 3.21.  So I initially skipped on using APM since I didn’t know what to trust.  However, the APM data set also includes possessions played.  I need that to calculate productivity for ezPM and RAPM.  ezPM comes with possessions, of course, but there are only three seasons of it.  RAPM comes with nothing beyond the ratings, but there are six seasons of it.  Note that I downloaded the numbers before Jeremias changed them on the website, but 2011’s ratings included the playoffs even then.  Since none of the other metrics use playoff data, I am not using RAPM in 2011.  The Wins Produced numbers come from Dave Berri.  So in the end I have the following: WP for a wide range of years, RAPM for 2006-2010, ezPM for 2010-2011, and APM for 2008-2011.

Once I decided to use the APM data, I was able to get possession data, which means I can look at more seasons of RAPM as well.  However, I noticed that the possession data from the APM files disagreed with the numbers from ezPM.  I gave preference to the APM numbers simply because there are more seasons of it.  APM lists both offensive and defensive possessions, so I averaged them.  Then I wanted to see how well possessions and minutes played lined up so that I could try to handle players who were on multiple teams within a season (which WP and ezPM give different values to but APM and RAPM do not).  It turns out that within the APM dataset, possessions correlates at .999 with minutes.  So every player is given 1.927 possessions per minute played, with the minutes coming from my WP data file (which agreed well with the APM minutes played).  Using the regression equation, I calculated possessions played for every player in the entire dataset.

There were two other changes I made.  First, Evan suggested in the comments that when RAPM says 200 possessions, it really means 100: 100 on offense and 100 on defense.  I’m happy for the clarification because I always found that confusing.  So now my RAPM calculations assume a productivity value as per 100 possessions.  The other change I made was to the assumption of rookie performance.  I used .045 for WP48 and -1.95 for ezPM/RAPM based on information from when I did this earlier in the season, but I never really checked to see if this was equivalent.  A .045 WP48 player will produce, obviously, .045 wins in 48 minutes of play.  But an average player will have created .1 wins.  With the conversion factors of 1 minute=1.927 possessions, 1 point per game = 2.54 wins in a season, and 82 games in a season, the difference in WP becomes -1.92 points per 100 possessions.  This is not a great change from the number I used before (-1.95), but I shifted just to be more consistent.  That will be my assumed level of performance for rookies/players not in the data set for ezPM, RAPM, and APM.  Some players don’t get enough playing time and are lumped together into the reference set for APM; they get an APM value of ‘NULL’ in the data.  They are generally considered replacement level and so I will give them an assumed APM value of -3.8, or twice as bad as an average rookie.

The other change was to go through by hand and tried to correct any mistakes I could find in the dataset; this was limited to players who were given a default value when I lined up the datasets due to mismatches in name (Ronald Murray versus Flip Murray and the like).  It also happened a lot in the RAPM 2006 file where suddenly first names were rarely used and I got to guess which J.Jackson was who.  So the numbers I post may be different from what appeared in my earlier posts, but they should be more accurate.

As an example of the method, I’ll walk through the 2011 retrodiction for the Detroit Pistons.  I’m going to report two retrodiction values for each metric: one that assumes rookies have the average rookie production of -1.92 pp100/.045 WP48, and one that grants rookies their actual 2011 production.  I think the first method is more fair, but the second numbers may also reveal something interesting.  To start, here are the numbers for the Pistons in 2011:

You’ll note that the RAPM numbers are all 0; recall that I’m not using them.  That won’t matter for the calculation where all rookies are given the same productivity, but it means RAPM (in 2011 only) will be at a disadvantage for the other calculation where rookies are given their actual productivity.  The next part is to find the per-minute productivity for each of these players the last time they played.  Everyone played in 2010 except Greg Monroe, who was a rookie.  For simplicity here I’ll list his productivity as what it was in 2011.  Also note that Tracey McGrady played for two teams in 2010, the Knicks and Rockets.  His ezPM and WP would be combined as a minute-weighted average to make the prediction for 2011.

Now we can go ahead and calculate the predicted production for each player.  To do that, I use their actual minutes played in 2011.  Again, that’s to eliminate any of the guessing games about rotations or injuries (or in the case of the Pistons, odd benchings).  For ezPM, RAPM, and APM I calculate the expected points produced over the course of the season by multiplying the 2010 per-100 possession productivity by the 2011 possessions played and dividing by 100.  Will Bynum, for example, had a 2010 RAPM of -2.7 and played 2168 possessions in 2011, so he is expected to produce about -58 points over the course of the season.  Wins Produced is a little different in that the ADJP48 from 2010 is adjusted by 2011’s position (so WP gets to know the position played by each player in 2011) and it spits out wins produced instead of points.  Will Bynum has a predicted WP48 of .045 and is predicted to produce just over 1 win.  Here are the predictions for each player:

Note that those numbers are the ‘actual production’ rookie predictions, so RAPM has Monroe at 0.  Under the -1.92 average rookie assumption he would have a prediction of -82.2 points for ezPM, APM, and RAPM and 2.08 wins for WP.  The final step is to sum them up and convert them to equivalent team-level predictions.  I decided to look at per-game point differential.  The procedure for WP is to sum up wins produced by each player, which comes to 31.74 with actual Monroe and 25.63 with average rookie Monroe (the Pistons actually won 30 games last year).  Team wins are converted to per-game point differential by inverting the usual regression equation; that means I subtract off 41 wins (for the average team) and divide by 2.54.  This gives a WP prediction of -3.65 for actual rookie and -6.05 for average rookie.  The Pistons actually had a point differential of -3.6, so the actual does better, which is to be expected since Monroe performed better than average (unless you believe APM) and played a fair amount.

For the per-possession metrics, I add up the points produced and divide by 82 games to get per-game differential, and that’s it.  For ezPM the actual and average rookie predictions are -1.95 and -4.27; RAPM predicts -3.61 and -16.6 (but remember it has Monroe at 0 in the first case since I’m not using any 2011 RAPM values); and APM predicts -10.98 and 12.67.  In the end I’ll be looking at mean absolute error, so for the actual rookie predictions I would have values of .05, 1.65, .01, and 7.38 for WP, ezPM, RAPM, and APM and average rookie values of 2.45, .87, 13, and 16.2.  As you would expect, the metrics do better in general when they know Monroe’s actual production.

In part 2: the results.

Advertisement