Predicting the Past: 2011 competition

Earlier in the season I took a look at how well a couple of statistical models (ezPM and Wins Produced) could predict team results.  This isn’t impressive on its face; everyone makes predictions.  But what I wanted to do was put them on even ground to more fairly evaluate which one does better.  When people make predictions, they make all sorts of assumptions beyond the quality of the players: how many minutes each player will get, who might get injured, who might get traded, how rookies will do, and so on.  My goal was to eliminate as many of the assumptions as possible.  You can read the articles to see what I did, but I’ll walk through it below as well.  Now that the season is over, I can retrodict the full results.  I also got a hold of some other numbers so I can compare more stats.  Let’s get to it.

The stats involved are Wins Produced, provided by Dave Berri; ezPM, provided by Evan Z; and RAPM, provided by Jeremias Engelmann.  I have WP numbers for every player since 1978.  Evan has ezPM calculated for the past three seasons, but has told me that the 2009 data is probably not correct so I’m not going to use it.  Finally, I have RAPM for 2006 through 2010.  I am not using the current season’s data because the numbers available on the site include the playoffs, which the other sets do not.  That isn’t too important because as you’ll see, this season’s player data would only be used to predict how teams do next year, and we aren’t doing that because next year hasn’t happened yet.  I was also going to include standard APM, but the data I downloaded from basketballvalue doesn’t match up to what’s posted on the site to the best I can tell, and I haven’t heard back from Aaron about it yet.  If you’d like to take a look, a few people have posted similar retrodictions for the past year at the APBR site.  They’re using different assumptions than I will be, but you can see numbers from Daniel (DSMok1), Evan, Jeremias’ 1 and 3 year RAPM, and Aaron’s APM.

So how does this work?  For every player this year I found the last season they played in the NBA.  For a lot of guys, this is 2010.  I took their per-minute productivity (for WP) or per-possession productivity (for ezPM and RAPM) and assumed that they would have the same productivity this year.  If a player played for multiple teams last season, their productivity is the minute/possession-weighted average across teams; this doesn’t apply to RAPM where players get a single number for the whole season.  If a player played in the past but not last season (like Von Wafer below), they get their productivity from that season if available.  If a player didn’t play before, such as a rookie or someone who isn’t in a dataset for some reason, they get an assumed level of production.  For WP that is .045 WP48; -1.95 ezPM100 as was used in my other post; and -3.9 points per 200 possessions for RAPM.  I picked that number because ezPM and RAPM are both point per-possession measures and thus giving them the same rookie/unknown production assumption seems fair.

Now I have a per-minute/possession productivity measure for every player this year.  One other thing happens for WP; as the only model that cares about position, I assume that we know the position that each player will get this year.  So WP48 for 2011 is calculated by taking 2010’s ADJ P48, adding 2011’s WP48, and subtracting 2011’s ADJ P48.  Now that everyone has their per-minute/possession productivity, I calculate total productivity by multiplying by 2011’s actual minutes/possessions played.  Why?  Because it takes out any assumptions about what rotation a team will use or who will get injured.  Every model is on equal ground.  All the players on a team have their total production summed up and thus we have predicted team wins (for WP) or predicted team point differential (for ezPM and RAPM).  I’ll use the typical wins-to-points equation that says that a per-game difference of one point is worth 2.54 wins so that I can convert between the two and get predictions for both wins and differential for each model.

Let’s walk through the Boston Celtics as an example.  The Celtics managed to have 21 players suit up this season.  Ray Allen was one.  Last year he also played for the Celtics, and only the Celtics.  He had a .126 WP48/.258 ADJ P48, a .3 ezPM100, and a 3.9 RAPM.  Each of those numbers are carried forward as per-minute/possession productivity predictions for 2011, with the exception that WP adjusts his position, ‘knowing’ that he’ll play at 2 instead of 2.06.  So his predictions for 2011 are .132, .3, and 3.9.  Ray played 5410 possessions in 2890 minutes, so we multiply each of those number appropriately and find that WP predicts that Ray would contribute 7.95 wins, ezPM predicts he would contribute 16 points, and RAPM predicts he would contribute 105.5 points.  Another Celtic was Luke Harangody.  Since he was a rookie, he gets .045 WP48, -1.95 ezPM, and -3.9 RAPM.  In the 462 possessions over 241 minutes played for Boston, WP predicts he would produce .23 wins while ezPM and RAPM predict he would produce -9 points.  As a final example, we can look at Von Wafer.  He hadn’t played since 2009, so I don’t have an ezPM estimate for him.  Also, since there’s no ezPM, I don’t have a count of possessions played.  So his WP estimate is based on 2009 and predicts a WP48 of .068 and .78 wins produced.  But he’s treated as a rookie for ezPM and RAPM and thus gets -1.95 points per 100 possession and -20.4 points produced.

If I add up all the predicted wins produced by the Celtics players, I get 48.9 wins.  This can be converted to a per-game point differential of 3.09 by subtracting off 41 (the average number of wins) and dividing by 2.54.  Similarly, I add up the predicted points produced by ezPM and get 364.8, which is a per-game differential of 4.45.  This converts to 52.3 wins.  RAPM produces a 1.34 differential and 44.4 wins.  Boston actually won 56 games and 440/82 = 5.37 point differential.  The measure of error I’ll use is absolute deviation.  In terms of wins, everyone was below how Boston actually did, but the error would always be a positive number even if someone predicted over.  WP has an error of 7.1, ezPM 3.7, and RAPM 11.6.  In terms of point differential, WP has an error of 2.28, ezPM .92, and RAPM 4.03.  So when it comes to Boston, ezPM produced the best predictions in terms of both wins and point differential.  We can look at the predictions to see why this happened.  Ray Allen, Paul Pierce, and Kevin Garnett did better than expected for WP to the tune of about 3 wins each, and Marquis Daniels was not as terrible as expected.  I can’t evaluate RAPM since I didn’t use their numbers for this season, but according to RAPM the Celtics only had 4 players of any note in the positive range (Rondo, Pierce, Garnett, and Allen).  Finally, ezPM was fairly on-target, but also underestimated Allen and Daniels.  He got most of the improvement by being close on Garnett and Rondo (actually slightly overestimating both).

Of course, we don’t want to look at just one team to draw conclusions.  I went through the same exercise for each team this past season.  Here’s the tally:  the mean absolute error for wins for WP is 8.3.  The biggest misses were Philly (mostly due to Elton Brand’s resurgence) and Minnesota (everyone not named Kevin Love was terrible) followed by a group of Miami, Golden State, and Phoenix.  For point differential the error was 3.32, which pretty much matches up with the win error.  For ezPM, the error for wins was 7.13 and the error for point differential was 2.78.  The biggest error was for Miami and then similar-sized errors for Philly, Minnesota, Washington, and the Lakers.  And finally for RAPM the average error for wins was 7.8 and the error for point differential was 2.88.  RAPM was most wrong about Chicago, Cleveland, Minnesota, and the Spurs.

To summarize, ezPM did best followed by RAPM and WP.  They were all within 1.3 wins or half a point of differential, however, so it’s pretty tight.  The most surprising team was Minnesota, which all three systems were pretty wrong about.  I think it’s interesting that RAPM was wrong about different teams than ezPM and WP though; the two box score methods had a bit more agreement.

Next post, possibly coming later today: 2010.