The WoW article weaves around a bit, but I think I can sum up the plus/minus part pretty succinctly: they think it does a bad job of telling you which players did well. Not all of their points are right on-target, but the critical ones (with a tiny bit of my own expansion) are that: a) with plus/minus you can’t tell which player was specifically responsible for the team doing well in that time period; b) if you move to adjusted plus/minus to attempt to correct for that, you run into huge issues with sample size and colinearity; and c) the final results don’t seem to do a very good job of describing player quality.
Now, it’s important to note at this point that I don’t think many basketball stats guys would disagree with those three points. Taking my usual reading habits (ESPN, WoW, APBR, a couple other things here and there) as representative, plus/minus and adjusted plus/minus simply aren’t used a lot. When they are used, it’s almost always with a caveat, only to get a sense of defense (which isn’t necessarily captured well by the box score), or with other all-in-one metrics to get a more rounded view of a player. A previous post of mine fleshes out some of the issues with APM and includes a number of links to other people describing and/or discussing it if you’re a curious reader.
So how about Tango? Tango says that he likes APM for hockey (there isn’t enough scoring) but loves it for basketball. As an aside, I imagine that if you ran your hockey APM regression using one of a logistic/mulitnomial/Poisson link, it would probably clean things up a bit. But you would still have terrible colinearity issues, so I don’t think it would be a big help. Moving on, plus/minus has a great intuitive hook in that it’s a with you/without you comparison; you typically run it with scoring but you could look at rebounding, fielding in baseball, whatever. If a team has more of a good thing (or less of a bad thing) when a certain player is playing than when he isn’t playing, that’s a sign that the guy is doing something good. Then it just becomes an issue of trying to tease out all the confounding info – the other guys on the field, who comes in to replace that player when he’s out, who the opponent has playing when he’s in versus out, and so on. That’s why Tango says that you live for sample size: you need enough circumstances where a guy was both in and out with enough other stuff going on (various teammates, various opponents, etc) that you get a good sense of any given situation.
I think both camps here are right and there’s just a bit of miscommunication. I wholeheartedly agree with Tango’s title: plus/minus is a GREAT concept. And I also agree that if you have a big enough sample, it would probably do just fine. The link he has to his catcher study sounds great. The issue (and where I agree with the WoW article) is that, at least in the NBA, we rarely have the sample sizes needed to get APM to behave well. And if you gather enough data to get it to behave well, now the interpretation of the number has changed quite a bit. You can’t run a regression and scale things a bit to say “Kobe Bryant was worth 2 points a game to the Lakers in 2010” with any confidence, but you might be able to say “Kobe Bryant was worth 2 points a game to the Lakers on average from 2005 to 2010”. And then how useful is that? If you wanted to know who the best player was over, say, a five year period, then I would tell you to fire up your APM calculations and have at it. But that is rarely a question worth asking.
I want to know how good a player is now, or is likely to be next year, so that I can sign him to my team (whether I’m a real GM or a fantasy GM). Even more helpful would be knowing why a player is good so that I can tell if he’ll mesh well with my system and the other guys on my team (if I’m a real GM; thank god that isn’t a fantasy issue). APM doesn’t tell me that at all, although I suppose I could run regressions for any facet of the game I could think of (adjusted rebounding, adjusted assists, adjusted shooting, etc) and try to use that to create a picture of what a player contributes. But the numbers APM gives me in any of those circumstances are going to be poor representations of the truth. They won’t be useless; lots of stats out there have some value. Player points per game tells you something about winning. But there are better stats out there than APM, at least when basketball is concerned.