InterSkeption: Sport Skeptic Skeptical about Skeptical Sports

Ben Morris at 538 (and formerly at Skeptical Sports) had an article yesterday looking at a few topics in the world of football.  I thought two of them could use a little work, so this is me complaining.  The first has to do with how much a win or loss at different points of the season affects a team’s playoff chances and the second is the bit on rookie quarterbacks.

The graph of the week is pretty straightforward and also a nice visualization; it shows the probability of a team making the playoffs given a certain record at any point of the season.  For example, every team that has won 11 games (whether they were 11-0, 11-1, 11-2, etc, at that point) has made the playoffs.  Teams that are 1-0 go on to make the playoffs 54% of the time, while teams that are 0-1 go on to make the playoffs only 25% of the time.  That observation leads to the next graph, which shows the leverage of any game by asking how much the probability changes depending on if you get a win or a loss.  At the beginning of the season, teams are 0-0.  But after the first game they can be 1-0 or 0-1 (ignoring ties), and those records are associated with a 29% difference in making the playoffs (the 54% mentioned before minus the 25% mentioned before).  Ben calls that 29% the leverage of that game.  You can similarly map out the other differences (a 3-0 team moving to 4-0 or 3-1, and 5-5 team moving to 6-5 or 5-6, and so on) and create his second graph.

Ben’s takeaway from the second graph is that the initial games of the season are surprisingly important.  You might think, hey, it’s only week 1, but every team just experienced a 29% leverage game.  He says “Early games are so rich with information value that this is basically the most informative period of the entire season.”  This is sort of true, but what jumped out at me is the swath of yellow on the diagonal; the high-leverage games aren’t all in the bottom corner for early games, but happen over the course of the season for teams with near-.500 records.  In fact, the highest-leverage game you can find in the whole chart is for a 8-7 team, at 41%; the second-highest is for 4-4 teams, then 9-6.

There are two reasons for this.  First, subtracting probabilities isn’t really a good idea because they don’t add normally; they’re bounded at 0 and 1 (which I managed to talk about four years ago, which makes me feel old).  For example, a 8-3 team simply can’t have a leverage of 30 or 40 points because they’re already at 92% to make the playoffs.  If they win they go to 97% but if they lose they’re still at 82%.  But speaking in probabilities, that move from 92% to 97% is very important; the closer you are to 100%, the harder it is (generally speaking) to keep closing that distance.  Consider a 10-4 team.  If they win, they hit that 11 win number and are, historically speaking, guaranteed a playoff spot.  If they lose, they still have a 97% chance of making it.  That’s obviously very good, but it isn’t guaranteed.  So while Ben’s leverage number says the game is only worth 3%, to the team it should be worth much more.  Similarly, at the low end of things, teams with poor records simply can’t move their probability that much.  If you win to go from 3-10 to 4-10, you’re still dead.  But in the middle range of records, there’s much more room to improve your chances.

The other reason is related to that last point, but a bit different.  The games that occur on the diagonal of that chart are informative, but the kind of information changes over the course of the season.  As the first point made clear, wins are simply important for making the playoffs towards the end of the season.  If we assume that 10 wins pretty much gets you in the playoffs, then at the end of the season you’re just trying to bank wins to get to that mark or over it.  It doesn’t matter if you’re actually good or have just been lucky; you’re just banking wins, and for teams near .500, that one-win difference can mean a lot.  So late in the season, a win is telling us directly who will make the playoffs.  Early in the season, the information is about team quality.  Better teams are probably going to win games, and so a 1-0 team (or 2-0 or 3-1) is probably better than a 0-1 team (or 0-2 or 1-3), and better teams are more likely to keep winning and make the playoffs.

But there isn’t a bright line where “team quality” shifts to “get 10 wins”; the season is short enough that every win is important (the teams that won last week are 10% of the way to a playoff berth) and every game is informative as to team quality (a 9-7 team is probably a bit better than an 8-8 team).  So it isn’t so much that the early games are the most informative in the season; it’s any game involving a 50/50 team, who might be good or might not, and who might need that one win later to get on the good side of 8-8 to make the playoffs.  We know that teams with more losses are bad and won’t make it, and we know that teams with more wins are good and probably will.  Middling teams are less certain, and uncertainty leads to more informative outcomes.

The other analysis that I took some issue with was Ben’s regression looking at future success for rookie quarterbacks.  He ran a regression predicting approximate career value (AV) after the rookie year from games started, yards per game, touchdown and interception rates, and completion percentage during the rookie season.  He found that games started was the most important predictor, and says that if you want to assess a rookie QB that’s what you should look at.  I thought this analysis was a mess.

First, let’s look at AV.  You start with points per drive for the whole offense, but then the O-line takes nearly half the credit.  QBs get some credit for running, but presumably it isn’t a large contribution since QBs aren’t a big part of team rushing yardage.  Passing credit is fixed from team to team (26% of the team offense after taking out O-line and rushing).  Finally, individual QBs will get their share of AV relative to how many of the team’s yards they throw for, with some small bonus for effective throwing (having adjusted yards above or below league average).

So, just to spell that out:  AV is a ‘counting stat’, not a ‘rate stat’.  Guys who play a lot will have more AV simply by playing a lot.  For a quarterback, AV will depend mostly on a) how many points a team scores and b) if that QB threw for all of a team’s yards.  And that’s it.  He can add some value by running, and he can add or lose some value by keeping his yards per pass and touchdowns up and interceptions down, but generally speaking all a QB needs to do is be the only QB to play.  Plus, AV is nearly always positive; there have only been 30 times where a QB had a negative AV in a season since the merger, and none of them started more than 10 games in that year.

So when Ben predicts AV using games started (more games started means you are the guy responsible for most of the team passing yards), TD percent (gets you a slight bonus via adjusted yards), interception percent (slight penalty via adjusted yards), yards per game (a rough indicator of a decent offense), and completion percentage (can only affect AV indirectly through adjusted yards and supporting a high-scoring offense), of course games started is vastly important.  Even if a QB leads a below-average offense, so long as he’s the only guy throwing passes, he’s going to get credit for that offense.  And since AV is hardly ever negative, he’s going to be getting positive credit.  More playing time, more AV.

Now let’s think about the ‘future AV’ part; Ben doesn’t predict rookie year AV but the rookie QB’s future AV.  How might a rookie get more value in the future?  By either a) being good (or theoretically being good) or b) the team not having a better option.  In either case the QB is going to get to play more now and in the future.  How would a rookie QB not get future value? By not really being part of the team’s plans and thus not playing, or by playing so poorly that they move to someone else.  In either of those cases, the QB is not going to play/start many games as a rookie.  Basically, if a team plays a rookie QB much at all as a rookie, he’s going to get more chances.  If he gets more chances, he gets more future value.  Neither of those means that he’s objectively good; it just means that the team thinks he might be good or that they can’t do any better.

If you want more of a feel for the disconnect between rookie starts and quality, just think of some of the big QB busts (here’s a list to get you started).  JaMarcus Russell started 25 games.  Joey Harrington played for six seasons.  Blaine Gabbert started every game he played in for three seasons.  If you look at first-round QB picks since the merger, only 7 failed to start at least 10 games in their career; all of them had at least two years in the league.  The list of guys who got to start any number of games as a rookie and didn’t get more chances is very short.

So we have two issues here.  First, AV is really a measure of playing football as opposed to being good at football.  Another example from last year: Nick Foles was the 15th QB by AV, and also happens to be the first one to play fewer than 15 games (every QB ahead of him actually started all 16 except for Tony Romo).  It’s tough to play a lot of games and not get more AV; the only QBs with more games but fewer AV than Foles last year were Flacco, Glennon, Eli, Glennon, and Geno Smith.  Foles was buoyed by the Eagles’ excellent overall offense, even though they ran more than they passed and Foles only threw about 60% of the team’s passes.  Second, the predictors that Ben uses are probably colinear; QBs with good numbers (more TDs, fewer interceptions, more yards per game) will generally start more.  So the values his regression spits out adjust for that the best they can, but can’t do it completely.  That’s why he ends up with a negative weight for completion percentage; why would you ever predict that a guy who completes a lot of passes would be worth less in the future?  The regression grabs on to what explains the most variance separate from the other predictors, and obviously ‘games started’ will have a lot of other factors (like the lack of QB competition, or things like ‘intangibles’ if you’re being generous).

Ben’s overall conclusion is that if you want to know who’s likely to add value in the future, you should be looking at Derek Carr and not so much at Teddy Bridgewater or Johnny Manziel (or Blake Bortles).  That makes a little sense in that, like I already mentioned, rookies who get to start are likely to get more opportunities to start, and thus add value.  But I don’t think there’s much direct evidence at all that Carr is actually a better QB than Bridgewater or Manziel; it just so happens that Minnesota and Cleveland had other options and were willing to let their guys learn from the bench.  Most things I read had Bridgewater and/or Manziel ahead of Carr as a draft prospect, and Bortles got the most hype in his preseason play.  I think what you can learn from Carr starting already is that he’ll likely get to start again in the future, but that doesn’t mean he’s the best rookie QB.

This entry was posted in Uncategorized and tagged . Bookmark the permalink.

4 Responses to InterSkeption: Sport Skeptic Skeptical about Skeptical Sports

  1. Pingback: NFL Week 3 Predictions | Sport Skeptic

  2. Pingback: NFL Week 4 Picks | Sport Skeptic

  3. Did I miss this before? Good article. I agree that predicting a rookie’s career prospects is very different from predicting their quality. But that was kind of my point—at least one of them, the other was that worrying about rookie QB efficiency is silly. The regression was just an illustration (of both points). Prospects are a function of skill and circumstances, and, like many things predictive, often turn on things that don’t always make sense.

    • Alex says:

      Re-reading the article now, I see that you were probably talking more about career prospects than quality per se. I feel like those get mixed together a fair amount though (do bad QBs get to play for lots of seasons? It should be rare), so that’s a tough call. Taking a guess on what I was focusing on back when I wrote this, it’s the claim that games started is really the most important thing when deciding the end career worth of a player. I think that’s due mostly to the choice of AV as your measurement, and that impact carries over to the values on everything else in the regression. I mean, does it make any sense at all that a better completion percentage is a negative predictor of career value? Is there any non-post hoc way to explain that? I could believe some other things, like maybe an inverse-U shape to downgrade poor accuracy as well as overly-safe throwers, but I have a hard time with a general conclusion that completing more passes is bad.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s