The Signal and The Noise: When Predictions Change the Future

One of the interesting discussions in The Signal and The Noise is about when the act of making a prediction can change the prediction’s outcome.  The background to that point is modeling the spread of diseases and the cool stuff that people do to do that; you (can) essentially model individual people and their incomes, habits, beliefs about vaccines and whatnot, and you let them wander around the ‘world’.  The goal is to see how diseases are transmitted, how quickly, where they’ll spread (in cities, out to rural areas, in certain neighborhoods versus other ones, etc), and what factors will impact the spread.

Aside from models that try to predict the actions of individual people, let’s say you’re the CDC.  You perform some kind of forecast and decide there’s going to be a nasty flu season this year.  You start putting out commercials telling people to get the flu shot, you tell doctors to encourage their patients to get it, schools put up signs telling people to cover their coughs, all that good stuff.  It turns out that winter ends and the flu wasn’t so bad.  Was your prediction wrong?  Or did people see your prediction and act on it, causing your assumptions to be wrong and changing the outcome?

This kind of discussion leads quickly into the realm of game theory and social interactions.  What do I do when I know that you know what I know?  Let’s go back to the CDC example.  You put out the forecast and people act on it, making your forecast wrong.  That’s fine for now, because fewer people became sick, which is a good thing.  But what if it means that next year people ignore your forecast because they think you’re bad at it?  It could end up as a ‘boy who cried wolf’/Cassandra effect situation.  But you can’t stop making forecasts; you’re the CDC and it’s your job to help keep people healthy.  It’s an interesting conundrum.

I was reminded of this part of the book when reading an article by Chuck Klosterman over at Grantland.  Chuck wants to know if the statistical prediction that teams should go for it on 4th down more often than they currently do will destroy the strategy of going for it on 4th down.  He notes that the prediction is largely based on averages of what has happened.  But going for it on 4th down happens fairly rarely and so is very contextualized: the conversion attempts that have been analyzed come, essentially, only from losing teams facing prevent defenses and from good teams that think they can pull it off.  Chuck is concerned that if everyone starts following the advancednflstats calculator that the calculator won’t work anymore, and that stats people will start advising teams to punt a little more often.

It’s an interesting question, so I’m going to address a couple aspects of it.  Let’s start with the game theory part first because I think it’s a non-factor, or definitely on the side of going for it more often.  If it’s fourth down and you decide to punt, the other team is off the hook.  They can relax (barring a fake punt attempt) and stop thinking about how to stop you because you’ve stopped yourself.  If you go for it, the defense has to decide how to stop you.  But knowing that you’re going to go for it doesn’t change anything from how it currently stands; when a team now or in past years has lined up to go for it, the defense has known that they’re going for it (barring the draw-them-offside tactic).  So if teams go for it more often, the decision of how to stop them will be the same as it is now (ignoring that a defense in the lead may be willing to trade first downs for time to a certain extent; I’ll get to that situation later).  However, going for it on 4th down has more of an impact on earlier downs.  Currently, if it’s third and long you can expect that the offense will either throw a long pass or run a draw/screen.  That’s because if they don’t get the first down the offense is going to punt.  But, if the offense is going to use all four downs the majority of the time, that no longer has to be the case.  I can throw a five or six yard pass to set up fourth and manageable; I can call a sweep or some other run that I think will get some yardage.  Or I can still throw it deep.  Basically, I can run all sorts of stuff, anything that I might run on first down.  In short, if teams start going for it on fourth down more often it will only make things easier for offenses because they will have more options on earlier downs.

So how about addressing Chuck’s direct question, that the conversion rates we currently have might be too high?  I don’t have the play by play data to look at who and when these attempts are coming from exactly, but I do know that the percentage of attempts has been increasing a bit in recent years (I can’t quickly find a direct link on advanced NFL stats, but there’s an article saying it was on the rise back in 2008, and I have to imagine it’s still true now).  And we can at least make some guesses as to when teams are going for it.  One situation is the one where teams have always gone for it, which is late in the fourth quarter when they’re losing.  Teams that are losing, at least on a given day, are likely to be poor offensive teams or are playing a good defense; otherwise they wouldn’t be losing.  That implies that conversion rates are underestimates because we’re missing what would happen when good teams go for it more often.  However, it could be that these losing teams are facing more prevent defenses that are willing to give up yards and time.  I’m not sure that’s a big concern for two reasons: one, you always want to stop a team on fourth down; you’re much more likely to win when you have the ball and the lead than when the other team has the ball.  Two, teams should only really be using the prevent when they’re up by more than a touchdown (and thus aren’t in danger of losing), whereas teams will be going on fourth down as long as they have some chance of winning, which should be pretty much anything from 16 points down given enough time on the clock.  So fourth down conversions shouldn’t be overly contaminated by prevent defenses, I don’t think.

If that was all we had to worry about, we would be done.  This is probably essentially true for the NFL up to 2000 and maybe later.  The Romer study came out in 2005, so that might be a plausible first point at which ‘smart’ teams would start going for it more often on fourth down.  Even with these additional attempts coming from presumably well-run teams, coaches would still be conservative due to the outside pressures involved.  Thus you might expect that those attempts would come from good teams in favorable situations.  But they wouldn’t be super-favorable; teams aren’t going to go for it if they’re up by 14.  So at least for the game being played, the teams are likely to be relatively closely matched.  So I think these new attempts are probably going to be more likely to succeed than the ones described above, since they’ll come from better quality teams in general, but they won’t be hugely different.  As time moves on and going for it becomes more common, you’ll add in more situations where average teams go for it under competitive situations, meaning you’ll add in attempts that are more like league average.

So to sum up, the NFL probably had most fourth down conversions consist of relatively low-probability tries for most of its history.  The initial additions to the list would have been better, but probably not better than what league average would be if everyone were going for it.  And as going for it becomes more acceptable, we’re going to keep approaching league average.  So my intuition is that over the years we’ve seen an increase in not only fourth down conversion attempts, but also fourth down conversion percentage.  If this is true, then Chuck can feel reassured that the advice coming from stat-heads isn’t going to ruin their prediction.  But he can also probably feel confident that no matter what, fans and writers are still going to question any attempt that happens to fail.

This entry was posted in Uncategorized and tagged , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s