The regular season had the nice feature of ending on the 30th, so I can take the last day of the year to recap how the model picks went for 2012. If you need a refresher on what Luigi, Yoshi 1, and Yoshi 2 are, click that ‘NFL prediction model’ link in the banner to get caught up.
Let’s start with the over/under. Luigi was 103-129-8 after going 105-129-6 last year and 120-115-5 the year before. It would almost have been worth going against the picks the last two years. I would have guessed that perhaps the model has fallen behind the times with passing going up, but the predictions actually tended to be higher than the over/under on average. So some work needs to be done, but barring a turn-around it might be time to retire Luigi’s totals. Yoshi 1 did a bit better at 117-115-8, but that obviously isn’t getting anything done. Yoshi 2 was 113-119-8. Since all three models work on the same inputs, my guess is that some other variables might help out, or nothing at all.
There was one tied game this year, so Luigi’s record at picking outright winners was 153-86-1. That’s pretty much identical to last year’s 154-86 and ‘better’ than the 147-93 from two years ago. I say ‘better’ because you expect picks to be wrong any time there’s an upset, so you shouldn’t expect or want to be 100%. But 63% across three seasons seems pretty decent. Yoshi 1 was 148-91-1 and Yoshi 2 was 144-95-1. Less impressive were the moneyline picks. All of the models like to pick underdogs, so you expect them to not come through especially often, but that’s ok if the odds are high enough to make up for it. That was true for Luigi two years ago, but not this year or last. This year Luigi was 71-145 compared to 53-88 and 46-79 from previous seasons. Obviously Luigi made many more picks than previous seasons; that could be due to using a new source for numbers. Yoshi 1 was 78-149 and Yoshi 2 was 75-149.
The models decide which (if either) team to go with based on the odds being offered compared to how likely they think it is that the team will win. If they are doing that badly, it could be that the win percentage prediction is off. To check, I got all of Luigi’s regular season games from the past three years and combined them to make sure the sample size was as good as it could be. Then I looked at games where Luigi predicted the home team would win over 90% of the time, or between 80% and 90%, and so on. If Luigi is correctly calibrated, the home team should actually win over 90% of the time (given some noise, obviously), between 80% and 90% of the time, and so on. The home team won 75% of the time in the 90% predicted bracket; there were only 4 games and three of them came from week 2 games, when team information isn’t so great. Even so, hard to evaluate 4 games. The 80-90% group wins 92% of the time in 25 games, so that isn’t too bad. In 107 games, the 70-80% group wins 75% of the time, so that seems right on. Continuing down in 10% bins, the predicted winner wins 61% in 151 games, 58.5% in 176 games, 43% in 136 games, 27% in 78 games, 45% in 33 games, 70% in 10 games, and there are no games where the home team is predicted to have less than a 10% chance at winning. It looks like things might be getting out of whack at the bottom there, where low probability teams win more often than predicted (45% for 20-30 and 70% for 10-20). Some of that could be because a lot of these games come from week 2 predictions. If I take those games out, there are only 4 games left between 10 and 20%, and in the 22 games between 20 and 30% the home team wins 27% of the time, which seems fine. The 30 to 40% range might be more of a problem because they only win 27% of the time, but it’s close. So I don’t think Luigi has a fundamental flaw, but it may have had some bad luck in the range where bets happen to be made. I’ll still give it a look in the offseason.
I initially started this enterprise with the goal of making picks against the spread. To be honest, the variables were selected for this and then I just kept them for the winner and total predictions, which could be part of their problem. So how have the spread picks gone? I would never advise picking every game you could, but if you used Luigi to do so you would have gone 102-129-9. Yoshi 1 was decent in his debut, going 124-107-9, and Yoshi 2 went 105-126-9. In the past two seasons in all games Luigi was 246-222-12, so this was definitely an off year. Yoshi 1 would have produced around a 2% return; I’ll have to see if that’s plausible across multiple seasons but I would kind of doubt it when picking all games.
The main way I try to improve accuracy is to only go with games where the model is more certain. Certainty is described by bigger differences between the spread and the predicted score difference. One level of certainty is to throw out any picks where the spread is within a point on either side of the predicted difference. A second level is to go with only the top five games in a week like in the Hilton SuperContest. Using that first level, Luigi picked 185 games instead of 240 this season, and was 85-92-8 in them. Obviously not good, but it is less bad than what happened in the every-game case. In the previous two years Luigi had a record of 197-176-9 in these kinds of games, so again signs that this was a down year. Yoshi 1 would have made choices in 193 games and gone 100-86-7, which is only the tiniest bit better than the full slate, and Yoshi 2 would have made choices in 189 games and gone 85-96-8, which is a little better (less worse) than the full slate. So the idea worked, but could only do so much in a poor year.
In the Hilton SuperContest I use their lines, but I missed one or two while I was traveling. I used my usual SBR Forum consensus lines to pick my top five games and the record in those situations. On the year Luigi and Yoshi 2 went 40-35-5 and Yoshi 1 went 41-37-2. If you give them a 3-2 record for week 1, that puts them each at about 45 points, which would be just below the 46 points they got two years ago and below the 50 points they got last year. So the top five picks each week still did pretty well. Luigi has a three-year record of 134-107-9, which is about 55% or about a 5.6% return. That seems pretty solid.
My other spread comparison is against Bill Simmons. In the past two years combined, Bill went 253-241-18 and this year he went 125-111-4 through week 16, 7-9 in week 17 by my count, and thus 132-120-4 on the whole season. I didn’t count any predictions for week 1 because the models can’t start making picks with no data, but from week 2 through 17 Luigi went 111-125-4 against Bill’s lines. In the previous two years it was 265-215-16, so Bill basically made up two years of picks on me this year alone. How quickly the tables can turn. Yoshi 1 went 121-115-4 and Yoshi 2 went 107-129-4. So Yoshi 1 kept things close, but Bill took it this year.
Overall this appears to just have been a bad year. But Yoshi 1 seems promising, and I have some new ideas to try out anyway. So over the course of the next year I’m hoping to try them here and see if I can do better, or at the very least learn something along the way. That’s one resolution for 2013. Happy New Year everyone!