Holding people accountable for predictions is one of my ever-so-exciting hobbies, and that’s particularly true when it comes to the topic of win totals. That extends to myself as well as many of the other unfortunate souls who foolishly attempt to predict the future.
Many listeners will recall the prediction podcast from all the way back in March, when John, Tom, Ben, and myself argued about how many games each team would win for an hour. I then recorded our predictions and compared them to FanGraphs and Baseball Prospectus projections. Now it’s time for the reckoning.
Last week, I evaluated a range of win total predictions and projections over at Banished to the Pen. I’ll use the same mean absolute error (MAE) and root mean squared error (RMSE) measurements to evaluate the BFN predictions. RMSE punishes the bigger misses more because the error is squared, so if you really miss by a massive amount on a team (*cough* Orioles), it’s going to be even worse than in the simple MAE, which is simply the average error across all the predictions.
So, how did we get on? Here are the BFN win predictions as compared to the actual team totals, followed by the ranks in MAE and RMSE compared to the other 7 sets from that BttP review:
| Team | Actual | BFN Diff |
|---|---|---|
| BOS | 108 | -17 |
| HOU | 103 | -4 |
| NYY | 100 | -7 |
| OAK | 97 | -23 |
| MIL | 96 | -12 |
| CHC | 95 | 1 |
| LAD | 92 | 10 |
| COL | 91 | -6 |
| CLE | 91 | 6 |
| TBR | 90 | -15 |
| ATL | 90 | -17 |
| SEA | 89 | -13 |
| STL | 88 | -6 |
| PIT | 82 | -10 |
| ARI | 82 | 4 |
| WAS | 82 | 9 |
| PHI | 80 | 0 |
| LAA | 80 | 7 |
| MIN | 78 | 8 |
| NYM | 77 | 8 |
| TOR | 73 | 9 |
| SFG | 73 | 12 |
| CIN | 67 | 4 |
| TEX | 67 | 8 |
| SDP | 66 | 6 |
| DET | 64 | -1 |
| MIA | 63 | -1 |
| CHW | 62 | 3 |
| KCR | 58 | 10 |
| BAL | 47 | 26 |
MAE: 8.77 (8th)
RMSE: 10.46 (6th)
Eighth in MAE is not great. That’s another way of saying last. It was only barely last, right behind the Banished to the Pen preview writers, so we weren’t really any worse than another group of humans making guesses about these things.
Sixth in RMSE is better, though. We even beat a projection system! Davenport had a worse RMSE, and the BttP writers also finished behind us. I am reliably informed by BP’s Rob Mains that RMSE is considered the more suitable measure of error, so I will declare that this is more relevant. We weren’t even all that far behind FanGraphs, missing out on beating them by just under a tenth of a win. Beating PECOTA was much tougher, as the BP system was the overall winner in 2018 by a clear margin.
As you can see from the table, the AL gave us the most trouble with its rather extreme results, most notably the Rays, Red Sox, A’s, and particularly those Orioles. I blame John and his unreasonably optimistic view of Baltimore for that, and clearly the A’s were Ben’s fault. In all seriousness, even I only wanted to make the A’s a .500 team, and the Orioles were as historically bad as the Red Sox were good, so it shouldn’t come as any surprise that both teams confounded most predictions and projections. The Braves winning the NL East was another big miss, one that was again universal among the competitors.
We were exactly right about the Phillies after a late season slump, and pretty much correct about the Cubs and how bad the Tigers, Marlins, and White Sox would be. We also successfully (and inadvertently) predicted that the NL would snap the AL’s 14-year interleague winning streak, which was a bolder call than any of those individual team predictions, no matter how accidental.
What about the order in which the teams finished? Did we get closer to the final rankings?
| Team | Rank | BFN Rank Diff |
|---|---|---|
| BOS | 1 | -5 |
| HOU | 2 | 0 |
| NYY | 3 | -2 |
| OAK | 4 | -17 |
| MIL | 5 | -9 |
| CHC | 6 | 2 |
| LAD | 7 | 6 |
| CLE | 8 | 5 |
| COL | 8 | -3 |
| TBR | 10 | -9 |
| ATL | 10 | -12 |
| SEA | 12 | -6 |
| STL | 13 | -2 |
| PIT | 14 | -10 |
| WAS | 14 | 8 |
| ARI | 14 | 5 |
| LAA | 17 | 9 |
| PHI | 17 | 0 |
| MIN | 19 | 10 |
| NYM | 20 | 9 |
| TOR | 21 | 6 |
| SFG | 21 | 10 |
| TEX | 23 | 4 |
| CIN | 23 | -3 |
| SDP | 25 | 1 |
| DET | 26 | -3 |
| MIA | 27 | -3 |
| CHW | 28 | 0 |
| KCR | 29 | 2 |
| BAL | 30 | 8 |
MAE: 5.63 (6th)
RMSE: 6.92 (6th)
These differences are naturally smaller because there are only 30 possible outcomes, so we can’t have those huge misses like the Orioles – the maximum error is 29. We beat the BttP predictions again, and also the FanGraphs projections. By this measure, the A’s were clearly the most surprising team.
Did we get any individual divisions right? Sadly not. The AL Central was closest in terms of the way the division played out, with Cleveland way ahead, the Twins stuck in no man’s land, and three terrible teams, but we had the Tigers last, not the Royals. The NL West looked like it might have a chance for a while, until the Giants collapsed and the Diamondbacks faded late.
So we weren’t too terrible at this. We stayed in the mix with some projection systems and performed much better than simply giving every team 81 wins, which is the minimum bar you really have to clear – if you can’t manage that, it’s really time to give up. Let’s do it all again next year.