Bat Flips and Nerds Versus the Projections: The Results

Holding people accountable for predictions is one of my ever-so-exciting hobbies, and that’s particularly true when it comes to the topic of win totals. That extends to myself as well as many of the other unfortunate souls who foolishly attempt to predict the future.

Many listeners will recall the prediction podcast from all the way back in March, when John, Tom, Ben, and myself argued about how many games each team would win for an hour. I then recorded our predictions and compared them to FanGraphs and Baseball Prospectus projections. Now it’s time for the reckoning.

Last week, I evaluated a range of win total predictions and projections over at Banished to the Pen. I’ll use the same mean absolute error (MAE) and root mean squared error (RMSE) measurements to evaluate the BFN predictions. RMSE punishes the bigger misses more because the error is squared, so if you really miss by a massive amount on a team (*cough* Orioles), it’s going to be even worse than in the simple MAE, which is simply the average error across all the predictions.

So, how did we get on? Here are the BFN win predictions as compared to the actual team totals, followed by the ranks in MAE and RMSE compared to the other 7 sets from that BttP review:

Team Actual BFN Diff
BOS 108 -17
HOU 103 -4
NYY 100 -7
OAK 97 -23
MIL 96 -12
CHC 95 1
LAD 92 10
COL 91 -6
CLE 91 6
TBR 90 -15
ATL 90 -17
SEA 89 -13
STL 88 -6
PIT 82 -10
ARI 82 4
WAS 82 9
PHI 80 0
LAA 80 7
MIN 78 8
NYM 77 8
TOR 73 9
SFG 73 12
CIN 67 4
TEX 67 8
SDP 66 6
DET 64 -1
MIA 63 -1
CHW 62 3
KCR 58 10
BAL 47 26

MAE: 8.77 (8th)

RMSE: 10.46 (6th)

Eighth in MAE is not great. That’s another way of saying last. It was only barely last, right behind the Banished to the Pen preview writers, so we weren’t really any worse than another group of humans making guesses about these things.

Sixth in RMSE is better, though. We even beat a projection system! Davenport had a worse RMSE, and the BttP writers also finished behind us. I am reliably informed by BP’s Rob Mains that RMSE is considered the more suitable measure of error, so I will declare that this is more relevant. We weren’t even all that far behind FanGraphs, missing out on beating them by just under a tenth of a win. Beating PECOTA was much tougher, as the BP system was the overall winner in 2018 by a clear margin.

As you can see from the table, the AL gave us the most trouble with its rather extreme results, most notably the Rays, Red Sox, A’s, and particularly those Orioles. I blame John and his unreasonably optimistic view of Baltimore for that, and clearly the A’s were Ben’s fault. In all seriousness, even I only wanted to make the A’s a .500 team, and the Orioles were as historically bad as the Red Sox were good, so it shouldn’t come as any surprise that both teams confounded most predictions and projections. The Braves winning the NL East was another big miss, one that was again universal among the competitors.

We were exactly right about the Phillies after a late season slump, and pretty much correct about the Cubs and how bad the Tigers, Marlins, and White Sox would be. We also successfully (and inadvertently) predicted that the NL would snap the AL’s 14-year interleague winning streak, which was a bolder call than any of those individual team predictions, no matter how accidental.

What about the order in which the teams finished? Did we get closer to the final rankings?

Team Rank BFN Rank Diff
BOS 1 -5
HOU 2 0
NYY 3 -2
OAK 4 -17
MIL 5 -9
CHC 6 2
LAD 7 6
CLE 8 5
COL 8 -3
TBR 10 -9
ATL 10 -12
SEA 12 -6
STL 13 -2
PIT 14 -10
WAS 14 8
ARI 14 5
LAA 17 9
PHI 17 0
MIN 19 10
NYM 20 9
TOR 21 6
SFG 21 10
TEX 23 4
CIN 23 -3
SDP 25 1
DET 26 -3
MIA 27 -3
CHW 28 0
KCR 29 2
BAL 30 8

MAE: 5.63 (6th)

RMSE: 6.92 (6th)

These differences are naturally smaller because there are only 30 possible outcomes, so we can’t have those huge misses like the Orioles – the maximum error is 29. We beat the BttP predictions again, and also the FanGraphs projections. By this measure, the A’s were clearly the most surprising team.

Did we get any individual divisions right? Sadly not. The AL Central was closest in terms of the way the division played out, with Cleveland way ahead, the Twins stuck in no man’s land, and three terrible teams, but we had the Tigers last, not the Royals. The NL West looked like it might have a chance for a while, until the Giants collapsed and the Diamondbacks faded late.

So we weren’t too terrible at this. We stayed in the mix with some projection systems and performed much better than simply giving every team 81 wins, which is the minimum bar you really have to clear – if you can’t manage that, it’s really time to give up. Let’s do it all again next year.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.