Writer’s observe: “5 Issues I Favored (Or Didn’t Like) This Week” is taking a brief break, however will return subsequent Friday for the top of the common season.
Earlier this week, I did the sabermetric equal of consuming my greens by testing the accuracy of our playoff odds projections. I discovered that our odds do a fairly good job of beating season-to-date odds (significantly late) and pure randomness (significantly early, every thing does fairly nicely late). It’s good to intermittently examine in on the accuracy of our predictions. It’s additionally useful to construct a baseline as a benchmark to measure future adjustments or updates in opposition to.
These are a bunch of strong, workmanlike causes to write down a measured, prolonged article. However boring! Who likes veggies? I wish to beat the percentages, and I wish to flex slightly mathematical muscle whereas doing it. So I goofed round with a pc program and tried to seek out methods to recombine our current numbers to provide you with improved odds constructed by slicing up current ones. It didn’t break the sport large open or something, however I’m going to speak about my makes an attempt anyway, as a result of it’s September 19, there aren’t many playoff races occurring, and you’ll solely write so many articles about whether or not the Mets will collapse or if Cal Raleigh will hit 60 dingers.
What if you happen to simply penalized excessive values?
I first tried to right for the truth that early-season projection-based odds (which I’m calling FanGraphs mode for the remainder of the article) appear to be too assured and thus vulnerable to giant misses. I did so by making use of a imply reversion issue that pulled each crew’s values towards the league-wide common playoff possibilities (i.e. what number of groups made the playoffs that yr). This methodology varies based mostly on the present playoff format; now we have 16-team, 12-team, and 10-team samples within the information, and I adjusted every appropriately. I set the imply reversion issue in order that it was robust early within the yr and decayed to zero by the top of the season.
This did nothing, principally. Extra particularly, making use of a variable reversion issue by month didn’t enhance the Brier rating of both FanGraphs mode or season-to-date mode odds. I additionally tried Winsorizing the percentages, or in different phrases making use of a cap and flooring to odds and recasting any quantity above the cap or under the ground to their respective ranges. This did nothing. I first tried a set cap and flooring, then had the pc set Brier-minimizing caps and flooring for every month. Neither labored; in reality, after I requested the pc for the “optimum” Winsorization issue, it returned, “Don’t give me a cap or flooring in any respect” for each month besides April, and used 2%/98% bands in April, or in different phrases principally nothing.
Fairly frankly, I didn’t anticipate this to work. Squeezing every thing in doesn’t enhance accuracy; it makes some predictions higher however others worse. You could do some form of focused squeezing to make a distinction, however the Winsor check confirmed that the optimum cap and flooring was, kind of, no cap and flooring. Oh nicely.
What if you happen to blended them?
Effective, squeezing the values doesn’t do something. What about setting a month-by-month mix of FanGraphs mode odds and season-to-date mode odds and utilizing that to take the most effective of every mannequin when it’s at its most helpful. To do that, I informed the pc to calculate the Brier-minimizing weight for every month by wanting on the information and discovering the combo of odds that produced the bottom Brier rating in that pattern. I additionally informed it to not cheat – or, extra particularly, I informed it that it wasn’t allowed to take a look at future information when guessing annually’s mix. When evaluating 2019, for instance, it set the weights based mostly on the 2014-2018 seasons.
This labored fairly nicely, because it seems. The Brier-minimizing weights begin with barely extra FanGraphs than season-to-date, then ramp up because the yr wears on. I attempted each an everyday model (calculate every month and use that worth with out adjustment) and a smoothed model that forces a smoother change in mix because the yr goes on. Each of them did fairly nicely, significantly in March/April and Might. They decreased the imply squared error by 5% relative to FanGraphs odds in March/April and by 3% in Might. By the top of the yr, although, issues get bizarre: The un-smoothed model makes use of 100% FanGraphs odds. Listed here are these weights, which labored out to a 2% lower in imply squared error general:
Blended Weights, FanGraphs Mode vs. Season-To-Date Mode
| Month | Un-Smoothed Weights | Smoothed Weights |
|---|---|---|
| March/April | 0.539 | 0.633 |
| Might | 0.619 | 0.664 |
| June | 0.734 | 0.700 |
| July | 0.700 | 0.681 |
| August | 1.000 | 0.780 |
| September/October | 0.996 | 0.764 |
Every reported worth is the fraction of full odds taken from FanGraphs odds. Knowledge skilled from 2014-2024.
If you’d like a helpful rule of thumb that may do ever so barely higher than the uncooked FanGraphs odds, you may consider beginning with a 60% FanGraphs 40% season-to-date mix, then growing it towards 100% FanGraphs because the yr winds on. The features are fairly small, as a result of it’s simply inherently tough to have small Brier scores with a lot uncertainty left within the season, however small features are the secret on this article.
What if you happen to use Bayesian inference?
We love Bayes right here at FanGraphs. Bayesian inference means adjusting your prior expectation based mostly on proof, and adjusting it by totally different quantities relying on what the proof says. Broadly talking, it asks how seemingly it’s that we’d see the noticed outcome (say, a crew taking part in .500 baseball for a month) given our prior expectation (let’s say we had them as a .560 crew), and adjusts our new expectation utilizing that proof.
This one isn’t fairly as simple as, “Use two-thirds FanGraphs odds early.” The rule varies its utilization of weights based mostly on what number of video games have been performed — the White Sox have been 1-0 this yr, that doesn’t imply we have been means off in our estimation of their talent — and the way huge the disagreement is. If our odds say a crew had a .530 profitable share in expectation and so they’ve performed to a .525 profitable share via two months, my confidence in our projections will increase. In the event that they’ve performed to a .510 profitable share, I’m barely extra skeptical. In the event that they’ve performed to a .630 profitable share, uh, possibly we received one thing improper. I additionally needed to consider the remainder of the league’s play, as a result of in case your crew begins proper as anticipated however different groups don’t, that may change our opinion of you as nicely. I added some phrases to make sure that every thing added as much as 100% in spite of everything of that, as a result of that, too, is vital.
In any case, I had the pc do it, and once more I informed the pc to not cheat. (Facet observe: It’s slightly extra sophisticated than that in observe, however making certain that your experimental design doesn’t cheat is half of the problem of getting good solutions out of fashions.) The weights and values used to make a Bayesian prediction have been chosen based mostly on backwards-looking information. Then, I measured the outcomes out-of-sample. In different phrases, my analysis used 2014-2017 information to provide adjusted odds for the 2018 season, then measured these odds’ accuracy. Then it used 2014-2018 information to provide adjusted odds for the 2019 season, and so forth. The best means to consider that is that we give an enormous worth to our current mannequin, however when it’s confirmed very improper early, we defer barely to the true world telling us what it thinks.
The Bayesian methodology did extremely nicely at the start of the yr. It’s higher than any of our fashions, higher than any mix of our fashions even. That’s as a result of it will probably “select” how a lot to hearken to the FanGraphs odds based mostly on how carefully they’re hewing to what’s occurring on the sector. That’s much less helpful later within the season, after all; as we’ve seen, FanGraphs odds do higher than odds based mostly on season-to-date play by the point half the yr is within the books. In March and April, although, our assumptions usually tend to be improper, so a little bit of Bayesian reasoning helps.
Once I informed the pc to interrupt issues up month-wise and provide you with Bayesian weights, it did OK, lowering imply squared error by roughly one p.c for the season as a complete. That’s disappointing, although, and it’s unnecessarily poor, as a result of the Bayesian methodology does worse than the FanGraphs-only methodology by the point August rolls round. It’s making an attempt its hardest, but it surely’s a easy rule that doesn’t know something concerning the mannequin efficiency. All it cares about is divergence between projected profitable share and season-to-date file, and as we all know, that issues fairly a bit much less by yr’s finish.
To repair that, I hacked issues up. Is that this good science? I’m undecided. However by telling my pc program that it ought to use a Bayesian methodology for the primary half of the season, then the FanGraphs methodology after the All-Star break, I received our greatest solutions but, shaving an additional two p.c price of imply squared error off of my earlier finest. Once more, deciphering a Brier rating by itself is tough, however I received my hybrid Bayesian mannequin down under 0.11. FanGraphs mode checked in at 0.118 with out modification, and that was the most effective particular person mode.
Only for enjoyable, I took that changed Bayesian mannequin and requested it to compute playoff odds on April 30, 2025. I picked the previous as a result of it wouldn’t be all that fascinating to examine as we speak (the mannequin simply makes use of FanGraphs mode this late within the yr), and April is the month that sees the biggest change. Right here’s how this model differs from our FanGraphs odds on that date:
Bayes-Modified Playoff Odds, April 30, 2025
| Workforce | FG Odds | Bayes Odds | Distinction |
|---|---|---|---|
| Dodgers | 98.2% | 87.1% | -11.0% |
| Mets | 86.7% | 86.9% | 0.3% |
| Yankees | 82.4% | 84.7% | 2.3% |
| Tigers | 78.3% | 82.5% | 4.2% |
| Mariners | 74.8% | 75.3% | 0.5% |
| Cubs | 67.3% | 69.2% | 1.9% |
| Astros | 58.6% | 61.7% | 3.1% |
| Padres | 50.2% | 58.8% | 8.6% |
| Phillies | 67.8% | 58.3% | -9.5% |
| Pink Sox | 59.0% | 56.0% | -3.0% |
| Braves | 68.8% | 50.7% | -18.1% |
| Giants | 46.9% | 50.1% | 3.1% |
| Diamondbacks | 54.3% | 49.6% | -4.8% |
| Rangers | 49.4% | 43.4% | -6.0% |
| Royals | 37.6% | 38.4% | 0.7% |
| Brewers | 28.4% | 38.3% | 10.0% |
| Twins | 36.7% | 35.6% | -1.0% |
| Guardians | 34.2% | 33.2% | -1.0% |
| Rays | 25.3% | 32.0% | 6.7% |
| Reds | 12.0% | 23.6% | 11.6% |
| Blue Jays | 27.1% | 21.0% | -6.1% |
| Cardinals | 13.5% | 19.2% | 5.7% |
| Athletics | 18.9% | 18.7% | -0.2% |
| Orioles | 15.5% | 13.3% | -2.2% |
| Angels | 2.3% | 3.9% | 1.6% |
| Pirates | 4.7% | 3.8% | -0.9% |
| Nationals | 0.9% | 2.4% | 1.4% |
| Marlins | 0.3% | 1.2% | 0.8% |
| White Sox | 0.0% | 0.8% | 0.8% |
| Rockies | 0.0% | 0.0% | 0.0% |
Desk is sortable. Skilled on 2014-2024 information.
It’s not good, however the greatest changes – Braves down, Reds up, Dodgers down, Brewers up – do a fairly good job of capturing the form of info I’d need a mannequin to select up in April. Certain, the Dodgers began nicely, however so did everybody of their division, and the playoff image seemed barely extra aggressive than anticipated. The Braves began poorly. Your complete NL Central seemed good.
The Bayesian model isn’t with out its misses, as a result of the season-to-date methodology isn’t with out its misses. Right here’s the identical snapshot on Might 31:
Bayes-Modified Playoff Odds, Might 31, 2025
| Workforce | FG Odds | Bayes Odds | Distinction |
|---|---|---|---|
| Tigers | 94.6% | 95.9% | 1.2% |
| Yankees | 97.2% | 94.8% | -2.5% |
| Dodgers | 98.4% | 94.0% | -4.4% |
| Cubs | 83.5% | 85.5% | 2.0% |
| Mets | 84.2% | 84.7% | 0.5% |
| Phillies | 90.5% | 72.8% | -17.8% |
| Twins | 66.9% | 68.0% | 1.1% |
| Astros | 67.0% | 63.7% | -3.3% |
| Mariners | 69.1% | 61.8% | -7.3% |
| Rays | 36.5% | 55.5% | 19.0% |
| Cardinals | 42.7% | 54.7% | 12.0% |
| Giants | 43.2% | 53.7% | 10.5% |
| Padres | 43.1% | 41.5% | -1.6% |
| Braves | 56.3% | 40.1% | -16.2% |
| Royals | 42.2% | 37.0% | -5.2% |
| Guardians | 44.5% | 35.7% | -8.8% |
| Blue Jays | 40.5% | 32.1% | -8.4% |
| Rangers | 21.5% | 27.5% | 6.0% |
| Brewers | 21.9% | 25.7% | 3.8% |
| Pink Sox | 15.3% | 22.0% | 6.6% |
| Reds | 5.5% | 20.0% | 14.5% |
| Diamondbacks | 27.7% | 19.5% | -8.2% |
| Nationals | 2.6% | 5.2% | 2.6% |
| Angels | 1.8% | 4.4% | 2.5% |
| Orioles | 1.8% | 0.8% | -1.0% |
| Athletics | 0.8% | 0.7% | -0.1% |
| Marlins | 0.1% | 0.5% | 0.4% |
| Pirates | 0.3% | 0.4% | 0.1% |
| White Sox | 0.0% | 0.3% | 0.3% |
| Rockies | 0.0% | 0.0% | 0.0% |
Desk is sortable. Skilled on 2014-2024 information.
Little too excessive on the Rays and Cardinals on reflection, and too low on the Phillies and Blue Jays. However the basic strikes the Bayes mannequin is making – shading groups whose efficiency and divisional place don’t match our preseason expectations – make plenty of sense to me.
Why present you all this? It’s to set down a efficiency marker. That is how good I can get the prevailing odds to be, utilizing all of the statistical strategies I’ve picked up through the years, plus a couple of new ones I realized from my trusty AI assistant whereas researching this text. Now that I’ve an excellent testing regime arrange and an thought of how far I can optimize a given set of odds utilizing statistical strategies, I’ve a goal for any future playoff odds calculations to be examined in opposition to. All I want is a time sequence of their predictions, and so they can slot proper into the prevailing testing framework.
May that be different websites’ odds? Certain, I suppose, although I don’t have a robust need to play web baseball referee. Extra seemingly, it’ll be future variations of our personal odds, like the assorted new ones we’ve added to the playoff odds web page within the final yr, or a totally operational model of our depth-aware methodology. No matter what the longer term holds, although, I’m assured that my present folding of our current methodologies is pretty much as good as I can do – and I’m additionally assured that the Bayes-aware model, whereas complicated, does an excellent job of selecting up a few of the early-season slack with out giving up the late-season excellence of projection-based modeling.