I built a Monte Carlo simulator that prices MLB hitter props, wrote down a pass/fail gate before testing it, and ran it against live sportsbook lines for 87 days without betting a dollar. It failed. This is the autopsy.
Across 87 days the model said 62.3% on average and things happened 59.6% of the time. Slightly overconfident, and drifting worse through the season — but the reliability curve tracks the diagonal everywhere the mass sits.
Betting doesn’t reward being right. It rewards being right where the market is wrong. So sort every prediction by how far the model departed from the de-vigged market price, and look at what happened.
| Decile | Mean disagreement | Predicted | Observed | n | ROI |
|---|---|---|---|---|---|
| 1 — most bearish | −4.9% | .568 | .608 | 1,093 | −7.9% |
| 3 | −0.2% | .610 | .617 | 1,093 | −5.9% |
| 5 | +2.1% | .626 | .615 | 1,092 | −4.8% |
| 7 | +4.3% | .638 | .606 | 1,093 | −4.3% |
| 9 | +7.2% | .645 | .590 | 1,093 | −3.7% |
| 10 — most bullish | +12.0% | .647 | .540 | 1,092 | −3.9% |
The pre-registered cohort came in at −3.69% ± 1.59% ROI on n = 2,749 — more than two standard errors below zero, on twice the sample the gate asked for. But look at how it got there.
Meanwhile the criterion that was actually pre-registered — closing line value — never moved. Across the 964 legs with a captured closing price, mean CLV is +0.002%; on the high-edge subset, +0.009% on n = 289. Zero, to as many decimals as anyone wants. The closing line never came toward these picks, which is the market saying — quietly, 964 times — that there was nothing there.
A model can lose to the market for two very different reasons: it knows nothing, or it knows something and is being used wrong. These have opposite remedies, so it is worth separating them properly rather than by feel.
The test is a horse race. Regress the realised outcome on both forecasts at once, in log-odds space:
logit P(win) = a + b · logit(p_model) + c · logit(p_market)
If the model is pure noise, b collapses to zero once the market price is
in the equation. It doesn’t.
Look at what the edge calculation was actually doing.
edge = p_model − p_market treats the two forecasts as
equally credible — an implicit 50/50 weight on something that
earns 22%. The model was levered at roughly 2.3× its true
strength, and the size of that error scales with the size of the
disagreement.
Which is exactly the pattern in Finding 02. The further the model departs from the market, the more it is over-weighted, the worse it does. Decile 10 is where the mis-specification is maximal, and decile 10 is where the model dies.
Corroborating: standalone Brier is 0.2392 for the model and 0.2371 for the market price alone. The market is simply the better forecaster of the two. The model’s job was never to beat it outright — only to add something on top — and the horse race says it does, just far less than the edge formula assumed.
So the conclusion is narrow and specific: the edge estimator is mis-specified, not the model. The right operator is a stacked blend at the measured weights —
logit(p_final) = −0.04 + 0.231 · logit(p_model) + 0.820 · logit(p_market)
— with edge computed off that. Under that blend the edges collapse toward zero and essentially nothing on this market clears the vig. Which is the honest answer, and a different sentence from “the model doesn’t work.”
On 2026-05-30, two days before the first prediction was logged, this went into the project’s methodology document at v0.2.0, §8:
Graduate the simulator to live firing only after N ≥ 200 shadow predictions accumulate with mean CLV > 0 AND calibration holding. Falling short on CLV after N=200 is a finding, not a failure — it means the backtest edge did not survive contact with market prices, and the model needs attribution work before risking bankroll.
Two things about that paragraph matter more than anything else in the project.
First, the bar is closing line value, not profit. Whether a bet won is mostly noise at any realistic sample size. Whether the closing line moved toward your side is a far higher-signal test of whether you knew something the market didn’t. Grading on profit is how people talk themselves into deploying noise.
Second, it says in advance what a negative result would mean, so that when one arrived it couldn’t be relitigated. That is the entire point of pre-registration, and it’s the part of this work I’d defend hardest.
Players get scratched. If a leg never settles it sits in the log as a blank, and blanks
are invisible to every summary statistic — so a record quietly drops its
inconvenient rows. I added an explicit void path gated on the game
actually being final with boxscores ingested.
Then the void path over-fired. A missed morning run made 79 real legs look unsettleable and voided them. I caught it, tightened the condition — postponed games never post as Final, so they need an age-based rule instead — and shipped a repair path that flips false voids back once stats arrive. It recovered 73 of the 79.
That second one is the tell. The cheap move was to leave 79 legs voided; nobody would have known, and the record would have looked marginally cleaner. Rebuilding the rows was strictly worse for the numbers and strictly better for the evidence.
p_market carries real measurement error,
which attenuates the horse-race coefficients — the model’s true
b is probably somewhat higher than 0.231.