There is a reflex almost everyone has when they see a forecast, and it is the wrong one. A model says there is a seventy percent chance of something; the thing does not happen; the model was wrong. It feels obvious. It is also a category error, and it is the single most common way that both audiences and the people selling forecasts misunderstand what a probability even is.
A seventy percent forecast that fails is not evidence the model was wrong. A model that says seventy percent and is right every single time is not a good model - it is a broken one, or a liar. The property that actually separates a forecaster worth listening to from a confident guesser has a name, it is measurable, and it is almost never the thing people check. It is called calibration.
The wrong way to grade a forecast
Grade a single probabilistic forecast by its outcome and you learn essentially nothing. Any single event either happens or it does not; a seventy percent call and a ninety-nine percent call look identical after one flip if that flip goes the wrong way. Outcome-grading a probability one event at a time is like judging a weighted coin from a single toss. The information was never in the single result. It is in the pattern across many.
This is not a technicality. It is the reason the entire genre of AI price prediction survives on screenshots: pick the calls that happened, show those, and any model looks prophetic. The defence against that trick is not a better screenshot. It is refusing to grade probabilities one at a time at all.
What calibration actually means
Calibration is a property of a forecaster measured across many forecasts. It asks one question: when you say seventy percent, does the thing happen about seventy percent of the time?
Take every occasion a model assigned roughly seventy percent to something. Look at how often those things actually occurred. If it is around seventy percent, the model is well calibrated at that level. If those seventy percent calls only come true half the time, the model is overconfident - its numbers are inflated. If they come true nine times in ten, it is underconfident - it knew more than it admitted. Do this across every probability band, from ten percent to ninety, and you have a full picture of whether a model's stated confidence means anything at all.
Notice what this measures and what it does not. It does not ask whether the model is smart, or whether it beat the market. It asks a narrower and more fundamental thing: are its probabilities honest? A number that says seventy and behaves like fifty is not a forecast. It is decoration with a percent sign.
A perfectly calibrated model is wrong constantly, on purpose
Here is the part that trips people up, and it is worth stating as plainly as possible. A perfectly calibrated model is wrong all the time, by design.
If a model is properly calibrated, then across every case where it said seventy percent, the outcome fails to happen in three of every ten. That is not a flaw in the model. That is what seventy percent means. A forecaster whose seventy percent calls always came true would be badly miscalibrated - they should have been saying ninety-nine percent, or a hundred. The misses are not evidence against the model. Over a large enough sample, the misses at the right rate are the strongest evidence for it.
This inverts the usual intuition completely. The honest forecaster expects to be wrong on a predictable fraction of calls and would be alarmed to find themselves right too often. The dishonest one shows you only the hits and hopes you never count the total.
Calibration is not the same as being right
Calibration is necessary but it is not the whole story, and the missing half has a name too: sharpness.
A weather model that says forty percent chance of rain every single day, in a climate where it rains forty percent of days, is perfectly calibrated. It is also useless, because it never commits to anything. Sharpness is the willingness to move away from the base rate - to say ninety percent on the days it is confident and ten percent on the days it is not - while staying calibrated when you do. The goal is not just honest probabilities; it is honest probabilities that are also informative. A good forecaster is sharp and calibrated at once: it takes real positions and those positions are honestly priced.
The two can trade off, and pushing sharpness without keeping calibration is exactly how models start lying. Confidence is cheap. Confidence that survives being counted is not.
How you actually measure it
Three tools do most of the work, and none of them can be faked with a good week.
The **reliability diagram** is the picture. Plot stated probability on one axis and observed frequency on the other, in bins. A perfectly calibrated model traces the diagonal. Points sagging below the line are overconfidence; points above are underconfidence. It is a portrait of a forecaster's honesty, and its failures are visible at a glance.
The **Brier score** is the single number. It is the mean squared error between the probability and the outcome, treated as one or zero, and lower is better. Crucially it decomposes into calibration and sharpness terms, so it rewards a model for being both honest and decisive at the same time - which is exactly the combination you want.
**Log loss** is the harsher cousin, and its harshness is the point. It punishes confident errors far more severely than timid ones. A model that says ninety-nine percent and is wrong takes a brutal hit under log loss, while a model that hedged at sixty percent and was wrong barely notices. If you want to expose a forecaster that manufactures confidence it has not earned, log loss is the instrument that finds it.
Compare any of these against baselines, always. A proper score in isolation is meaningless; the honest move is to report it next to a naive baseline - a flat base rate, a random walk - and let the comparison speak. If the sophisticated model cannot beat the dumb one, publish that it could not.
Why AI models are usually overconfident, and how that gets fixed
Modern machine-learning models, and neural networks in particular, are frequently miscalibrated straight out of training, and almost always in the same direction: overconfident. A classifier will hand you a 0.99 that behaves like a 0.9. The raw scores that come out of these systems are not probabilities in any trustworthy sense; they are unnormalised confidences that happen to live between zero and one.
The fix is a discipline, not a trick. Post-hoc calibration methods - temperature scaling, Platt scaling, isotonic regression - take a trained model's raw outputs and stretch or compress them, on held-out data, until the stated probabilities match observed frequencies. The important detail is the held-out part: calibration measured on the data a model trained on is worthless, because the model has already seen the answers. Calibration only means something out of sample, on cases fixed after the model was frozen. A model that reports its calibration on its own training set is telling you nothing, confidently.
Why calibration only counts when it is committed in advance
All of this collapses the moment you allow the forecaster to grade themselves after the fact. Calibration is a claim about probabilities assigned before outcomes are known, and the only thing that makes it credible is that the assignment cannot be quietly revised once reality arrives.
This is the whole reason our forecasts are written down first, hashed with SHA-256, and anchored to the Bitcoin blockchain through OpenTimestamps before they are public. Not to prove the forecast was good - the reliability diagram and the score do that - but to prove the probability existed, at that value, before the event it describes. A calibration curve built from timestamped forecasts is evidence. A calibration curve built from numbers the author could have edited is a drawing. We publish ours with the misses at their honest rate, because a forecaster whose seventy percent calls fail three times in ten is not failing. They are working.
None of this is an edge, and it is worth saying so. Calibration does not tell you the future and it will not beat the market; it only guarantees that when a model quantifies its uncertainty, the number means what it says. No method reliably beats a liquid market, and anyone promising that is selling something. What calibration offers is smaller and rarer and more valuable than a hot tip: a probability you can actually trust, and a way to check, in public, that the trust was earned.
Educational content - not financial advice, and not a betting tip.