Every model that works does so on the assumption that tomorrow resembles the data it learned from. Most of the time that assumption is quietly true. When it stops being true, the model does not announce it. It keeps producing confident output, in exactly the same format, and only the results get worse.
This is the most common way machine learning fails in production, and it is much harder to notice than a crash.
What "regime" means
A regime is a period during which the relationships in your data hold roughly steady. Markets have calm regimes and turbulent ones. Traffic has term-time and holiday. Demand has pre-recession and post-recession. Inside a regime, a model trained on the past is doing something reasonable. Across a regime boundary, it is answering a question about a world that no longer exists.
Researchers split this into two failures worth telling apart.
The inputs change. The kind of data arriving is different from the kind you trained on - new users, a new instrument, a quieter market. The relationship you learned may still be valid; you are simply extrapolating outside where you have evidence.
The relationship changes. The inputs look familiar but they mean something different now. This is worse, because nothing about the incoming data looks unusual. The model is confidently applying a rule that has expired.
Why your backtest will not warn you
A backtest run over several years averages across every regime in that period. The result is a single number that describes a market which never existed - part calm, part violent, permanently in between.
The average hides the structure. A model can score respectably over five years while being systematically wrong in every quiet stretch and systematically wrong the other way in every violent one, with the errors cancelling in the total. The summary looks fine. Every individual period is a mess.
This is why aggregate accuracy is a weak diagnostic and why splitting results by condition is where the information actually is. If you have never looked at your metrics broken down by how volatile, busy or unusual the period was, you do not yet know whether your model has this problem.
Our own case, with numbers
We publish interval forecasts: a range, with a stated probability that the outcome lands inside it. A 50% range should contain the outcome about half the time. That is the entire contract, and it is falsifiable, which is why we state it.
Across 56 resolved forecasts, ours contained the outcome 47 times. Roughly 84 per cent against a target of 50.
Over-coverage is a failure, and it is the more dangerous kind. An interval that is too narrow gets caught immediately - outcomes fall outside, someone complains. An interval that is too wide produces nothing but successes. Every result lands inside, the page looks healthy, and there is no symptom to investigate. We were wrong for weeks in a way that generated no evidence of being wrong.
Our first hypothesis was that the sample was too short. We were mistaken, and the walk-forward study says so: rebuilding every interval from prior data only, across 20,058 observations, coverage came out at 55.6% against the 50% target and 85.3% against the 80% target - drifting high regardless of history length.
Splitting by regime found it. On the calmest fifth of days, the same intervals covered 67.1% and 92.1%. The bands were carrying a permanent allowance for turbulence that a quiet market had not earned, and averaging concealed it.
Conditioning the sample on the volatility regime measured at the moment the forecast opens - not at the close, which would leak the outcome we are trying to predict - brought coverage to 50.8% and 79.6%, and on the calmest fifth to 52.4% and 79.1%. The distortion in quiet markets, which had been the worst case, essentially disappeared.
We published the diagnosis before the fix was deployed, and did not re-seal the affected forecasts. Rewriting them would have destroyed the only property that makes the record worth anything.
Catching it earlier than we did
Four habits, in rough order of how much they would have helped us.
Monitor calibration, not accuracy. Accuracy answers "was it right". Calibration answers "was its confidence honest", and confidence goes wrong first. A model whose stated 70% events happen 90% of the time is already broken, however good its hit rate looks.
Treat over-performance as a defect. If a metric is beating its target, something is mis-specified. Nobody investigates good news, which is exactly why it survives so long.
Segment every metric by condition. Volatility, volume, time of day, whatever defines a regime in your domain. The aggregate is where problems go to hide.
Condition on the state at decision time. Whatever the model uses to know which regime it is in must be observable at the moment it decides. Using anything from the window it is forecasting is leakage, and it produces excellent test results and a useless system.
The uncomfortable general point
The model that has performed well for a year is not thereby proven. It may simply have run inside a single regime, and be untested rather than robust. Time in production is not evidence of generality - it is evidence of exposure to whatever conditions happened to occur.
The only real answer is to keep a scored record, split it by condition, and treat a run of unbroken success as a hypothesis rather than a result.
What we do
We build AI forecasting agents and the terminals they run in. Every forecast is hashed and Bitcoin-timestamped before the event and scored publicly afterwards, misses included. The record is not marketing - it is the instrument that caught the failure described above, and it is the reason the failure lasted weeks rather than indefinitely.
Educational content - not financial advice.