Why Most Model Comparisons Are Misleading
Comparing forecasting models is harder than it looks. The natural instinct is to train two models on the same dataset, check which one has the lower error, and declare a winner. But that approach has enough holes to invalidate a research paper — let alone a production system.
The problem is almost never the models themselves. It is the evaluation protocol. Subtle choices around data splitting, metric selection, and baseline comparison can make a mediocre model look brilliant or a strong model look ordinary. If you are serious about forecasting — in finance, supply chains, energy, or any other domain — you need a framework that is honest before it is flattering.
This article lays out that framework.
---
The Foundation: Fix Your Evaluation Protocol First
Before touching a single hyperparameter, get the evaluation protocol right. Everything downstream depends on it.
The Train / Validation / Test Split
The standard split assigns data to three non-overlapping sets:
- **Training set** — the model learns here. - **Validation set** — you tune hyperparameters and make architectural choices here. - **Test set** — you look at this *once*, after every decision is already made.
The test set is sacred. The moment you use it to inform any modeling choice, it becomes a second validation set and your final numbers are optimistic. One look, one number.
For time-series data specifically, the split must always be chronological. Never shuffle time-series observations before splitting; doing so leaks future information into the training set — a form of data leakage covered in detail below.
Walk-Forward (Rolling) Validation
A single chronological split can still be fragile. You might happen to choose a calm or a turbulent period as your test window, and that accident of timing dominates your results.
Walk-forward validation, also called rolling-origin cross-validation, addresses this. The idea: train on all data up to time *t*, forecast the next *h* periods, record the errors, then roll the origin forward and repeat. You accumulate many test windows that collectively reflect how the model would have performed if deployed sequentially over time.
This is especially important in financial and macro data, where volatility regimes shift. A model that performs well in a calm period and poorly in a turbulent one — or vice versa — is giving you information that a single split would hide entirely.
---
Choose the Right Error Metrics
No single metric tells the whole story. Using multiple metrics, each measuring a different dimension of forecast quality, gives a far more complete picture.
RMSE and MAE: Magnitude Matters
**Mean Absolute Error (MAE)** is the average of absolute differences between forecasts and actuals. It is interpretable, robust to outliers, and easy to explain to non-technical stakeholders.
**Root Mean Squared Error (RMSE)** squares errors before averaging, then takes the square root. This means large errors are penalized more heavily than small ones. If a large miss carries a disproportionate cost in your application, RMSE better reflects that reality.
The discipline here is to choose the metric that matches the cost structure of your specific problem — not to default to whichever metric appears most often in published papers.
MAPE and Its Blind Spots
**Mean Absolute Percentage Error (MAPE)** expresses error as a fraction of the actual value, making it scale-independent and easy to communicate across different series. But it has well-documented failure modes: it is undefined when actuals pass through zero, and it is asymmetric — an over-forecast and an under-forecast of the same magnitude receive different penalties.
**Symmetric MAPE (sMAPE)** corrects some of this asymmetry but introduces its own edge cases. Use percentage-based metrics with full awareness of their limits, not as a default.
Directional Accuracy: Are You Right About Direction?
In many contexts — particularly in markets — the direction of a forecast matters as much as its magnitude. A model that consistently predicts upward movement when the actual direction is downward is operationally useless, even if its RMSE is technically acceptable.
**Directional accuracy**, sometimes called the hit rate, measures the proportion of forecasts where the predicted sign of change matched the actual sign. It is a simple but powerful check that magnitude-based metrics can completely mask.
When direction is meaningful to your use case, include it in every evaluation table.
---
Always Benchmark Against a Naive Model
This rule is violated more often than it should be. Before claiming that a sophisticated model works, you must show that it outperforms something trivially simple.
Standard naive benchmarks include:
- **Naïve forecast** — the next value equals the last observed value (the random walk). - **Seasonal naïve** — the next value equals the value from the same season in the previous cycle. - **Historical mean** — the next value equals the average of all historical observations. - **Simple exponential smoothing** — a low-parameter, well-understood adaptive average.
If a neural network or ensemble model cannot consistently beat a three-line random-walk implementation on your data, that is important information. It does not necessarily mean the model is worthless — it might reduce tail risk, or outperform in specific regimes — but it demands explanation, not dismissal.
Many published models beat naive benchmarks on average but fail on a regime-specific or risk-adjusted basis. Always disaggregate results before drawing conclusions.
---
Test for Statistical Significance
A model that beats its benchmark by two percent on one test set might simply have been lucky. Statistical significance testing formalizes whether observed differences are real or noise.
The **Diebold-Mariano (DM) test** is the standard instrument for pairwise model comparison. It tests whether the difference in forecast accuracy between two models is statistically distinguishable from zero under a null hypothesis of equal predictive ability.
The **Model Confidence Set (MCS)** procedure extends this to multiple models at once, identifying the set of models that cannot be statistically separated from the best performer at a given confidence level.
Without these tests, model selection is intuition dressed as analysis. With them, you can make defensible, reproducible claims about which models are genuinely superior — and which are noise.
---
Guard Against Data Leakage
Data leakage is the most common silent killer of model comparisons. It occurs when information from outside the legitimate training window finds its way into the model, inflating apparent performance in ways that never survive real deployment.
Common sources in time-series and market forecasting:
- **Lookahead bias** — features computed using statistics from the full dataset, including the test period. Normalization is a frequent offender: if you scale a feature using its global mean and standard deviation, the test set has already influenced the training representation. - **Target leakage** — including variables that are proxies for the target and are only knowable in hindsight. - **Temporal contamination** — shuffling or re-indexing time-series data before splitting, which randomly distributes future observations into the training set.
Preventing leakage is a process discipline as much as a technical one. Reproducible pipelines, explicit feature-construction timestamps, and systematic code reviews are more reliable than any single check.
---
Model Complexity Is Not Free
Occam's Razor applies directly to forecasting. A more complex model is not automatically superior — it carries real costs that must be weighed against any accuracy gain:
- **Overfitting risk** — complex models fit training noise and may generalize poorly outside the training distribution. - **Interpretability** — a model you cannot explain is a model you cannot trust or audit in high-stakes environments. - **Maintenance burden** — more parameters mean more failure modes over time. - **Computational cost** — relevant whenever forecasts must be generated at scale or in near-real-time.
When two models perform comparably under rigorous evaluation, the simpler one is preferable. This is not a philosophical stance; it is grounded in decades of forecasting research.
**AIC (Akaike Information Criterion)** and **BIC (Bayesian Information Criterion)** formalize this trade-off by penalizing complexity directly. They are most informative when comparing models within the same family, but they provide a useful check against complexity inflation in any setting.
---
A Practical Comparison Checklist
Before reporting any model comparison result, verify each of the following:
1. **Chronological split enforced** — no future data leaks into training. 2. **Test set examined exactly once** — after all architectural and tuning decisions. 3. **Walk-forward validation used** — multiple windows, not a single split. 4. **Multiple metrics reported** — at least one magnitude metric and one directional metric. 5. **Naive baseline included** — always, without exception. 6. **Statistical significance tested** — Diebold-Mariano or equivalent. 7. **Leakage audit completed** — every feature checked for temporal validity. 8. **Complexity cost documented** — added parameters justified explicitly.
---
Final Thought
Fair model comparison is not about being harsh on your own work. It is about being honest with yourself before the market provides its own verdict.
The discipline of rigorous evaluation is what separates research that holds up from research that flatters. Forecasting models that prove themselves under these conditions are the ones worth building on. Everything else is a hypothesis — and markets test hypotheses in ways no validation set can fully replicate.
The framework above does not guarantee good forecasts. Nothing does. What it guarantees is that when you claim a model performs, you have earned the right to that claim.