Evaluate a backtest, describe a Monte Carlo simulation and calculate value at risk.
Applying a trading rule to historical data to see how it would have performed. A backtest is evidence, not proof, and it is easy to produce a beautiful one that is worthless.
| Problem | Description | Defence |
|---|---|---|
| Look-ahead bias | The rule uses data not available at the decision date | Lag all inputs to their true publication dates |
| Survivorship bias | The test universe excludes delisted and failed companies | Use point-in-time data including dead stocks |
| Data snooping (overfitting) | Many rules or parameters tried; only the best reported | Count the trials; hold back data; raise the bar |
| Ignoring costs | No commissions, spreads, market impact, borrowing costs or taxes | Include realistic costs, especially for high turnover |
| Unrealistic execution | Trades assumed at prices or sizes not available | Trade at the next available price; cap position size by volume |
| Short or special sample | Tested over one regime | Test across cycles and markets |
When a question is too complicated for a formula, simulate it.
A retirement plan that 'succeeds in 85% of simulations' comes from this method. The answer is only as good as the model: assuming normal, independent returns understates bad sequences. An alternative, the bootstrap, resamples actual historical returns instead of drawing from a formula.
The loss that should not be exceeded over a given period at a given confidence level.
How bad the loss is on the days it is exceeded. Expected shortfall (conditional VaR) answers that: the average loss in the worst 5% of cases. VaR based on a calm period will be far too low when conditions change, as banks learned in 2008. Stress tests, which ask what happens in a specific severe scenario, complement it.
(1) What is the sample and how was it chosen? (2) How many things were tried before this one was reported? (3) What is the size of the effect and its standard error? (4) Does it survive out of sample and after costs? (5) Is there a reason it should persist? (6) What happens in the tail?
A researcher tests 100 strategies, none of which has any real edge, on 10 years of data. Each one's measured Sharpe ratio is noise with a standard error of about 1 / square root of 10 = 0.32. The highest of 100 such draws is typically about 2.5 standard errors above zero, a Sharpe ratio of about 0.8. A backtest Sharpe of 0.8 chosen from 100 trials is therefore no evidence of skill. Methods such as the 'deflated Sharpe ratio' adjust for the number of trials.
The five worst daily returns in the last 100 days were -3.0%, -3.5%, -4.0%, -5.0% and -6.5%.
Two portfolios with the same VaR can have very different expected shortfalls; the one with the fatter tail is riskier. Banking regulation now uses expected shortfall for market risk for this reason.
A strategy that makes 24 round trips a year at a cost of 0.2% each loses 4.8% a year to trading costs alone. Many attractive gross backtests turn negative at realistic costs, especially in small, illiquid stocks where spreads and market impact are larger than the backtest assumes.
Estimate the rule's parameters on years 1 to 5, trade year 6; re-estimate on years 2 to 6, trade year 7; and so on. The stitched-together out-of-sample years are the honest record. If the rule only works with parameters chosen on the full sample, it is fitted to the past.
A plan that 'succeeds in 85% of simulations' means that in 85% of the simulated return paths the money lasted as long as needed. It does not mean an 85% return or a guarantee. The 15% of failing paths are usually those with poor returns early in retirement (sequence risk), and the result is only as good as the return model, the spending rule and the assumption about lifespan.
Instead of a probability, a stress test asks a specific question: what happens to this portfolio if 2008, 2020 or 2022 repeats, or if rates rise three points in a year? Stress tests catch exposures that a VaR estimated in calm markets misses, such as illiquidity and correlations that jump in a crisis.
Backtest one simple rule on your index with ten years of data: hold the index when its price is above its 200-day moving average, otherwise hold cash. Use the first six years as in-sample and the last four as out-of-sample. Report return, volatility, Sharpe ratio, maximum drawdown and number of trades for the rule and for buy-and-hold in each period, with a 0.1% cost per trade. Then compute the portfolio's one-day 95% VaR by the parametric and historical methods.
Educational content only; not investment, tax or legal advice. Companies, tickers and figures are illustrative examples, not recommendations. Rules, rates and limits change, so check current details with the regulator, exchange or tax authority before relying on them. Worked examples are not forecasts. Written by Elizabeta Dimoska. RiskStock is not a registered dealer or adviser.