Calculate a standard error and a confidence interval and identify sampling biases.
A sample mean is one draw from the range of sample means that could have occurred. The standard error measures how much it would vary from sample to sample:
Whatever the shape of the underlying distribution, the average of a large number of independent observations is approximately normally distributed. This is why the normal distribution can be used for averages even when individual returns are not normal.
A 95% confidence interval is the sample mean plus or minus 1.96 standard errors (use the t value for small samples).
Example: 64 monthly returns average 1.0% with a standard deviation of 4.0%. Standard error = 4.0 / 8 = 0.5%. The 95% interval is 1.0% plus or minus 0.98%: from 0.02% to 1.98% a month. The true mean could plausibly be almost zero or almost 2%.
Ten years of annual returns with a mean of 8% and a standard deviation of 16% give a standard error of 16 / square root of 10 = 5.1%. The 95% interval runs from about minus 2% to +18%. A decade of data cannot distinguish a poor asset from an excellent one. Using monthly data does not help with the mean: only a longer calendar span does.
Their estimates improve with more frequent observations, so risk can be measured far more precisely than expected return. This is why portfolio construction leans on risk estimates and treats return forecasts with suspicion.
To halve the standard error you need four times the data.
| Bias | What goes wrong | Example |
|---|---|---|
| Survivorship | Failures are missing from the sample | Fund databases that drop closed funds overstate average performance |
| Selection | The sample is not representative | Studying only companies that chose to report a metric |
| Look-ahead | Using information not available at the time | Ranking stocks on earnings before they were published |
| Time-period | Results depend on the window chosen | A strategy tested only in a bull market |
| Backfill | A fund's good early record is added when it joins a database | Hedge fund indexes |
Every number computed from data should be reported with its uncertainty. An estimate without a standard error is half an answer.
To estimate an asset's mean annual return with a standard error of 0.5 percentage points when its volatility is 16%, we would need (16 / 0.5)^2 = 1,024 years of data. Even a standard error of 4 points needs (16 / 4)^2 = 16 years. Means are the hardest thing in finance to estimate.
A strategy shows a Sharpe ratio of 0.5 over 10 years of data. An approximate standard error is the square root of (1 + Sharpe^2 / 2) / years = square root of 1.125 / 10 = 0.34. The 95% interval runs from about -0.16 to 1.16. Ten years cannot tell a mediocre strategy from an excellent one, or rule out one with no edge at all.
The standard error of a volatility estimate is roughly the volatility divided by the square root of twice the number of observations. With 120 monthly observations and an annualized volatility of 20%, the standard error is about 20 / square root of 240 = 1.3 points; with 2,500 daily observations it is under half a point. This is why risk models can be useful while return forecasts are not.
With 10 observations, the 95% two-sided critical value is 2.26 instead of 1.96, so intervals are about 15% wider. With 30 observations it is 2.05; with 120, 1.98. The difference matters for short track records.
Resample the actual data with replacement thousands of times, compute the statistic on each resample, and read the confidence interval from the spread of results. The bootstrap needs no assumption about the distribution, which suits skewed and fat-tailed returns. A block bootstrap resamples runs of consecutive observations to keep volatility clustering intact (Module 9).
A database shows 200 equity funds with a 10-year average return of 7.5%. Investigation shows 60 funds that existed at the start were closed or merged, mostly after poor results. Including them at their returns until closing lowers the average to 6.4%. Survivorship bias of one percentage point a year is common in fund studies.
For your index, compute the mean monthly return, its standard error and the 95% confidence interval using the full ten years. Repeat for the first five and the last five years separately. Observe how much the estimates differ.
Educational content only; not investment, tax or legal advice. Companies, tickers and figures are illustrative examples, not recommendations. Rules, rates and limits change, so check current details with the regulator, exchange or tax authority before relying on them. Worked examples are not forecasts. Written by Elizabeta Dimoska. RiskStock is not a registered dealer or adviser.