Module 6 · Hypothesis testing Advanced

A hypothesis test asks whether a result could plausibly be due to chance.

The steps

1. State the null hypothesis: the dull explanation, such as 'this manager's true alpha is zero'.
2. State the alternative: 'the alpha is not zero'.
3. Choose a significance level, usually 5%.
4. Compute the test statistic: t = (sample estimate minus value under the null) / standard error.
5. Compare with the critical value (about 2 in absolute value for a 5% two-sided test with a reasonable sample) or read the p-value.
6. Reject the null if the statistic is beyond the critical value; otherwise fail to reject it.

The p-value

The probability of seeing a result at least this extreme if the null hypothesis were true. A p-value of 0.03 does not mean a 97% chance the effect is real. It means that, if there were no effect, data like these would appear 3% of the time.

Two kinds of error

The significance level is the Type I error rate you accept. Power is the chance of detecting a real effect; it rises with sample size and with the size of the effect. Short track records have low power, so failing to find skill is not proof of its absence either.

One-tailed and two-tailed

Use a two-tailed test unless a result in one direction would be meaningless. Choosing one-tailed after seeing the data is a way of cheating.

The multiple testing problem

At a 5% level, one test in twenty of a true null will look significant by luck. Test 20 useless strategies and the chance that at least one appears significant is 1 minus 0.95^20 = 64%. Researchers, funds and newsletter writers who try many ideas and report the best one are publishing noise. Remedies: demand a higher bar (some researchers propose a t-statistic above 3 for a newly discovered factor), ask how many variations were tried, and require confirmation on data not used in the search.

Statistical versus economic significance

With enough data, a tiny effect can be statistically significant and still be worthless after trading costs. With little data, a large and real effect can fail the test. Report both the size of the effect and its t-statistic.

More worked examples

Correcting for many tests

The Bonferroni correction divides the significance level by the number of tests: testing 20 ideas at an overall 5% level means requiring 0.25% for each, a t-statistic of about 3.0. It is conservative, but it captures the right instinct: the more you search, the higher the bar. The t-statistic of 2.07 above would not survive if it were the best of 20 tries.

Confidence intervals and tests agree

A 95% confidence interval that excludes zero corresponds to rejecting a null of zero at the 5% level. An interval for monthly alpha of -0.1% to 0.7% includes zero, so the test does not reject; but it also says the true alpha could plausibly be as high as 0.7% a month. Report the interval, not just the verdict.

Power, worked through

Suppose a manager truly earns 2% a year of alpha with 5% tracking error. After 4 years, the expected t-statistic is 2 / (5 / square root of 4) = 0.8; the test will usually fail to find skill that is really there. After 25 years the expected t-statistic reaches 2.0, and even then the test rejects only about half the time. Low power is why investors cannot rely on track records alone.

P-hacking

Choosing the sample period, the benchmark, the outlier rule or the variables after seeing which choice gives the smallest p-value produces significant results from noise. The remedies are pre-registration (writing the test down first), reporting every specification tried, and confirming on new data.

Reading a research claim

A press report says a newsletter's picks 'beat the market with 99% confidence'. Ask: over what period, against which benchmark, after costs, how many picks were made, and whether the result was tested on picks made after the claim.

Put it into practice