Module 8 · Regression Advanced

Estimate and interpret a simple regression, read standard output and recognize overfitting.

The model

A simple linear regression fits the line y = a + b x + error that minimizes the sum of squared errors (ordinary least squares).

• Slope: b = covariance(x, y) / variance(x) = correlation x (standard deviation of y / standard deviation of x).
• Intercept: a = mean of y minus b x mean of x.
• R-squared: the share of the variance of y explained by x; in a simple regression it equals the correlation squared.

The market model

Regress a stock's excess return (over the risk-free rate) on the market's excess return.

• The slope is beta.
• The intercept is alpha, the average return not explained by the market.
• R-squared is the share of the stock's risk that is market risk; one minus R-squared is specific risk.

Shortcut: with a correlation of 0.6, stock volatility of 30% and market volatility of 15%, beta = 0.6 x 30 / 15 = 1.2 and R-squared = 36%.

Reading regression output

A regression of 60 monthly returns reports:

Beta is clearly different from zero (t of 9.6). Its 95% confidence interval is roughly 1.15 plus or minus 2 x 0.12: from 0.91 to 1.39, so it is not clearly different from 1. Alpha has a t of 0.8 and is not significant. If the market's excess return is 5% in a month, the predicted excess return is 0.20 + 1.15 x 5 = 5.95%.

Assumptions

The relationship is linear; the errors are independent of each other; they have constant variance; they are unrelated to x. Check by plotting the residuals: they should look like shapeless noise. Patterns, funnels or a few dominant points mean the standard errors, and sometimes the estimates, cannot be trusted.

Multiple regression

Several explanatory variables at once, as in the Fama-French factor model (RS 302, Module 6): excess return = alpha + b x market + s x size + h x value. Each coefficient is the effect of that variable holding the others fixed.

• Adjusted R-squared penalizes extra variables; plain R-squared always rises when one is added.
• Multicollinearity: when explanatory variables are highly correlated with each other, individual coefficients become unstable.
• Dummy variables (0 or 1) capture categories such as a month or a recession.

Overfitting

A model with enough variables can fit any past data perfectly and predict nothing. Signs: many variables relative to observations, coefficients with no economic logic, a fit that collapses on new data. Prefer simple models, and judge them on data they have not seen.

Limits

Regression describes association in the sample. It does not prove causation, and relationships estimated in one regime can break in the next.

More worked examples

Adjusted R-squared

Adding any variable, even random noise, never lowers R-squared. Adjusted R-squared subtracts a penalty for each variable, so it can fall when a useless variable is added. Compare models on adjusted R-squared, and better still on out-of-sample fit.

Dummy variables, worked

A regression of monthly returns includes a dummy equal to 1 in recession months and 0 otherwise. A coefficient of -1.5 means that, holding the other variables fixed, returns averaged 1.5 percentage points a month lower in recession months. With few recession months in the sample, check its standard error before trusting it.

Reading a factor regression

A fund regressed on the Fama-French factors shows: alpha 0.10% a month (t 0.6), market 0.98 (t 25), SMB 0.45 (t 6), HML -0.30 (t -4). The fund behaves like the market with a strong small-cap tilt and a growth tilt, and shows no statistically significant alpha. Its returns are explained by exposures an investor could buy cheaply.

Prediction intervals

The fitted line gives the expected value of y for a given x. An actual outcome will scatter around it. The prediction interval for a single new observation is much wider than the confidence interval for the line itself, because it includes the residual noise. Forecasting next month's stock return from the market's return is uncertain even when beta is estimated precisely.

Regression to the mean

Francis Galton found that very tall parents tend to have children who are tall but closer to average. Investment results behave the same way: the top fund of one year usually lands nearer the middle the next, because part of its top result was luck that does not repeat. Mistaking regression to the mean for a loss of skill, or a change of strategy for its cause, is a common error.

Put it into practice