The model
Regression measures how a dependent variable Y changes with an independent variable X:
Yi = b0 + b1 × Xi + εi
- b0, the intercept: predicted Y when X = 0.
- b1, the slope: change in Y for a one-unit change in X.
- ε, the error: what the line does not explain.
Ordinary least squares (OLS) picks the line that minimizes the sum of squared errors.
Estimating the coefficients
b1 = Cov(X, Y) / Var(X)
b0 = Ȳ − b1 × X̄
Example. Regress a stock’s monthly returns on the market’s. Cov = 0.0012, Var(market) = 0.0016, mean stock return 1.0%, mean market return 0.8%.
- b1 = 0.0012 / 0.0016 = 0.75 (the stock’s beta).
- b0 = 1.0% − 0.75 × 0.8% = 0.4%.
- Predicted return when the market returns 2%: 0.4% + 0.75 × 2% = 1.9%.
Assumptions
- Linearity: the true relationship is linear.
- Homoskedasticity: the variance of the errors is the same for all X.
- Independence: the errors are uncorrelated with each other (no serial correlation, a common problem with time series).
- Normality: the errors are normally distributed (needed for hypothesis tests in small samples).
Plots of the residuals against X are the usual way to spot violations: a funnel shape suggests heteroskedasticity; a pattern over time suggests the errors are not independent.
Measuring fit
Variation in Y splits into explained and unexplained parts:
SST (total) = SSR (explained by the regression) + SSE (errors)
| Measure | Formula | Meaning |
|---|---|---|
| R² | SSR / SST | Share of Y’s variation explained by X. In simple regression, R² = r² |
| Standard error of estimate | √[SSE / (n − 2)] | Typical size of a prediction error |
| F-statistic | MSR / MSE = (SSR / 1) / [SSE / (n − 2)] | Tests whether the model explains anything |
With SST = 200 and SSE = 50: R² = 150 / 200 = 0.75.
Testing the slope
To test whether X explains Y (H0: b1 = 0):
t = (b1 − 0) / s(b1) with n − 2 degrees of freedom
Reject H0 if |t| exceeds the critical value. You can test against other values too: to test whether a stock’s beta equals 1, use (b1 − 1) / s(b1). In simple regression, the F-test of the whole model is equivalent to this t-test (F = t²).
Prediction intervals
A forecast of Y for a given X is uncertain for two reasons: the error term and the uncertainty in the estimated coefficients. A prediction interval is:
Ŷ ± t(critical) × sf
where sf, the standard error of the forecast, grows the further X is from its mean. Forecasts far outside the data are least reliable.
Functional forms
When the relationship is not linear, transform the variables:
| Model | Form | Use |
|---|---|---|
| Log-lin | ln Y = b0 + b1X | Y grows at a constant rate |
| Lin-log | Y = b0 + b1 ln X | Y responds to percentage changes in X |
| Log-log | ln Y = b0 + b1 ln X | b1 is an elasticity |
Exam focus
Expect to compute a slope and intercept, make a prediction, calculate R² from an ANOVA table, test a slope coefficient, and identify a violated assumption from a description.