The backtesting process
- Strategy design: define the investment hypothesis, universe, factors, rebalancing frequency, weighting and benchmark.
- Historical investment simulation: build portfolios at each rebalancing date using only data available then; record returns.
- Output analysis: returns, risk (volatility, drawdown, VaR), turnover, factor exposures, performance in sub-periods.
Rolling window: estimate on an in-sample window, test on the next out-of-sample period, roll forward. It mimics real-time investing and reveals instability.
Common problems
- Survivorship bias: using only companies that still exist inflates returns. Use point-in-time databases including delisted firms.
- Look-ahead bias: using data not yet available (restated figures, late earnings). Use reporting lags and point-in-time data.
- Data snooping: testing many variations until something works. Require economic reasoning and stricter significance.
- Transaction costs and shorting constraints ignored.
- Regime changes: relationships change across economic or market regimes.
- Non-normal returns: fat tails and skewness that averages hide.
Spot the bias
Beyond backtesting
Scenario analysis: test the strategy in specific environments (recessions, rate shocks, high volatility) or structural breaks.
Historical simulation: draw past returns randomly (bootstrapping) to create many paths.
Monte Carlo simulation: specify distributions (possibly with fat tails, regime switching, changing correlations) and simulate.
Sensitivity analysis: vary assumptions to test robustness.
Exam focus
Describe the backtesting steps, identify survivorship, look-ahead and data-snooping biases, and compare historical simulation, Monte Carlo and scenario analysis.