Types of machine learning
- Supervised learning: labelled data (inputs with known outputs). Regression for continuous targets, classification for categories.
- Unsupervised learning: no labels; find patterns, groups or simpler representations.
- Deep learning: neural networks with many layers for complex data such as images and text.
- Reinforcement learning: an agent learns by trial and error to maximize rewards.
Fit and generalization
- Bias error: the model is too simple (underfitting): high error on both training and test data.
- Variance error: the model is too complex (overfitting): low training error, high test error.
- Base error: irreducible noise.
Split data into training, validation and test sets. k-fold cross-validation rotates the validation set to estimate out-of-sample error. Regularization penalizes complexity.
Supervised methods
| Method | Idea | Note |
|---|---|---|
| Penalized regression (LASSO) | Regression with a penalty on coefficients | Selects features, reduces overfitting |
| Support vector machine | Finds the boundary with the widest margin between classes | Soft margins allow some errors |
| k-nearest neighbours | Classify by the majority of the k closest observations | Choice of k and distance measure matter |
| Classification and regression trees (CART) | Split data by feature thresholds | Easy to interpret; prone to overfitting; prune |
| Ensembles: bagging, random forests, boosting | Combine many models | Lower variance; less interpretable |
Unsupervised methods
- Principal components analysis (PCA): turns correlated features into uncorrelated components; keep those explaining most variance. Use a scree plot to choose how many.
- k-means clustering: assigns observations to k clusters by distance to cluster centres, iterating until stable.
- Hierarchical clustering: builds clusters step by step (agglomerative, bottom-up; divisive, top-down) and shows them in a dendrogram.
Neural networks
Layers of nodes: input, hidden and output. Each node sums weighted inputs and applies an activation function (such as sigmoid or ReLU) to capture non-linearity. Weights are learned by backpropagation. Deep networks have many hidden layers. Powerful but data-hungry and hard to interpret.
Exam focus
Match a problem to the right method, explain bias versus variance error and how to reduce overfitting, and describe PCA, clustering and neural networks.