What is big data?
Big data means data sets too large or too complex for traditional tools. It is usually described by:
- Volume: the quantity of data.
- Velocity: how fast it arrives, sometimes in real time.
- Variety: structured (tables), semi-structured and unstructured (text, images, audio).
- Veracity: how reliable and accurate it is. Big data is often messy.
Sources of alternative data
| Source | Examples |
|---|---|
| Individuals | Social media posts, product reviews, search trends |
| Business processes | Credit card transactions, corporate data exhaust, supply chain records |
| Sensors and devices | Satellite images of car parks, shipping and location data, internet-of-things devices |
Analysts use these to spot trends before they appear in official figures, for example estimating a retailer’s sales from foot traffic. Using such data raises legal and ethical questions, such as privacy and whether data counts as material nonpublic information.
Machine learning
Machine learning (ML) finds patterns in data and uses them to make predictions, often with little human direction on what to look for.
Supervised learning
The training data has labelled outcomes, and the model learns to predict them.
- Regression problems predict a number (next month’s return).
- Classification problems predict a category (default or no default).
- Examples of methods: penalized regression (which limits model complexity), support vector machines, k-nearest neighbours, classification and regression trees, random forests.
Unsupervised learning
The data has no labels. The model finds structure on its own.
- Clustering: group similar observations, such as stocks that behave alike (k-means, hierarchical clustering).
- Dimension reduction: summarize many correlated variables with a few factors (principal components analysis).
Deep learning and reinforcement learning
Neural networks with many layers (deep learning) handle complex, non-linear patterns such as images, speech and text. Reinforcement learning learns by trial and error to maximize a reward, as in game-playing programs.
Overfitting and how to avoid it
A model can learn the noise in its training data as well as the real pattern. It then looks excellent on that data and fails on new data: overfitting. The opposite, underfitting, is a model too simple to capture the pattern.
To manage this, split the data:
- Training set: fit the model.
- Validation set: tune and choose between models.
- Test set: measure final performance on data the model has never seen.
A big gap between training and test performance signals overfitting. Simpler models, fewer features and penalties on complexity help.
Limits for analysts
- ML finds correlations, not causes. A pattern can disappear when conditions change.
- Some models are black boxes: hard to explain to clients or regulators.
- Results depend on data quality. Biased or incomplete data produces biased models.
Exam focus
Expect to classify a described method as supervised, unsupervised or deep learning, to recognize overfitting, to know the purpose of training, validation and test sets, and to name the characteristics of big data.