The five steps
- Conceptualize: define the question, the output and how it will be used.
- Collect data: internal, vendor or web data; structured or unstructured (text, images).
- Prepare and wrangle: clean and transform data into a usable form.
- Explore: visualize, find patterns, engineer and select features.
- Train and evaluate: choose a method, train, tune and test.
Structured data
Cleaning: fix incompleteness (missing values), invalidity (out of range), inaccuracy, inconsistency, non-uniformity (mixed formats) and duplication.
Preprocessing:
- Extraction of new variables (age from date of birth).
- Aggregation, filtration, selection.
- Outliers: trim or winsorize (replace extremes with a percentile value).
- Scaling: normalization (to 0–1) or standardization (to mean 0, sd 1).
- Encoding: turn categories into dummies (one-hot encoding).
Text data
Cleaning: remove HTML tags, punctuation (or replace it with tokens), numbers, extra spaces.
Wrangling:
- Tokenization: split text into words (tokens).
- Lowercasing.
- Stop words: remove common words like “the” and “and”.
- Stemming / lemmatization: reduce words to roots.
- N-grams: keep useful word combinations (“interest rate”).
- Bag of words and the document-term matrix: rows are documents, columns tokens, cells counts.
Exploration and feature selection: word clouds, term frequency, TF-IDF, chi-square tests and mutual information to find words that distinguish classes. Features can also be engineered: number of words, sentiment scores, named entities.
Evaluating models
For classification, the confusion matrix:
- Precision = TP ÷ (TP + FP): of predicted positives, how many are right. Matters when false positives are costly.
- Recall (sensitivity) = TP ÷ (TP + FN): of actual positives, how many were found. Matters when false negatives are costly.
- Accuracy = (TP + TN) ÷ total. Misleading with imbalanced classes.
- F1 = harmonic mean of precision and recall.
- ROC curve: true positive rate against false positive rate as the threshold changes. AUC summarizes it: 0.5 is random, 1 is perfect.
For continuous targets: RMSE.
Tuning: adjust hyperparameters on validation data. A grid search tests combinations. Fitting curves (error versus complexity) show where overfitting begins.
Exam focus
Order the project steps, name the text-processing techniques, interpret a document-term matrix and compute precision, recall, accuracy and F1.