Wednesday, November 18, 2026
Data from SEC filings, checked daily
My DeskFree
Public filings, plainly told
Big data projects
Summarize with AIChatGPTClaudePerplexityGoogle AIGrokCopilot
CFA® Level II · Quantitative Methods · Lesson 7 of 7

Big data projects

Working through a text and data project from start to finish.

Written by Sepehr Parsa9 min read

The five steps

  1. Conceptualize: define the question, the output and how it will be used.
  2. Collect data: internal, vendor or web data; structured or unstructured (text, images).
  3. Prepare and wrangle: clean and transform data into a usable form.
  4. Explore: visualize, find patterns, engineer and select features.
  5. Train and evaluate: choose a method, train, tune and test.

Structured data

Cleaning: fix incompleteness (missing values), invalidity (out of range), inaccuracy, inconsistency, non-uniformity (mixed formats) and duplication.

Preprocessing:

  • Extraction of new variables (age from date of birth).
  • Aggregation, filtration, selection.
  • Outliers: trim or winsorize (replace extremes with a percentile value).
  • Scaling: normalization (to 0–1) or standardization (to mean 0, sd 1).
  • Encoding: turn categories into dummies (one-hot encoding).

Text data

Cleaning: remove HTML tags, punctuation (or replace it with tokens), numbers, extra spaces.

Wrangling:

  • Tokenization: split text into words (tokens).
  • Lowercasing.
  • Stop words: remove common words like “the” and “and”.
  • Stemming / lemmatization: reduce words to roots.
  • N-grams: keep useful word combinations (“interest rate”).
  • Bag of words and the document-term matrix: rows are documents, columns tokens, cells counts.

Exploration and feature selection: word clouds, term frequency, TF-IDF, chi-square tests and mutual information to find words that distinguish classes. Features can also be engineered: number of words, sentiment scores, named entities.

Evaluating models

For classification, the confusion matrix:

  • Precision = TP ÷ (TP + FP): of predicted positives, how many are right. Matters when false positives are costly.
  • Recall (sensitivity) = TP ÷ (TP + FN): of actual positives, how many were found. Matters when false negatives are costly.
  • Accuracy = (TP + TN) ÷ total. Misleading with imbalanced classes.
  • F1 = harmonic mean of precision and recall.
  • ROC curve: true positive rate against false positive rate as the threshold changes. AUC summarizes it: 0.5 is random, 1 is perfect.

For continuous targets: RMSE.

Tuning: adjust hyperparameters on validation data. A grid search tests combinations. Fitting curves (error versus complexity) show where overfitting begins.

Exam focus

Order the project steps, name the text-processing techniques, interpret a document-term matrix and compute precision, recall, accuracy and F1.

Check yourself

Tap an answer. You get the explanation straight away.

  1. A model flags 100 companies as likely to cut dividends: 80 do, 20 don't. Of all companies that actually cut, 120 existed. Precision and recall are:

  2. Reducing words such as 'earning', 'earned' and 'earns' to 'earn' is called:

  3. When false negatives are more costly than false positives (for example, missing a fraud), the analyst should focus on:

Questions people ask

What is TF-IDF?

Term frequency × inverse document frequency. It weights a word highly if it appears often in a document but rarely across all documents, picking out distinctive words.

Why scale features?

Methods based on distances or gradients, such as k-nearest neighbours and neural networks, are distorted if one feature has much larger values than others. Normalization or standardization fixes this.

What does an AUC of 0.5 mean?

The model is no better than random guessing. An AUC of 1 means perfect separation of positives and negatives.

Unlimited practiceGenerate more Quantitative Methods questionsFresh questions with worked solutions every time, in the Level II style.
Progress is saved in this browser.
Topic finishedTake the Level II practice examTimed, in the real exam format, with an explanation for every answer.

CFA Institute does not endorse, promote or warrant the accuracy or quality of WallStPal. CFA® and Chartered Financial Analyst® are registered trademarks owned by CFA Institute.

Exam format, timing and topic weights are from CFA Institute's Level I, II and III exam pages, checked November 15, 2026. Check CFA Institute for the year you sit. Lessons and practice questions are WallStPal's own and are not CFA Institute material.

Report an error

Tell us what looks wrong on this page. We check every report against the original filing and fix mistakes on our corrections page.

Share this trade
XTelegram
My Desk · free

Sign in to follow

Follow stocks, members of Congress, famous investors and funds, and see what's new each time you come back.