Data Preprocessing

Reviewed & published by Brayan K

Turn messy, raw data into clean, model-ready numbers — handle missing values, scale features, encode categories, and split safely without leaking the answers.

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

What You'll Learn in This Lesson

• You'll be able to find and fill missing values with the mean or median
• You'll be able to scale features with min-max (0–1) by hand
• You'll be able to standardise features to a z-score (mean 0, std 1)
• You'll be able to one-hot encode nominal categories with lists
• You'll be able to pick label vs one-hot encoding correctly
• You'll be able to split train/test data without leaking it

🍳 Real-World Analogy: Prepping Ingredients

Imagine cooking with unwashed vegetables, ingredients still in their packaging, and spices measured in totally different units — pinches, kilograms, cups. The meal would be a disaster, no matter how good the recipe is.

Data preprocessing is the prep work before cooking. Filling missing values is like replacing the one carrot that went bad. Scaling is putting every ingredient on the same measuring scale so none overwhelms the dish. Encoding is translating "a handful of basil" into an exact number the recipe can use. And splitting train/test is tasting with a clean spoon you never dipped back in the pot — so your judgement stays honest.

⚠️ Golden Rule: "Garbage in, garbage out." Data scientists spend 60–80% of their time here because the best algorithm on earth cannot rescue badly prepared data.

1 Handling Missing Values

A missing value is a hole in your data — a survey question left blank, a sensor that dropped out, a field that was never filled in. In Python you'll see these as None (or NaN in pandas). Most ML algorithms crash or misbehave if you feed them holes, so you must deal with every one.

You have three everyday strategies:

• Fill with the mean — the average of the values you do have. Good for roughly symmetric numeric data.
• Fill with the median — the middle value. Better when the data is skewed (a few huge salaries pull the mean upward).
• Drop the row — delete records that have holes. Only safe when very few rows are affected.

Read the worked example below line by line, then run it. Every line states the result in a comment.

# Handling Missing Data — the #1 real-world data problem
# Real datasets are NEVER perfectly clean. A "missing value" is a hole
# in your data — here we mark holes with Python's None.

# A small table stored as plain lists (no pandas needed)
age    = [25, None, 35, 28, None, 42, 31, None, 38, 29]
salary = [50000, 60000, None, 55000, 70000, None, 65000, 48000, 72000, 51000]

# 1) Count the holes in each column
def count_missing(col):
    return sum(1 for v in col if v is None)   # None counts as missing

print("age missing   :", count_missing(age))     # 3
print("salary missing:", count_missing(salary))  # 2

# 2) Fill holes with the MEAN (average of the values we DO have)
present = [v for v in age if v is not None]       # drop the Nones first
mean_age = sum(present) / len(present)            # average of what's left
filled_age = [v if v is not None else round(mean_age, 1) for v in age]
print("mean age      :", round(mean_age, 1))      # 32.6
print("filled age    :", filled_age)

# 3) Fill holes with the MEDIAN (middle value — better for skewed data)
present_sal = sorted(v for v in salary if v is not None)
median_sal = present_sal[len(present_sal) // 2]   # middle of the sorted list
print("median salary :", median_sal)              # 60000

# 4) Or DROP any row that has a hole (use sparingly!)
rows = list(zip(age, salary))
complete = [r for r in rows if None not in r]
print("rows kept     :", len(complete), "of", len(rows))  # 5 of 10

2 Scaling & Normalisation

Scaling means rewriting every feature so they all cover a similar range. Why bother? Imagine one column is age (20–60) and another is salary (35,000–140,000). Many algorithms measure distance between numbers — and salary's huge values would completely dominate, drowning out age entirely. Scaling levels the playing field.

Two methods cover almost everything:

Min-Max scaling → range 0 to 1

(x - min) / (max - min)

The smallest value becomes 0, the largest becomes 1. Great for neural networks and any bounded input.

Standardisation → mean 0, std 1

(x - mean) / std

Re-centres data around 0. The "std" (standard deviation) measures spread. Great for linear models, SVMs, and k-NN.

# Feature Scaling — put every feature on the SAME scale
# Without it, big-number features (salary) drown out small ones (age).

ages    = [25, 30, 35, 40, 45, 50, 55, 60]
salaries = [35000, 45000, 55000, 70000, 85000, 110000, 125000, 140000]

# --- Min-Max scaling: squashes every value into the range 0..1 ---
# formula:  (x - min) / (max - min)
def min_max(col):
    lo, hi = min(col), max(col)
    return [round((x - lo) / (hi - lo), 2) for x in col]

print("ages  min-max  :", min_max(ages))      # starts at 0.0, ends at 1.0
print("salary min-max :", min_max(salaries))  # same 0..1 range now

# --- Standardisation (z-score): mean becomes 0, spread becomes 1 ---
# formula:  (x - mean) / std
def standardise(col):
    mean = sum(col) / len(col)
    var = sum((x - mean) ** 2 for x in col) / len(col)   # average squared gap
    std = var ** 0.5                                       # square root = std dev
    return [round((x - mean) / std, 2) for x in col]

print("ages  z-score  :", standardise(ages))      # centred around 0
print("salary z-score :", standardise(salaries))  # centred around 0

# Expected output (first three lines):
# ages  min-max  : [0.0, 0.14, 0.29, 0.43, 0.57, 0.71, 0.86, 1.0]
# salary min-max : [0.0, 0.1, 0.19, 0.33, 0.48, 0.71, 0.86, 1.0]
# ages  z-score  : [-1.53, -1.09, -0.65, -0.22, 0.22, 0.65, 1.09, 1.53]

# 🎯 YOUR TURN — implement min-max scaling yourself
# Fill in the blanks marked with ___ to squash every score into 0..1.

scores = [40, 55, 70, 85, 100]

# 1) Find the smallest and largest values
lo = ___        # 👉 replace ___ with min(scores)
hi = ___        # 👉 replace ___ with max(scores)

# 2) Apply the min-max formula to each score:  (x - lo) / (hi - lo)
scaled = [round((x - lo) / (hi - lo), 2) for x in scores]

print("scaled:", scaled)

# ✅ Expected output:
# scaled: [0.0, 0.25, 0.5, 0.75, 1.0]

3 Encoding Categorical Data

A categorical feature is text that names a group — a city, a plan, a colour. ML maths needs numbers, so you must translate categories into numbers. How you translate depends on whether the categories have an order.

• Ordinal categories have a real order: Basic < Pro < Premium, or low < medium < high. Use label encoding — one number per category (0, 1, 2…). The order you assign actually means something.
• Nominal categories have no order: London, Paris, Berlin. Use one-hot encoding — one column per category with a single 1 and the rest 0. This avoids implying a fake ranking.

# Encoding Categorical Data — ML maths only understands NUMBERS
# So every text category has to become a number first.

cities = ["London", "Paris", "Berlin", "London", "Paris"]
plans  = ["Basic", "Pro", "Premium", "Pro", "Basic"]

# --- Label encoding: ONE number per category. Use for ORDINAL data ---
# "ordinal" = the categories have a real order (Basic < Pro < Premium).
plan_rank = {"Basic": 0, "Pro": 1, "Premium": 2}   # you choose the order
plan_codes = [plan_rank[p] for p in plans]
print("plans  :", plans)
print("encoded:", plan_codes)        # [0, 1, 2, 1, 0]

# --- One-hot encoding: ONE column per category, a single 1, rest 0 ---
# Use for NOMINAL data (no order — London is not "less than" Paris).
categories = sorted(set(cities))     # ['Berlin', 'London', 'Paris']
print("\ncategories:", categories)
for city in cities:
    row = [1 if city == c else 0 for c in categories]
    print(city.ljust(7), "->", row)

# Expected output:
# plans  : ['Basic', 'Pro', 'Premium', 'Pro', 'Basic']
# encoded: [0, 1, 2, 1, 0]
#
# categories: ['Berlin', 'London', 'Paris']
# London  -> [0, 1, 0]
# Paris   -> [0, 0, 1]
# Berlin  -> [1, 0, 0]
# London  -> [0, 1, 0]
# Paris   -> [0, 0, 1]

# 🎯 YOUR TURN — build a one-hot encoder
# Each colour should become a list with a single 1 and the rest 0.

colours = ["red", "blue", "green", "red"]

# 1) Get the sorted list of unique categories
categories = sorted(set(colours))   # ['blue', 'green', 'red']

# 2) For each colour, put a 1 in the matching slot, 0 everywhere else
for colour in colours:
    # 👉 replace ___ with: 1 if colour == c else 0
    one_hot = [___ for c in categories]
    print(colour.ljust(6), "->", one_hot)

# ✅ Expected output:
# red    -> [0, 0, 1]
# blue   -> [1, 0, 0]
# green  -> [0, 1, 0]
# red    -> [0, 0, 1]

4 Train/Test Split & Avoiding Data Leakage

To know if a model actually learned (rather than just memorised), you hold back some data. You train on most of it (commonly 80%) and test on the rest (20%) — data the model has never seen. Always shuffle first so the split isn't accidentally ordered by date or class.

Data leakage is the silent killer of ML projects. It happens whenever information from the test set sneaks into training. The most common cause is scaling before splitting: if you compute the min and max over the whole dataset, those numbers already "know" about your test rows.

# Train/Test Split — and the trap that ruins real projects: LEAKAGE
# Rule: the model must NEVER see the test data while it is learning.

import random
random.seed(42)                       # makes the shuffle repeatable

data = list(range(1, 11))             # 10 samples: 1..10
random.shuffle(data)                  # ALWAYS shuffle before splitting
print("shuffled:", data)

# 80% train, 20% test
cut = int(len(data) * 0.8)            # 8
train, test = data[:cut], data[cut:]
print("train   :", train)             # 8 samples
print("test    :", test)              # 2 samples

# --- The right way: fit the scaler on TRAIN ONLY, then reuse it on TEST ---
lo, hi = min(train), max(train)       # learned from training data only
def scale(x):
    return round((x - lo) / (hi - lo), 2)

print("train scaled:", [scale(x) for x in train])
print("test scaled :", [scale(x) for x in test])  # reuses TRAIN's lo/hi

print("\nLEAKAGE happens if you compute lo/hi from ALL data BEFORE splitting")
print("— the test set has secretly influenced training. Always split first.")

🐼 The Real-World Version: pandas + scikit-learn

You just implemented every step by hand so you understand exactly what happens. In real projects you'd reach for pandas and scikit-learn, which do the same work in a handful of lines — and crucially, enforce the "fit on train, transform on test" rule for you.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer

df = pd.read_csv("customers.csv")

# 1) Split FIRST — before any scaling or fitting (prevents leakage)
X = df[["age", "salary", "city"]]
y = df["churned"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)
# Expected: X_train has ~80% of rows, X_test the other ~20%

# 2) Fill missing numbers with the column mean — FIT on train only
imputer = SimpleImputer(strategy="mean")
X_train[["age", "salary"]] = imputer.fit_transform(X_train[["age", "salary"]])
X_test[["age", "salary"]]  = imputer.transform(X_test[["age", "salary"]])
# Expected: no NaNs remain in age/salary

# 3) Standardise numbers — FIT on train, then reuse on test
scaler = StandardScaler()
X_train[["age", "salary"]] = scaler.fit_transform(X_train[["age", "salary"]])
X_test[["age", "salary"]]  = scaler.transform(X_test[["age", "salary"]])
# Expected: train columns now have mean ~0 and std ~1

# 4) One-hot encode the nominal "city" column
encoder = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
city_train = encoder.fit_transform(X_train[["city"]])
city_test  = encoder.transform(X_test[["city"]])
# Expected: one 0/1 column per city, e.g. city_Berlin, city_London, city_Paris

Notice the pattern repeated four times: fit_transform on train, then transform on test. That single discipline is what keeps your evaluation honest.

🎯 Mini-Challenge: Median Imputation

Time to fly with less support. Fill in the missing temperatures using the median of the values that are present. The starter below is just a comment outline — write the logic yourself.

# 🎯 MINI-CHALLENGE: fill missing values with the MEDIAN
#
# temps below has two holes (None). Replace each None with the median
# of the values that are NOT None.
#
# Steps:
#   1. Build a list of only the non-None values
#   2. Sort it
#   3. The median is the middle item:  sorted_list[len(sorted_list) // 2]
#   4. Build a new list, swapping each None for that median
#   5. print the filled list
#
# ✅ Expected output:
# filled: [18, 21, 20, 24, 20, 20, 19]
# (present values sorted: [18, 19, 20, 21, 24] -> middle item is 20)

temps = [18, 21, None, 24, 20, None, 19]

# your code here

! Common Errors (And How to Fix Them)

These four mistakes quietly wreck more ML projects than any algorithm bug. Learn to spot them.

❌ Scaling before the split (data leakage)

Fitting the scaler on the whole dataset lets the test set influence training. Scores look great, then collapse in production.

# ❌ WRONG — min/max computed over ALL data, then split
scaled = (data - data.min()) / (data.max() - data.min())
train, test = split(scaled)

✅ Fix: split first, fit on train, apply to test:

train, test = split(data)             # split FIRST
lo, hi = train.min(), train.max()     # learn from train only
train_s = (train - lo) / (hi - lo)
test_s  = (test  - lo) / (hi - lo)    # reuse train's lo/hi

❌ Fitting the scaler on the test set

Calling fit (or recomputing min/max) on test data is leakage in disguise — the test set is supposed to be unseen.

scaler.fit(X_test)        # ❌ never fit on test
X_test = scaler.transform(X_test)

✅ Fix: only ever transform the test set:

scaler.fit(X_train)              # ✅ fit on train
X_test = scaler.transform(X_test)  # ✅ transform only

❌ Dropping every row with a missing value

If 30% of rows have one hole each, dropping them throws away a third of your data — and may bias what's left.

clean = [r for r in rows if None not in r]  # ❌ lost most rows

✅ Fix: impute (fill) instead, drop only when <5% affected:

# Fill numeric holes with the mean/median, keep the rows
filled = [v if v is not None else mean for v in column]

Mapping unordered categories to 0,1,2 invents a fake ranking the model will believe.

{"London": 0, "Paris": 1, "Berlin": 2}  # ❌ implies Berlin > London

✅ Fix: one-hot encode anything without a natural order:

# London -> [0,1,0], Paris -> [0,0,1], Berlin -> [1,0,0]
[1 if city == c else 0 for c in sorted(set(cities))]

Reserve label encoding for genuinely ordinal data (low/medium/high).

📋 Quick Reference

🎉 Lesson Complete!

You can now spot and fill missing values, scale features with min-max and z-score, encode categories the right way, and split data without leaking it into your evaluation. You implemented every step in plain Python, so the pandas and scikit-learn versions will feel like shortcuts rather than magic.

Practice quiz

Continue this course

• Previous: Python for Machine Learning
• Next: Linear Regression — Predict continuous values — your first machine learning model
• Quick reference: AI & Machine Learning cheat sheet › Data Preparation
• From the blog: Getting Started with Machine Learning in Python · Data Preprocessing for Machine Learning

Frequently asked questions

Should I scale my data before or after the train/test split?

After. Split first, fit the scaler on the training set only, then apply that same scaler to the test set. Fitting on all the data before splitting lets test information leak into training and inflates your scores.

What is the difference between min-max scaling and standardisation?

Min-max scaling squashes values into a fixed 0-to-1 range using (x - min) / (max - min). Standardisation (z-score) re-centres values so the mean is 0 and the standard deviation is 1 using (x - mean) / std. Use min-max for bounded inputs like neural networks; use z-score for linear models, SVMs, and k-NN.

When do I use label encoding versus one-hot encoding?

Use label encoding for ordinal categories that have a real order (low/medium/high). Use one-hot encoding for nominal categories with no order (London/Paris/Berlin), because label encoding would wrongly imply Paris > London.

Is it OK to just delete rows that have missing values?

Only if very few rows are affected (roughly under 5%). Dropping rows blindly throws away real signal and can bias your dataset. For numeric columns, filling with the mean (normal data) or median (skewed data) usually keeps more information.

Do I need pandas and scikit-learn to do preprocessing?

No — every step here is plain arithmetic you can do with lists and dictionaries, which is why this lesson implements them by hand. In real projects you reach for pandas (fillna) and scikit-learn (MinMaxScaler, StandardScaler, OneHotEncoder, train_test_split) because they are faster and battle-tested.