Data Preprocessing | AI Pathshala

Definition

Why does a model struggle when salary is in lakhs and age is in years?

Data preprocessing transforms raw features into a form a model can learn from: imputing missing values, scaling numeric features to comparable ranges, encoding categorical features as numbers, and handling outliers. Every transformation must be fitted on training data only and then applied unchanged to validation and test data.

In simple words

Distance-based and gradient-based models treat a change of 10,000 in salary as far bigger than a change of 10 in age, purely because of units. Standardisation puts every feature on the same footing: subtract the mean and divide by the standard deviation. Categories such as city become one column per city, holding 0 or 1.

Why it matters

Preprocessing often changes results more than the choice of algorithm. Doing it inside a pipeline, fitted only on training folds, is also the main defence against data leakage.

How it works

Problem	Common fix	scikit-learn
Different scales	Standardise or min-max scale	StandardScaler, MinMaxScaler
Missing values	Impute the median or most frequent value	SimpleImputer
Categories	One-hot encode	OneHotEncoder
Outliers	Robust scaling, clipping or transforming	RobustScaler
Mixed column types	Apply different steps per column	ColumnTransformer

The maths

Standardisation (z-score)

x^{'} = \frac{x - μ}{σ}

$μ$: The mean of the feature in the training data
$σ$: Its standard deviation in the training data

Min-max scaling to [0, 1]

x^{'} = \frac{x - x _{m i n}}{x _{m a x} - x _{m i n}}

In code

A ColumnTransformer that imputes and scales numbers and one-hot encodes a category, fitted on training rows only.

preprocessing.py

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

train = pd.DataFrame({
    "age": [25, 32, np.nan, 47, 51],
    "salary": [300000, 520000, 410000, np.nan, 1200000],
    "city": ["Pune", "Delhi", "Pune", "Chennai", "Delhi"],
})
test = pd.DataFrame({"age": [29], "salary": [450000], "city": ["Kolkata"]})

numeric = make_pipeline(SimpleImputer(strategy="median"), StandardScaler())
categorical = OneHotEncoder(handle_unknown="ignore")
prep = ColumnTransformer([("num", numeric, ["age", "salary"]), ("cat", categorical, ["city"])])

prep.fit(train)                                   # learn medians, means, std and categories from training only
print("Output columns:", list(prep.get_feature_names_out()))
print("Training rows:")
print(np.round(prep.transform(train), 2))
print("Test row (an unseen city becomes all zeros):")
print(np.round(prep.transform(test), 2))

Output

Output columns: ['num__age', 'num__salary', 'cat__city_Chennai', 'cat__city_Delhi', 'cat__city_Pune']
Training rows:
[[-1.46 -0.87  0.    0.    1.  ]
 [-0.72 -0.19  0.    1.    0.  ]
 [ 0.06 -0.53  0.    0.    1.  ]
 [ 0.85 -0.36  1.    0.    0.  ]
 [ 1.27  1.95  0.    1.    0.  ]]
Test row (an unseen city becomes all zeros):
[[-1.04 -0.4   0.    0.    0.  ]]

Try it yourself

Which transform?

Match each data problem to the right preprocessing step.

Income ranges from 0 to 10,000,000; age from 18 to 90
10 per cent of ages are blank
A column of city names
Numbers and categories need different steps
A few extreme incomes distort the scale

Common mistakes

Fitting the scaler on all data before splitting, which leaks test statistics.
Label-encoding unordered categories as 1, 2, 3, which invents a false order.
Dropping every row with a missing value and losing most of the data.

Check yourself

1. Where should a scaler learn its mean and standard deviation?

2. Which model needs feature scaling least?

3. After standardisation, a training feature has…

Summary

Impute, scale, encode and handle outliers before modelling.
ColumnTransformer applies the right steps to each column type.
Fit every step on training data only.