Decision Trees | AI Pathshala

Definition

A decision tree is a model that predicts by asking a series of yes-or-no questions about the features, following a path from the root to a leaf. It learns which questions to ask by choosing, at each step, the split that best separates the classes, using measures such as Gini impurity or information gain.

In simple words

It's a game of 20 Questions. Is income above 50,000? If yes, is the loan under 5 lakhs? If yes, approve. Each question splits people into smaller, more similar groups. Training is just finding the best questions and the best order.

Why it matters

Trees are easy to explain, work with little preparation, and handle both numbers and categories without scaling. They're also the building block of random forests and XGBoost, which are among the best methods for tabular data.

How it works

Start with all the data at the root.
Try every possible split of every feature.
Choose the split that makes the child groups purest.
Repeat on each child, until a stopping rule applies, such as a maximum depth.
Each leaf predicts its majority class, or its average for regression.

Algorithm	Split measure	Note
ID3	Information gain	Categorical features
C4.5	Gain ratio	Handles numbers and missing values
CART	Gini (or variance for regression)	Used by scikit-learn

The maths

Gini impurity

Gini = 1 - k \sum p_{k}^{2}

$p_{k}$: The share of class k in this group

Entropy

H = - k \sum p_{k} lo g_{2} p_{k}

Information gain

I G = H (parent) - child \sum \frac{n _{child}}{n _{parent}} H (child)

A group has 6 "yes" and 4 "no". One split gives left: 5 yes, 0 no. Right: 1 yes, 4 no.

Parent Gini = 1 − (0.6² + 0.4²) = 1 − 0.52 = 0.48
Left Gini   = 1 − (1² + 0²) = 0
Right Gini  = 1 − (0.2² + 0.8²) = 1 − 0.68 = 0.32
Weighted    = (5/10)(0) + (5/10)(0.32) = 0.16
Impurity falls from 0.48 to 0.16, so this is a good split.

In code

scikit-learn can print the learned tree as readable rules. That's the great strength of this model.

tree.py

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier, export_text

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, random_state=0)

tree = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X_train, y_train)
print(export_text(tree, feature_names=list(iris.feature_names)))
print("Test accuracy:", round(tree.score(X_test, y_test), 3))

Output

|--- petal width (cm) <= 0.80
|   |--- class: 0
|--- petal width (cm) >  0.80
|   |--- petal length (cm) <= 4.95
|   |   |--- class: 1
|   |--- petal length (cm) >  4.95
|   |   |--- class: 2

Test accuracy: 0.895

Common mistakes

Leaving depth unlimited, which leads to overfitting.
Trusting one tree too much. Small data changes can produce a very different tree, so forests are more stable.
Scaling features for a tree. It isn't needed, because trees only compare values with thresholds.

Check yourself

1. What is the Gini impurity of a pure group?

2. A tree is 100% on training and 70% on test. What should you do?

Summary

A tree predicts through a series of yes-or-no questions.
It picks splits that make groups purest (Gini or entropy).
It's easy to read and easy to overfit, so limit its depth.

Finished reading?

Sources: scikit-learn user guide