Model Interpretability | AI Pathshala

Definition

Why was this loan application rejected? Can the model explain itself?

Model interpretability methods explain what a model has learned. Global methods describe overall behaviour: permutation importance measures how much the score drops when a feature's values are shuffled, and partial dependence shows how the average prediction changes with a feature. Local methods explain single predictions: SHAP values split a prediction into additive contributions from each feature, based on Shapley values from game theory.

In simple words

Shuffle one column, say income, so it becomes meaningless. If accuracy collapses, the model relied on income. For one customer, SHAP says something like: the average approval score is 0.6; low income pulled it down by 0.2; a long credit history pushed it up by 0.1; final score 0.5.

Why it matters

Explanations are needed to debug models, catch leakage and bias, build trust with users, and meet legal duties to explain automated decisions in areas such as lending, hiring and healthcare.

How it works

Method	Scope	Answers
Coefficients (linear models)	Global	Direction and size of each effect
Permutation importance	Global	Which features the model relies on
Partial dependence	Global	How predictions change with one feature
SHAP	Local and global	Why this prediction, feature by feature

The maths

Permutation importance of feature j

I_{j} = s - \frac{1}{K} k = 1 \sum K s_{k, j}

$s$: The score on intact data
$s_{k, j}$: The score after the k-th shuffle of feature j

SHAP: additive explanation of one prediction

f (x) = ϕ_{0} + j = 1 \sum p ϕ_{j}

In code

Permutation importance for a random forest on held-out data, and a partial dependence curve for the top feature. The shap library adds per-prediction explanations on your own machine.

interpretability.py

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import partial_dependence, permutation_importance
from sklearn.model_selection import train_test_split

data = load_breast_cancer()
X_tr, X_te, y_tr, y_te = train_test_split(data.data, data.target, test_size=0.3, stratify=data.target, random_state=0)
model = RandomForestClassifier(n_estimators=200, random_state=0).fit(X_tr, y_tr)

result = permutation_importance(model, X_te, y_te, n_repeats=10, random_state=0)
order = np.argsort(result.importances_mean)[::-1][:5]
print("Top features by permutation importance (accuracy drop):")
for j in order:
    print(f"  {data.feature_names[j]:<22} {result.importances_mean[j]:.3f} +/- {result.importances_std[j]:.3f}")

top = order[0]
pd_result = partial_dependence(model, X_tr, [top], grid_resolution=5)
grid = pd_result["grid_values"][0]
for value, avg in zip(grid, pd_result["average"][0]):
    print(f"  {data.feature_names[top]} = {value:7.2f} -> average P(benign) {avg:.2f}")

Output

Top features by permutation importance (accuracy drop):
  worst radius           0.021 +/- 0.005
  worst area             0.019 +/- 0.007
  worst texture          0.016 +/- 0.004
  worst perimeter        0.012 +/- 0.007
  worst concave points   0.012 +/- 0.008
  worst radius =   10.47 -> average P(benign) 0.67
  worst radius =   14.23 -> average P(benign) 0.67
  worst radius =   18.00 -> average P(benign) 0.51
  worst radius =   21.77 -> average P(benign) 0.50
  worst radius =   25.54 -> average P(benign) 0.50

Try it yourself

Read the explanation

Match each question to the method that answers it.

Which features does the model rely on overall?
Why was this one application rejected?
How does the predicted risk change as age increases?
In a linear model, what does a one-unit increase do?

Common mistakes

Computing permutation importance on training data, which rewards memorised features.
Reading importances of correlated features separately. Shuffling one leaves its twin.
Treating explanations as causal proof.

Check yourself

1. Permutation importance measures…

2. SHAP values for one prediction add up to…

3. A feature has high importance. Does it cause the outcome?

Summary

Global methods: coefficients, permutation importance, partial dependence.
Local method: SHAP splits one prediction into feature contributions.
Compute on held-out data, beware correlation, and don't read causation.

Finished reading?

Sources: scikit-learn: Permutation importance · Lundberg and Lee, A Unified Approach to Interpreting Model Predictions (2017) · Molnar, Interpretable Machine Learning