Imbalanced Data and Threshold Tuning

Definition

What if only 1 in 1,000 payments is fraud?

Imbalanced classification is when one class is far rarer than the other. Remedies include class weights that make errors on the rare class cost more, resampling (oversampling the minority, for example with SMOTE, or undersampling the majority) applied only to training folds, and, most importantly, choosing the decision threshold to minimise the expected cost of errors rather than using 0.5.

In simple words

A model trained on 999 normal payments and 1 fraud learns that 'normal' is almost always right. Class weights say 'missing a fraud costs 100 times more than a false alarm'. Even then, the default threshold of 0.5 is rarely right: if a missed fraud costs ₹10,000 and a check costs ₹50, flag payments at a much lower probability.

Why it matters

Fraud, disease, defects, churn and security attacks are all rare-event problems. Handling imbalance correctly is the difference between a model that looks good and one that saves money or lives.

How it works

Use stratified splits and metrics such as recall, precision, PR-AUC or cost, never accuracy alone.
Try class_weight="balanced" before resampling.
If you resample, do it inside the training folds only.
Choose the threshold on validation data by minimising expected cost.

The maths

Expected cost at threshold t

Cost (t) = c_{FP} \cdot FP (t) + c_{FN} \cdot FN (t)

Cost-optimal threshold for calibrated probabilities

t^{*} = \frac{c _{FP}}{c _{FP} + c _{FN}}

Balanced class weight

w_{k} = \frac{n}{K \cdot n _{k}}

$n_{k}$: Number of examples in class k
$K$: Number of classes

In code

A fraud-like problem with 1 per cent positives. Class weights and a threshold chosen from costs on training folds each cut the cost, and together they cut it most. Watch accuracy fall as the cost improves: accuracy is the wrong guide here.

imbalanced.py

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix, recall_score
from sklearn.model_selection import StratifiedKFold, cross_val_predict, train_test_split

X, y = make_classification(n_samples=60000, n_features=12, n_informative=6, weights=[0.99], flip_y=0, class_sep=1.2, random_state=1)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
COST_FP, COST_FN = 50, 10000                     # rupees: a manual check against a missed fraud
cv = StratifiedKFold(5, shuffle=True, random_state=0)
grid = np.linspace(0.001, 0.99, 400)

def cost(y_true, pred):
    tn, fp, fn, tp = confusion_matrix(y_true, pred).ravel()
    return int(fp * COST_FP + fn * COST_FN)

print(f"Positives in test: {y_te.sum()} of {len(y_te)}")
for weight in [None, "balanced"]:
    model = LogisticRegression(class_weight=weight, max_iter=2000)
    oof = cross_val_predict(model, X_tr, y_tr, cv=cv, method="predict_proba")[:, 1]   # training data only
    best = min(grid, key=lambda t: cost(y_tr, (oof >= t).astype(int)))
    p = model.fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
    for t in [0.5, best]:
        pred = (p >= t).astype(int)
        print(f"{str(weight):>8} weights, threshold {t:.3f}: accuracy {accuracy_score(y_te, pred):.3f}  recall {recall_score(y_te, pred):.2f}  cost Rs {cost(y_te, pred):,}")
print(f"Theory for calibrated probabilities: threshold {COST_FP / (COST_FP + COST_FN):.4f}")

Output

Positives in test: 180 of 18000
    None weights, threshold 0.500: accuracy 0.996  recall 0.62  cost Rs 690,400
    None weights, threshold 0.006: accuracy 0.891  recall 0.91  cost Rs 266,850
balanced weights, threshold 0.500: accuracy 0.921  recall 0.92  cost Rs 210,100
balanced weights, threshold 0.343: accuracy 0.889  recall 0.95  cost Rs 189,550
Theory for calibrated probabilities: threshold 0.0050

Try it yourself

Handle a rare-event problem

Put the steps in a sensible order.

1Report recall, precision and cost on the test set
2Make stratified train, validation and test splits
3Train with class weights (or resample inside training folds)
4Pick the threshold that minimises cost on out-of-fold predictions from the training data
5Agree the cost of a false positive and a false negative

Common mistakes

Oversampling before splitting, so copies of the same fraud appear in train and test.
Celebrating 99 per cent accuracy on 1 per cent positives.
Keeping the 0.5 threshold when error costs are unequal.

Check yourself

1. If a false positive costs 1 and a false negative costs 99, the cost-optimal threshold for calibrated probabilities is…

2. When should resampling be applied?

3. What does class_weight="balanced" do?

Summary

Accuracy hides failure on rare classes; use recall, precision, PR-AUC and cost.
Class weights and in-fold resampling help the model learn the rare class.
Choose the threshold from costs, on validation data.

Finished reading?

Sources: imbalanced-learn documentation · scikit-learn: Tuning the decision threshold