Classification Metrics in Depth

Definition

When is 99 per cent accuracy useless?

In simple words

If 1 in 100 patients has a disease, a model saying 'healthy' to everyone is 99 per cent accurate and misses every sick patient: recall 0. Precision asks 'when it raises an alarm, how often is it right?'; recall asks 'of all real cases, how many did it catch?'. Raising one usually lowers the other.

Why it matters

Choosing the metric is a product decision. Cancer screening prioritises recall; a spam filter that hides important email needs high precision; credit decisions weigh both with money. Reporting accuracy alone on imbalanced data is a classic error.

How it works

The maths

A fraud model flags 50 transactions. 40 really are fraud. There were 60 frauds in total among 1,000 transactions.

TP = 40, FP = 10, FN = 20, TN = 930
Precision = 40 / 50 = 0.80
Recall    = 40 / 60 = 0.67
F1        = 2 × 0.80 × 0.67 / (0.80 + 0.67) = 0.73
Accuracy  = (40 + 930) / 1000 = 0.97

In code

The worked example computed by hand and checked with scikit-learn, then how precision and recall move as the threshold changes.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, f1_score, precision_score, recall_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

y_true = np.array([1] * 40 + [0] * 10 + [1] * 20 + [0] * 930)
y_pred = np.array([1] * 50 + [0] * 950)
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
print("TP FP FN TN:", tp, fp, fn, tn)
print(f"precision {precision_score(y_true, y_pred):.2f}  recall {recall_score(y_true, y_pred):.2f}  F1 {f1_score(y_true, y_pred):.2f}  specificity {tn / (tn + fp):.3f}")

X, y = load_breast_cancer(return_X_y=True)
y = 1 - y                                         # make 'malignant' the positive class
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
proba = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
print("threshold  precision  recall")
for t in [0.2, 0.5, 0.8]:
    pred = (proba >= t).astype(int)
    print(f"{t:>9}  {precision_score(y_te, pred):9.3f}  {recall_score(y_te, pred):6.3f}")

TP FP FN TN: 40 10 20 930
precision 0.80  recall 0.67  F1 0.73  specificity 0.989
threshold  precision  recall
      0.2      0.824   0.953
      0.5      0.938   0.938
      0.8      1.000   0.891

Try it yourself

Common mistakes

• Reporting accuracy alone on imbalanced data.
• Mixing up which class is 'positive' when computing precision and recall.
• Optimising F1 when the costs of FP and FN are very different. Use F-beta or a cost.

Check yourself

Summary

• Precision: how often alarms are right. Recall: how many positives are caught.
• Specificity, F1 and F-beta complete the picture.
• Choose the metric from the real cost of each error.