ROC-AUC and Precision-Recall Curves

Definition

Which of two models is better at every threshold, not just at 0.5?

A ROC curve plots the true positive rate (recall) against the false positive rate for every decision threshold; the area under it, ROC-AUC, is the probability that the model ranks a random positive above a random negative. A precision-recall (PR) curve plots precision against recall; its summary, average precision, is more informative than ROC-AUC when positives are rare.

In simple words

Instead of judging a model at one threshold, slide the threshold from strict to lenient and record what happens. A perfect model's ROC curve hugs the top-left corner (AUC 1); random guessing gives the diagonal (AUC 0.5). When only 1 in 1,000 cases is positive, the PR curve shows how painful the false alarms really are.

Why it matters

Curves separate how well a model ranks cases from where you set the threshold. They let you compare models fairly and then choose an operating point that meets a business constraint, such as 'at most 2 per cent false positives'.

How it works

	ROC curve	Precision-recall curve
Axes	TPR vs FPR	Precision vs recall
Random model	Diagonal, AUC 0.5	Flat at the positive rate
Best when	Classes are roughly balanced	Positives are rare
Summary	ROC-AUC	Average precision

The maths

Rates on the ROC axes

TPR = \frac{TP}{TP + FN}, FPR = \frac{FP}{FP + TN}

AUC as a ranking probability

AUC = P (s (x^{+}) > s (x^{-}))

$s (x)$: The model's score for an example

In code

ROC-AUC and average precision for two models on a rare-positive problem, a few points on each curve, and AUC computed as a ranking probability.

roc_pr.py

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, precision_recall_curve, roc_auc_score, roc_curve
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=5000, n_features=20, weights=[0.95], random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.4, stratify=y, random_state=0)
print("Positive rate in test:", round(y_te.mean(), 3))

for name, model in [("logistic", LogisticRegression(max_iter=1000)), ("forest", RandomForestClassifier(200, random_state=0))]:
    s = model.fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
    print(f"{name:>8}: ROC-AUC {roc_auc_score(y_te, s):.3f}  average precision {average_precision_score(y_te, s):.3f}")

fpr, tpr, _ = roc_curve(y_te, s)
for target in [0.01, 0.05, 0.2]:
    print(f"At FPR {target:.2f}, the forest's TPR is {tpr[np.searchsorted(fpr, target)]:.2f}")
precision, recall, _ = precision_recall_curve(y_te, s)
print(f"At recall 0.8 or more, best precision is {precision[recall >= 0.8].max():.2f}")

pos, neg = s[y_te == 1], s[y_te == 0]
print("AUC as ranking probability:", round(float(np.mean(pos[:, None] > neg[None, :])), 3))

Output

Positive rate in test: 0.055
logistic: ROC-AUC 0.952  average precision 0.897
  forest: ROC-AUC 0.943  average precision 0.898
At FPR 0.01, the forest's TPR is 0.88
At FPR 0.05, the forest's TPR is 0.89
At FPR 0.20, the forest's TPR is 0.93
At recall 0.8 or more, best precision is 0.98
AUC as ranking probability: 0.932

Try it yourself

Read the curve

Match each curve description to what it tells you.

ROC curve along the diagonal
ROC curve hugging the top-left corner
High ROC-AUC but low average precision
One model's ROC curve is above another's everywhere

Common mistakes

Relying on ROC-AUC when positives are very rare.
Computing curves from hard 0/1 predictions instead of scores.
Choosing the threshold on the test set.

Check yourself

1. A ROC-AUC of 0.5 means…

2. Which curve is better for rare positives?

3. What do the curves need from the model?

Summary

ROC: TPR against FPR; AUC is a ranking probability.
PR curves and average precision suit rare positives.
Compare models by curves; pick thresholds on validation data.