Logistic Regression

Definition

How can a weighted sum answer a yes-or-no question, with a probability?

In simple words

Compute a score, like linear regression, then squash it into 0 to 1 with the S-shaped sigmoid. A score of 0 becomes 50 per cent; large positive scores approach 100 per cent. The loss punishes confident wrong answers very heavily, so the model learns calibrated probabilities.

Why it matters

Logistic regression is the standard baseline for classification in medicine, finance and marketing, because it is fast, well understood and interpretable: each weight is a change in log-odds. It is also exactly a one-neuron neural network.

How it works

1. Score: z = wᵀx + b.
2. Probability: p = 1 / (1 + e^(−z)).
3. Loss per example: −[y log p + (1 − y) log(1 − p)].
4. Gradient: Xᵀ(p − y) / n, the same shape as linear regression.
5. Predict the positive class (label 1) when p ≥ threshold, usually 0.5.

The maths

In code

Logistic regression trained from scratch with gradient descent, then compared with scikit-learn on the same data.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
scaler = StandardScaler().fit(X_tr)
X_tr, X_te = scaler.transform(X_tr), scaler.transform(X_te)

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

Xb_tr = np.column_stack([np.ones(len(X_tr)), X_tr])
Xb_te = np.column_stack([np.ones(len(X_te)), X_te])
w = np.zeros(Xb_tr.shape[1])
for step in range(2000):
    p = sigmoid(Xb_tr @ w)
    w -= 0.1 * Xb_tr.T @ (p - y_tr) / len(y_tr)

p_tr = sigmoid(Xb_tr @ w)
loss = -np.mean(y_tr * np.log(p_tr) + (1 - y_tr) * np.log(1 - p_tr))
print("From scratch: training log loss", round(float(loss), 4))
print("From scratch: test accuracy", round(float(np.mean((sigmoid(Xb_te @ w) >= 0.5) == y_te)), 3))

sk = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
print("scikit-learn: test accuracy", round(sk.score(X_te, y_te), 3))
print("Probability for the first test case:", round(float(sk.predict_proba(X_te[:1])[0, 1]), 3))

From scratch: training log loss 0.0478
From scratch: test accuracy 0.965
scikit-learn: test accuracy 0.958
Probability for the first test case: 0.996

Try it yourself

Common mistakes

• Using it for non-linear boundaries without adding features.
• Assuming 0.5 is always the right threshold.
• Interpreting weights of unscaled or correlated features as importance.

Check yourself

Summary

• p = σ(wᵀx + b): a linear score squashed into a probability.
• Trained by minimising convex log loss, with gradient Xᵀ(p − y)/n.
• Weights are changes in log-odds; the boundary is linear.