Logistic Regression | AI Pathshala

Definition

Despite its name, logistic regression is for classification. It answers yes-or-no questions with a probability.

Logistic regression is a classification algorithm that computes a weighted sum of the inputs and passes it through the sigmoid function, giving a probability between 0 and 1. It's trained by minimising log loss.

In simple words

Each clue about an email adds or takes away points: "free" adds 2, a known sender takes away 3. Total the points. Then the sigmoid squashes any total into a chance between 0% and 100%. A very high score gets close to 100%, a very low score gets close to 0%.

Why it matters

It's fast and explainable, and it gives probabilities rather than just labels, which is why banks and hospitals use it. It's the standard baseline for classification. A single neuron with a sigmoid is exactly logistic regression, so it's also your first step to neural networks.

How it works

Compute the score: a weighted sum of the features plus a bias.
Apply the sigmoid to turn the score into a probability.
Choose a threshold, usually 0.5, to get a yes or no.
Train by reducing log loss with gradient descent.

Why not plain linear regression?

A straight line can predict values below 0 or above 1, which make no sense as probabilities. It's also pulled around badly by outliers. The sigmoid keeps the output in range.

The maths

Sigmoid function

σ (z) = \frac{1}{1 + e ^{- z}}

$z$: The score, any number
$σ (z)$: A probability between 0 and 1

The model

P (y = 1 ∣ x) = σ (w^{⊤} x + b)

Log loss

J = - \frac{1}{n} i = 1 \sum n [y_{i} lo g \overset{y}{^}_{i} + (1 - y_{i}) lo g (1 - \overset{y}{^}_{i})]

Weights: w₁ = 2 for the word "free", w₂ = −3 for a known sender, and b = −1. An email contains "free" from an unknown sender, so x = [1, 0].

z = 2(1) + (−3)(0) + (−1) = 1
σ(1) = 1 / (1 + e⁻¹) = 1 / 1.368 ≈ 0.731
P(spam) = 73.1%, above 0.5, so: spam

If it really is spam, the loss = −ln(0.731) ≈ 0.313

In code

logistic.py

import math
from sklearn.linear_model import LogisticRegression

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

for z in (-5, -1, 0, 1, 5):
    print(f"sigmoid({z:>2}) = {sigmoid(z):.3f}")

hours = [[1], [2], [3], [4], [5], [6], [7], [8]]
passed = [0, 0, 0, 0, 1, 1, 1, 1]
model = LogisticRegression().fit(hours, passed)
for h in (2, 4.5, 7):
    print(f"Studied {h} hours: chance of passing = {model.predict_proba([[h]])[0][1]:.2f}")

Output

sigmoid(-5) = 0.007
sigmoid(-1) = 0.269
sigmoid( 0) = 0.500
sigmoid( 1) = 0.731
sigmoid( 5) = 0.993
Studied 2 hours: chance of passing = 0.05
Studied 4.5 hours: chance of passing = 0.50
Studied 7 hours: chance of passing = 0.95

Common mistakes

Thinking it does regression because of the name.
Keeping a threshold of 0.5 without thinking about the cost of each error.
Skipping feature scaling, which can slow training and make weights hard to compare.

Check yourself

1. What is σ(0)?

2. Logistic regression is used for:

3. For cancer screening, should the threshold usually be lower or higher than 0.5?

Summary

Weighted sum + sigmoid = a probability.
Trained by minimising log loss.
Choose the threshold based on what your errors cost.

Finished reading?

Sources: scikit-learn user guide · Google Machine Learning Crash Course