Naive Bayes Classifiers

Definition

Which words make a message spam, and how sure can you be from counting them?

In simple words

Count how often each word appears in spam and in normal messages. For a new message, multiply the chances of its words under each class, times how common each class is, and pick the larger. The independence assumption is wrong ('free' and 'prize' often appear together), yet the ranking is usually right.

Why it matters

Naive Bayes trains in one pass, works with very little data, scales to huge vocabularies and is a classic baseline for text classification and spam filtering.

How it works

1. Prior: P(c) = share of training messages in class c.
2. Likelihood: P(word | c) = (count of word in c + α) / (total words in c + α·V).
3. Score each class with log P(c) + Σ log P(wordᵢ | c), adding logs to avoid underflow.
4. Predict the class with the highest score.

The maths

In code

A multinomial Naive Bayes spam filter from scratch with Laplace smoothing, checked against scikit-learn.

import math
from collections import Counter
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

train = [("win a free prize now", "spam"), ("free money click now", "spam"), ("claim your free gift", "spam"),
         ("meeting at noon tomorrow", "ham"), ("lunch with the team", "ham"), ("project notes for the meeting", "ham")]
classes = ["ham", "spam"]
counts = {c: Counter() for c in classes}
for text, label in train:
    counts[label].update(text.split())
vocab = {w for c in classes for w in counts[c]}
prior = {c: sum(1 for _, l in train if l == c) / len(train) for c in classes}

def scores(text, alpha=1.0):
    result = {}
    for c in classes:
        total = sum(counts[c].values())
        s = math.log(prior[c])
        for word in text.split():
            s += math.log((counts[c][word] + alpha) / (total + alpha * len(vocab)))
        result[c] = round(s, 3)
    return result

for message in ["free prize meeting", "team meeting notes", "click to win"]:
    s = scores(message)
    print(f"{message!r:24} log-scores {s} -> {max(s, key=s.get)}")

vec = CountVectorizer().fit([t for t, _ in train])
nb = MultinomialNB(alpha=1.0).fit(vec.transform([t for t, _ in train]), [l for _, l in train])
print("scikit-learn agrees:", list(nb.predict(vec.transform(["free prize meeting", "team meeting notes", "click to win"]))))

'free prize meeting'     log-scores {'ham': -10.174, 'spam': -9.193} -> spam
'team meeting notes'     log-scores {'ham': -8.787, 'spam': -11.272} -> ham
'click to win'           log-scores {'ham': -11.272, 'spam': -9.886} -> spam
scikit-learn agrees: [np.str_('spam'), np.str_('ham'), np.str_('spam')]

Try it yourself

Common mistakes

• Forgetting smoothing, so one unseen word sets a class probability to zero.
• Multiplying many small probabilities instead of adding logs, causing underflow.
• Trusting its probabilities as calibrated. The ranking is good; the numbers are often too extreme.

Check yourself

Summary

• Naive Bayes multiplies the prior by per-feature likelihoods.
• Gaussian, multinomial and Bernoulli variants suit different features.
• Use Laplace smoothing and log probabilities.