TF-IDF

Definition

In simple words

In a collection of cricket articles, "the" appears everywhere, so it says nothing. "Cricket" appears in all of them too. But "Dhoni" appears often in only a few, and tells you a lot about those. TF-IDF rewards the words that make a document different.

Why it matters

It's a simple, fast and explainable way to turn text into numbers. It's still a strong baseline for text classification, and related ideas such as BM25 power keyword search and hybrid RAG.

How it works

1. TF: count how often the word appears in this document.
2. IDF: check how rare the word is across all documents.
3. Multiply the two.

The maths

10 documents. "cricket" appears 3 times in document 1, and in 2 of the 10 documents. "the" appears 20 times in document 1, and in all 10. Using log base 10:

cricket: 3 × log(10/2)  = 3 × 0.699 = 2.10
the:     20 × log(10/10) = 20 × 0   = 0

"the" scores zero however often it appears.

In code

scikit-learn uses a smoothed version of the formula, so a word in every document gets a small score instead of exactly zero. It still ranks far below the rare words.

from sklearn.feature_extraction.text import TfidfVectorizer

documents = [
    "the match was great and dhoni played well",
    "the weather was great today",
    "the budget speech was long",
]
vectorizer = TfidfVectorizer()
matrix = vectorizer.fit_transform(documents).toarray()
words = vectorizer.get_feature_names_out()

scores = sorted(zip(matrix[0], words), reverse=True)
print("Top words in document 1:", [word for score, word in scores[:3]])
print("Score of 'the':", round(matrix[0][list(words).index("the")], 3))
print("Score of 'dhoni':", round(matrix[0][list(words).index("dhoni")], 3))

Top words in document 1: ['well', 'played', 'match']
Score of 'the': 0.236
Score of 'dhoni': 0.399

Common mistakes

• Fitting the vectoriser on test data. Fit on the training set only.
• Using it where meaning matters more than keywords. Consider embeddings.

Check yourself

Summary

• It rewards words that are frequent here and rare elsewhere.
• tf-idf = tf × log(N/df).
• It's a strong, simple baseline that ignores meaning.