Hyperparameter Tuning

Definition

How do you find the best settings without trying every combination?

In simple words

Grid search is checking every seat in a cinema for your lost phone, row by row. Random search checks seats at random and, surprisingly often, finds it sooner, because what matters is usually one or two settings. Bayesian search remembers where you've looked and where it was 'warmer'.

Why it matters

Good tuning can turn an average model into a strong one, but it is also the easiest place to overfit the validation data. A disciplined search, with cross-validation and a final untouched test set, gives gains you can trust.

How it works

In code

Grid search and random search over the same budget for a random forest, both scored by cross-validation and checked once on a held-out test set. Optuna works the same way on your own machine.

from scipy.stats import randint
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
forest = RandomForestClassifier(n_estimators=100, random_state=0)

grid = GridSearchCV(forest, {"max_depth": [3, 6, None], "min_samples_leaf": [1, 5, 10]}, cv=4)
grid.fit(X_tr, y_tr)
print("Grid (9 tries):  ", grid.best_params_, f"CV {grid.best_score_:.3f}  test {grid.score(X_te, y_te):.3f}")

space = {"max_depth": randint(2, 20), "min_samples_leaf": randint(1, 20), "max_features": ["sqrt", "log2", None]}
rand = RandomizedSearchCV(forest, space, n_iter=9, cv=4, random_state=0)
rand.fit(X_tr, y_tr)
print("Random (9 tries):", rand.best_params_, f"CV {rand.best_score_:.3f}  test {rand.score(X_te, y_te):.3f}")

Grid (9 tries):   {'max_depth': 6, 'min_samples_leaf': 1} CV 0.965  test 0.944
Random (9 tries): {'max_depth': 14, 'max_features': 'log2', 'min_samples_leaf': 1} CV 0.965  test 0.944

Try it yourself

Common mistakes

• Tuning on the test set.
• Searching a linear grid for parameters that act on a log scale.
• Reporting the best cross-validation score as the expected real-world score. It is optimistic.

Check yourself

Summary

• Hyperparameters are chosen before training; tuning searches them with cross-validation.
• Grid for small spaces, random for many parameters, Bayesian for expensive models.
• Use log scales and keep the test set for one final check.