XGBoost, LightGBM and CatBoost

Definition

Why do these three libraries win so many data competitions?

XGBoost, LightGBM and CatBoost are optimised gradient-boosting libraries. XGBoost adds second-order (Newton) updates and explicit regularisation of tree complexity. LightGBM bins features into histograms and grows trees leaf-wise for speed on large data. CatBoost uses ordered boosting and built-in target statistics for categorical features. scikit-learn's HistGradientBoosting implements the histogram approach.

In simple words

They all do gradient boosting; they differ in engineering. Histograms turn a million unique values into 256 buckets, so finding splits is fast. Built-in handling of missing values and categories saves preprocessing. Early stopping on a validation set picks the number of trees automatically.

Why it matters

For structured business data, a tuned gradient-boosting library is usually the model to beat. Knowing their differences helps you choose quickly and tune the parameters that matter.

How it works

Library	Strength	Reach for it when
XGBoost	Mature, regularised, widely deployed	A solid default with strong tooling
LightGBM	Very fast on large data, leaf-wise growth	Millions of rows, many features
CatBoost	Native categorical handling, robust defaults	Many high-cardinality categories
HistGradientBoosting	In scikit-learn, histogram-based	You want no extra dependency

The parameters that matter most: learning rate, number of trees (with early stopping), tree size (max_depth or num_leaves), minimum samples per leaf, and subsampling of rows and columns.

The maths

XGBoost's regularised objective at round m

L^{(m)} = i \sum [g_{i} f_{m} (x_{i}) + \frac{1}{2} h_{i} f_{m} (x_{i})^{2}] + γ T + \frac{1}{2} λ j = 1 \sum T w_{j}^{2}

$g_{i}, h_{i}$: First and second derivatives of the loss
$T$: Number of leaves
$w_{j}$: Leaf weights

Optimal leaf weight

w_{j}^{*} = - \frac{\sum _{i \in j} g _{i}}{\sum _{i \in j} h _{i} + λ}

In code

Histogram gradient boosting with early stopping and native missing-value handling. The same pattern works with xgboost.XGBClassifier or lightgbm.LGBMClassifier on your own machine.

hist_gradient_boosting.py

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier, HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=6000, n_features=30, n_informative=12, random_state=0)
rng = np.random.default_rng(0)
X[rng.random(X.shape) < 0.05] = np.nan            # 5 per cent missing values
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)

hgb = HistGradientBoostingClassifier(learning_rate=0.1, max_iter=500, early_stopping=True,
                                     validation_fraction=0.15, n_iter_no_change=20, random_state=0)
hgb.fit(X_tr, y_tr)                               # missing values are handled natively
print("Trees used after early stopping:", hgb.n_iter_)
print("Histogram boosting test accuracy:", round(hgb.score(X_te, y_te), 3))

X_filled = np.nan_to_num(X_tr, nan=0.0)          # classic GradientBoosting needs no NaNs
gb = GradientBoostingClassifier(n_estimators=hgb.n_iter_, random_state=0).fit(X_filled, y_tr)
print("Classic boosting test accuracy:", round(gb.score(np.nan_to_num(X_te, nan=0.0), y_te), 3))

Output

Trees used after early stopping: 189
Histogram boosting test accuracy: 0.916
Classic boosting test accuracy: 0.872

Try it yourself

Which library when?

Match each situation to the library you'd try first.

50 million rows and speed matters most
Dozens of categorical columns with thousands of values each
No extra libraries allowed in production
A well-tested default with broad deployment support

Common mistakes

Fixing the number of trees by hand instead of early stopping.
One-hot encoding hundreds of categories for CatBoost, which handles them natively.
Tuning everything at once. Start with learning rate, tree size and early stopping.

Check yourself

1. What makes LightGBM fast?

2. What does XGBoost use beyond plain gradient boosting?

3. How should the number of trees be chosen?

Summary

XGBoost, LightGBM and CatBoost are engineered gradient boosting.
Histograms, missing-value handling and early stopping make them fast and practical.
Tune learning rate, tree size and subsampling; let early stopping pick the trees.

Finished reading?

Sources: XGBoost documentation · LightGBM documentation · CatBoost documentation