Gradient Boosting: XGBoost & Friends

Boosting: Learn from Your Mistakes

A random forest trains trees independently and averages. Boosting trains trees in sequence, each one predicting the errors of the ensemble so far:

prediction = tree1(x) + lr * tree2(x) + lr * tree3(x) + ...

Tree 1 makes a rough guess. Tree 2 fits tree 1’s residuals. Tree 3 fits what’s still wrong. Hundreds of small corrections, damped by a learning rate, compound into an extremely accurate model. For tabular data, gradient boosting has been the state of the art for a decade — it’s what wins most Kaggle competitions on spreadsheet-shaped problems.

XGBoost in Practice

pip install xgboost

from xgboost import XGBClassifier

model = XGBClassifier(
    n_estimators=2000,
    learning_rate=0.05,
    max_depth=5,
    early_stopping_rounds=50,
    eval_metric="auc",
)
model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],   # watch a validation set...
    verbose=False,
)
model.best_iteration             # ...and stop when it plateaus

Early stopping is the key trick: set n_estimators high, let the validation score decide when to stop. This needs a third split — train / validation / test — because the validation set now influences training and can no longer give an unbiased final score.

The Hyperparameters That Matter

• learning_rate + n_estimators — lower rate with more trees generalises better; early stopping finds the count for you.
• max_depth — 3 to 8 covers nearly every problem; deeper = more interactions but more overfitting.
• subsample / colsample_bytree — train each tree on a random fraction (≈0.8) of rows/columns to decorrelate trees.

Tune those four and stop. Days spent on the remaining dozen knobs typically buy a third decimal place.

The Modern Lineup

• XGBoost — the battle-tested standard.
• LightGBM — faster on large datasets, native categorical support.
• CatBoost — best automatic handling of categorical features, great defaults.

All expose the scikit-learn API, so swapping between them is a one-line change. A practical workflow: prototype with random forest, then switch to LightGBM or XGBoost with early stopping for the final model.