Decision Trees & Random Forests

A Model Made of Questions

A decision tree learns nested if/else rules: area > 1400? → bedrooms > 2? → predict ₹82L. Training greedily picks, at each node, the split that best separates the data (lowest Gini impurity or MSE). Trees handle non-linear patterns and feature interactions natively, need no feature scaling, and you can read the result like a flowchart:

from sklearn.tree import DecisionTreeClassifier, plot_tree

tree = DecisionTreeClassifier(max_depth=3)
tree.fit(X_train, y_train)
plot_tree(tree, feature_names=X.columns, filled=True)

The Catch: Trees Memorise

Grown without limits, a tree keeps splitting until every leaf holds one sample — 100% training accuracy, terrible generalisation, and a completely different tree if you nudge the data. Classic high variance. You can cap it (max_depth, min_samples_leaf), but there’s a better idea.

Random Forests: Average Away the Noise

Train hundreds of trees, each on a random bootstrap sample of the rows and a random subset of features at each split, then vote:

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(
    n_estimators=300,      # number of trees
    max_features="sqrt",   # features considered per split
    n_jobs=-1,             # use every CPU core
    random_state=42,
)
rf.fit(X_train, y_train)
rf.score(X_test, y_test)

Each tree is overfit in its own random way; averaging cancels the noise while keeping the signal. This is bagging, and it’s why forests are so hard to beat as a first serious model — strong accuracy with almost no tuning.

Free Bonus: Feature Importance

import pandas as pd

pd.Series(rf.feature_importances_, index=X.columns).sort_values(ascending=False)
# area         0.52
# location     0.31
# bedrooms     0.12 ...

Importances tell you what the model relies on — invaluable for debugging (a feature at 0.95 usually means leakage) and for explaining results to stakeholders.

When to Reach for a Forest

Tabular data (spreadsheet-shaped), mixed feature types, < a few million rows: random forest is an excellent default. It rarely wins competitions — that honour goes to gradient boosting, up next — but it gets you 95% of the way with 5% of the effort.