Overfitting, Regularization & Bias-Variance

Memorising vs Learning

Give a model enough capacity and it will memorise the training data — noise, quirks, and all — then fail on anything new. That’s overfitting. The opposite, underfitting, is a model too simple to capture the real pattern (fitting a line to a curve).

The diagnostic is always the same pair of numbers:

model.score(X_train, y_train)   # 0.99  ← suspiciously perfect
model.score(X_test,  y_test)    # 0.71  ← the truth

A large train–test gap = overfitting. Both scores low = underfitting.

Bias and Variance

• High bias (underfit): wrong on average, consistently. The model’s assumptions are too rigid.
• High variance (overfit): wildly different results if you retrain on slightly different data. The model is unstable.

More model complexity trades bias for variance. The art of ML is finding the sweet spot — and more training data shifts the whole curve in your favour, which is why data beats cleverness so often.

Regularization: A Penalty for Complexity

Add the size of the weights to the loss, and the optimiser must justify every parameter it uses:

loss = MSE + alpha * sum(w**2)    # L2 (Ridge)
loss = MSE + alpha * sum(|w|)     # L1 (Lasso)

from sklearn.linear_model import Ridge, Lasso

ridge = Ridge(alpha=1.0).fit(X_train, y_train)   # shrinks weights smoothly
lasso = Lasso(alpha=0.1).fit(X_train, y_train)   # drives some weights to 0

L1’s zeroing behaviour doubles as automatic feature selection — inspect lasso.coef_ and the dead features are right there. In LogisticRegression, regularization is on by default; C is the inverse strength (smaller C = stronger penalty).

Tuning alpha Properly

Never tune hyperparameters against the test set. Use cross-validation on the training data:

from sklearn.model_selection import GridSearchCV

grid = GridSearchCV(Ridge(), {"alpha": [0.01, 0.1, 1, 10, 100]}, cv=5)
grid.fit(X_train, y_train)
grid.best_params_     # e.g. {'alpha': 10}
grid.score(X_test, y_test)   # touch the test set exactly once

Other Weapons Against Overfitting

• More data — the most reliable fix, when you can get it.
• Fewer/better features — remove leaky or noisy columns.
• Early stopping — halt training when validation loss stops improving (standard in boosting and neural nets).
• Ensembles — average many overfit models and the noise cancels; that’s the next topic.