Topic 05 / 12

Model Evaluation: Beyond Accuracy

~10 min read  //  AI & ML Series  //  Coding India

The Accuracy Trap

A fraud dataset where 99% of transactions are clean lets a model that always predicts “clean” score 99% accuracy while catching zero fraud. Accuracy collapses the four possible outcomes into one number and hides the failure that matters.

The Confusion Matrix

from sklearn.metrics import confusion_matrix
confusion_matrix(y_test, preds)
#                 predicted: clean   fraud
# actual: clean        [9890,         10]     ← false positives
# actual: fraud        [  60,         40]     ← false negatives
  • Precision = of everything flagged as fraud, how much really was? 40 / 50 = 0.80
  • Recall = of all real fraud, how much did we catch? 40 / 100 = 0.40
  • F1 = harmonic mean of the two — a single number when you must rank models.
from sklearn.metrics import classification_report
print(classification_report(y_test, preds))

Precision and Recall Trade Off

Raise the decision threshold → fewer flags → precision up, recall down. Lower it → the reverse. Which side to favour is a product question:

  • Spam filter — favour precision; losing a real email is worse than seeing spam.
  • Cancer screening — favour recall; a missed case is catastrophic, a false alarm is a follow-up test.

ROC-AUC: Threshold-Free Comparison

from sklearn.metrics import roc_auc_score
roc_auc_score(y_test, clf.predict_proba(X_test)[:, 1])

AUC is the probability the model ranks a random positive above a random negative. 0.5 = coin flip, 1.0 = perfect ranking. Use it to compare models before you’ve committed to a threshold.

Cross-Validation: Trust Your Numbers

One train/test split is one noisy sample of performance. K-fold cross-validation trains K times, each fold taking a turn as the test set:

from sklearn.model_selection import cross_val_score
scores = cross_val_score(clf, X, y, cv=5, scoring="f1")
print(scores.mean(), "+/-", scores.std())

Report the mean and the spread. A model scoring 0.82 ± 0.01 is more trustworthy than one scoring 0.84 ± 0.09. For regression, swap in scoring="neg_mean_absolute_error" — the workflow is identical.