Model Evaluation and Cross-Validation

The Danger of Overfitting

A model that perfectly predicts its training data is usually useless. It has memorized the answers rather than learning the underlying patterns. This is called overfitting. To detect this, we must always evaluate models on data they have never seen before.

Evaluation Metrics

Accuracy isn’t always the best metric. If only 1% of transactions are fraudulent, a model that simply guesses “Not Fraud” every single time will be 99% accurate—but completely useless.

• Precision: Out of all cases predicted positive, how many were actually positive? (Minimizes False Positives)
• Recall: Out of all actual positive cases, how many did we find? (Minimizes False Negatives)
• F1-Score: The harmonic mean of Precision and Recall.

from sklearn.metrics import classification_report

print(classification_report(y_test, predictions))

K-Fold Cross-Validation

A single train/test split can be lucky or unlucky depending on which rows ended up in the test set. K-Fold Cross-Validation splits the data into k different chunks, trains k models, and averages the results to give you a much more robust estimate of performance.

from sklearn.model_selection import cross_val_score

# Run a 5-fold cross-validation
scores = cross_val_score(model, X, y, cv=5)
print(f"Average Accuracy: {scores.mean():.2f}")