Memory tricks that make precision, recall, F1 and AUC click

True Positive (TP): predicted positive, actually positive. False Positive (FP): predicted positive, actually negative -- Type I error, false alarm. False Negative (FN): predicted negative, actually positive -- Type II error, missed detection. True Negative (TN): predicted negative, actually negative. Every classification metric -- accuracy, precision, recall, F1, AUC -- derives from these four numbers.

Precision is PICKY: only says yes when really sure -- few false alarms. TP divided by (TP + FP). Recall REMEMBERS everything: sweeps wide to catch every real positive -- few misses. TP divided by (TP + FN). Tradeoff: raising classification threshold raises precision but lowers recall. Spam filter: be PICKY (don't block real email). Cancer screening: REMEMBER everything (don't miss a cancer).

F1 combines precision and recall into one metric. Harmonic mean punishes extreme imbalance -- you cannot hide 10% recall behind 99% precision. F1 = 1.0 is perfect, 0 is worst. Use when class imbalance exists and you care about both FP and FN. F-beta: beta greater than 1 emphasizes recall (medical). Beta less than 1 emphasizes precision.

ROC plots True Positive Rate (recall) vs False Positive Rate at every classification threshold. AUC = 0.5: no better than random. AUC = 1.0: perfect. AUC = 0.8+: good. Threshold-independent -- evaluates ranking ability. Use PR curve instead of ROC when positive class is very rare -- ROC can be optimistic with severe class imbalance.

MAE (Mean Absolute Error): average of absolute differences -- robust to outliers, original units. MSE: average of squared differences -- penalizes large errors, not in original units. RMSE: square root of MSE -- back in original units, most commonly reported. R-squared: proportion of variance explained -- 1.0 perfect, 0 no better than mean, can be negative.

Split data into K equal folds. For each fold: train on K-1 folds, evaluate on remaining fold. Average performance across all K folds. 5-fold and 10-fold standard. Stratified K-fold: ensures each fold has same class distribution -- important for imbalanced data. Time series: always split chronologically -- never random.

A model that always predicts the majority class gets 99% accuracy but 0% recall on minority -- useless. SMOTE: creates synthetic minority examples by interpolating between existing ones. Class weights: tell algorithm to penalize minority class errors more (class_weight=balanced). Use F1 or AUC, not accuracy, for evaluation.

Data leakage: information from outside training set contaminates the model. Sources: fitting scaler on full dataset before splitting (never do this), including features that wouldn't be available at inference time, using future information for historical prediction. Detection: unrealistically high performance, feature importances that don't make causal sense, dramatic performance drop in production.

Calibration measures whether predicted probabilities match actual frequencies. Reliability diagram: predicted probability vs actual frequency -- diagonal line = perfect calibration. Overconfident: predictions cluster near 0 and 1. Underconfident: cluster near 0.5. Temperature scaling: divide logits by T before softmax -- T greater than 1 softens probabilities. Critical for medical AI, financial risk, any application where you act on confidence.

MLOps applies software engineering to ML systems. Model monitoring: track data drift (input distribution changes), concept drift (label relationship changes), model performance degradation. Version control: code (Git), data (DVC), models (MLflow). CI/CD pipelines: automatically retrain and test models when data or code changes. Feature stores: centralized reusable feature computation.

Split live traffic: 50% current model (A), 50% new model (B). Measure business metric (conversion, revenue) not just ML metric (accuracy). Key principles: random assignment (eliminates selection bias), single change (isolate cause), sufficient sample size, pre-specified duration. Common mistake: stopping test when p<0.05 appears -- leads to inflated false positive rates.

Precision measures how trustworthy your positive predictions are. Formula: TP / (TP + FP). If your spam filter flags 100 emails as spam and 90 are actually spam, precision = 90%. High precision = when you say "spam," you're almost always right. Low precision = lots of false alarms. Use precision when false positives are costly — like flagging innocent people as criminals or legitimate emails as spam.

Recall (sensitivity) measures how many actual positives you caught. Formula: TP / (TP + FN). If 100 patients actually have cancer and your model catches 90, recall = 90%. High recall = you miss very few real cases. Low recall = dangerous missed detections. Use recall when false negatives are costly — missing a cancer diagnosis, missing a fraud transaction. The sniper (precision) vs the net (recall) — choose based on which error costs more.

F1 is the harmonic mean of precision and recall: 2×(P×R)/(P+R). If precision=1.0 and recall=0.0, F1=0 — no credit for being perfect at one while failing the other. F1 punishes extreme imbalance between the two. Use F1 when both false positives and false negatives matter and classes are imbalanced. A model that predicts "no cancer" for everyone gets 0% recall — F1 catches this while raw accuracy might look fine.

The ROC curve plots True Positive Rate (recall) on the Y-axis vs False Positive Rate on the X-axis at every possible threshold. A perfect classifier hugs the top-left corner (high TPR, low FPR). A random classifier is a diagonal line. AUC (Area Under the Curve) summarizes the whole curve in one number: AUC=1.0 is perfect, AUC=0.5 is random guessing. AUC lets you compare classifiers regardless of threshold — it answers "how good is this model overall?"

The confusion matrix shows all four prediction outcomes: True Positive (predicted yes, was yes ✓), False Positive (predicted yes, was no ✗ — Type I error), False Negative (predicted no, was yes ✗ — Type II error), True Negative (predicted no, was no ✓). All evaluation metrics — precision, recall, F1, accuracy — are calculated from these four numbers. Memorize the layout: TP is top-left when positive is the "interesting" class.

If 99% of emails are not spam, a model that predicts "not spam" for everything achieves 99% accuracy — and catches zero spam. Accuracy hides this completely. Use: Precision (when false positives are costly — spam filter flagging good emails), Recall (when false negatives are costly — missing cancer diagnoses), F1 (when both matter and classes are imbalanced). On balanced datasets accuracy is fine. On real-world imbalanced data, always report all three.

MSE (Mean Squared Error): squares each error, so large errors are penalized much more than small ones. Use when large errors are especially bad (predicting structural safety). MAE (Mean Absolute Error): average of absolute errors — all errors weighted equally, robust to outliers. RMSE: square root of MSE — back in original units like MAE but still penalizes large errors. R² (coefficient of determination): 1.0 = perfect, 0.0 = model no better than predicting the mean, can be negative.

Regular K-fold splits data randomly. With imbalanced classes (e.g., 5% positive), some folds might have 0% or 15% positives by chance — giving wildly variable and unreliable scores. Stratified K-fold ensures each fold has the same class proportions as the full dataset. Always use stratified K-fold for classification, especially with class imbalance. For regression, regular K-fold is fine. Rule: if you are classifying, use StratifiedKFold.

A model might say "70% probability of rain" — but does it actually rain 70% of the time in those cases? A well-calibrated model's predicted probabilities match real-world frequencies. Poorly calibrated: model says 90% confident but is only right 60% of the time — dangerous for medical diagnosis or financial risk. Check with reliability diagrams (calibration curves). Fix with Platt scaling (logistic regression on top) or isotonic regression.