Memory tricks that make machine learning algorithms click

Supervised learning is memorization: every training example has a known correct answer (a label) attached, and the model memorizes the relationship between question and answer — like a flashcard with the answer on the back. Unsupervised learning is a search for patterns and deviations: no known answers exist, so the model finds natural groupings on its own and flags anything that deviates from them. This matters beyond just cost — supervised learning can only ever replicate patterns humans already identified, while unsupervised learning can uncover genuinely new patterns nobody has found yet, giving it real potential to solve mysteries and support proving theoretical ideas once a pattern shows up in real data. Supervised's edge is precise, measurable accuracy since the answers are known; the tradeoff is the cost of labeling every example, and it can only ever recognize categories it was explicitly trained on.

Imagine you're blindfolded on a hilly landscape. Your goal is to reach the lowest valley (minimum loss). You feel the slope under your feet and take a small step downhill. Repeat thousands of times. That's gradient descent — the model checks which direction reduces error, takes a small step that direction, and repeats until it can't go lower. Learning rate = how big each step is. Too big = overshoot. Too small = takes forever. It's also the same underlying process nearly every ML model uses to learn, from simple linear regression up to massive neural networks.

Select the right algorithm for your problem type. Prepare data: clean, normalize, handle missing values, encode categoricals, split train/test. Learn: train the model -- algorithm finds optimal parameters. Interpret: examine what was learned. Test: evaluate on held-out test set -- never seen during training. Performance on test data is the only honest measure of model quality.

Linear Regression: y = mx + b, minimizes squared errors, predicts continuous values (house price, temperature). Logistic Regression: despite the name, used for CLASSIFICATION -- uses sigmoid function to output probability 0-1. Decision boundary at 0.5. Above = positive class. Below = negative class. This distinction appears on virtually every ML exam.

Decision Trees split data by feature questions using Gini impurity or Information Gain. Prone to overfitting alone. Random Forest (bagging): parallel trees on random data and feature subsets -- reduces variance. XGBoost (boosting): sequential trees, each corrects previous errors -- dominant for tabular data competitions. No feature scaling needed for tree-based methods.

Find K closest training examples (Euclidean distance), take majority vote (classification) or average (regression). Always normalize -- large-range features dominate distance. Curse of dimensionality: too many features makes distance meaningless. Choose K with cross-validation. K=1 overfits. Large K underfits.

SVM finds the hyperplane that maximally separates classes. Wider margin = better generalization. Kernel trick: maps data to higher dimensions where linearly separable -- RBF kernel handles nonlinear data. C parameter: low C = wider margin, more misclassifications tolerated. High C = narrow margin, fewer misclassifications. Best for high-dimensional sparse data.

Steps: choose K, randomly initialize K centroids, assign each point to nearest centroid, recalculate centroids as cluster means, repeat until stable. Elbow method: plot inertia vs K, choose where improvement slows. DBSCAN alternative: finds arbitrary-shaped clusters and outliers automatically without specifying K.

Normalization (Min-Max): scales to 0-1. Standardization (Z-score): mean=0, std=1. Scale for: K-NN, SVM, neural networks, PCA. NOT for: tree-based methods. One-hot encoding: categoricals to binary columns. Label encoding: integers to categories -- only when ordinal. PCA: reduce dimensions by keeping directions of maximum variance. Golden rule: fit transformers on training data only.

P(class|features) proportional to P(class) times product of P(feature|class). Despite independence assumption being almost always violated, it works well for text -- word frequencies are roughly independent given class. Gaussian NB: continuous features. Multinomial NB: word counts. Laplace smoothing: add 1 to all counts to prevent zero probabilities.

Agent: learner. State: current situation. Action: choice made. Reward: feedback signal. Policy: strategy mapping states to actions. Goal: maximize expected cumulative reward. Exploration vs exploitation. Q-learning: learns value of state-action pairs. RLHF (Reinforcement Learning from Human Feedback): human raters rank outputs, reward model trained, RL maximizes it -- standard for aligning LLMs.

Trend: long-term direction. Seasonality: regular repeating patterns (daily, weekly, yearly). Autocorrelation: current value correlates with its own past values. Cycle: irregular multi-year patterns. NEVER shuffle before splitting -- always chronological. Classic models: ARIMA. Modern: LSTMs, Temporal CNNs, Transformer-based forecasters.

Isolation Forest: anomalies isolated by fewer random splits (short path = anomaly) -- fast, scalable, no distance metric needed. Statistical: z-score, flag beyond N standard deviations. Autoencoder: train on normal data only, flag high reconstruction error points as anomalies. Applications: fraud detection, network intrusion, manufacturing defects, medical outliers.

AutoML tools: Google AutoML (cloud, no-code), H2O AutoML (open-source), Auto-sklearn. NAS (Neural Architecture Search): discovered EfficientNet -- outperformed human-designed networks. Active Learning: model queries the most uncertain examples for labeling -- reduces labeling cost by 5-10x. Limitations: still requires clean data, correct evaluation metric, good problem definition.

Reinforcement learning has three parts: an Agent (the dog/AI), an Environment (the world it acts in), and a Reward signal (treat or punishment). The agent takes actions, receives rewards or penalties, and learns which actions maximize total reward over time. No labels needed — just trial, error, and feedback. This is how AlphaGo beat world chess champions and how robots learn to walk.

K-fold cross-validation splits data into K equal chunks. In each round, one chunk is the test set and the rest train the model. Rotate K times so every chunk gets tested once. Average the K scores for a reliable estimate. Why? One test split might be lucky or unlucky. K-fold gives a fair average across many splits. K=5 or K=10 are standard. Think of it as rotating who grades your exam to get a fair average score.

A single decision tree can overfit badly. A Random Forest grows hundreds of trees on random data subsets and averages their predictions. More diverse opinions = more stable result. This is ensemble learning — combining many weak learners into one strong learner. Bagging (Random Forest) averages parallel models. Boosting (XGBoost) trains models sequentially, each fixing the last one's errors. Either way: the forest always beats the tree.

Picture a dartboard. Bias = how far from the bullseye your average shot lands (systematic error — wrong aim). Variance = how spread out your shots are (inconsistency). High Bias = underfitting — model too simple, misses the pattern every time. High Variance = overfitting — model too complex, nails training data but wild on new data. The goal: low bias AND low variance. Increasing model complexity reduces bias but raises variance — the unavoidable tradeoff.

Overfitting: training accuracy very high, validation accuracy much lower — the model memorized the training set including its noise. Like a student who memorized every practice exam verbatim but can't handle new questions. Underfitting: both training AND validation accuracy are low — the model hasn't learned the pattern at all. The diagnostic: plot learning curves. A big gap between training and validation = overfitting. Both curves low and flat = underfitting.

Bagging (Bootstrap Aggregating): trains many models IN PARALLEL on random data subsets, averages their predictions. Reduces variance. Random Forest is the gold standard. Boosting: trains models SEQUENTIALLY — each new model focuses on examples the previous one got wrong. Reduces bias. XGBoost, LightGBM, and AdaBoost are the most powerful implementations. Rule of thumb: Random Forest when you need speed and reliability. XGBoost when you need maximum accuracy on tabular data.

Regularization adds a penalty to the loss function for large weights — taxing the model for complexity. L1 (Lasso): penalty = sum of absolute values of weights. Drives unimportant weights to exactly zero — built-in feature selection. L2 (Ridge): penalty = sum of squared weights. Shrinks all weights toward zero but rarely to exactly zero. Elastic Net combines both. Use L1 when you suspect only a few features matter. Use L2 when all features contribute something. Lambda controls the penalty strength.