Memory tricks that make neurons, backprop and deep architectures click

Inputs times Weights, summed plus bias: z = w1x1 + w2x2 + b. Activation function applies nonlinearity. Without activations, any depth network can only learn linear functions. Nonlinear activations allow networks to approximate any function (Universal Approximation Theorem). The activation function is what makes deep learning powerful.

Sigmoid (Silly): squashes to 0-1, use ONLY at binary output layer, vanishing gradient kills it in hidden layers. Tanh (Tom): output -1 to 1, centered at zero, still vanishes in deep nets. ReLU (Really): max(0,x), THE default for hidden layers, fast, avoids vanishing gradient. Leaky ReLU (Likes): fixes dying ReLU with small negative slope. Softmax: always at multi-class output -- converts logits to probabilities summing to 1.

Input layer: one neuron per feature, no computation. Hidden layers: where representations are learned. Output layer: one per class (classification) or one neuron (regression). Depth allows hierarchical representation learning. Universal Approximation Theorem: even one hidden layer with enough neurons can approximate any continuous function.

Forward pass: data flows forward producing a prediction. Loss: measures how wrong the prediction was -- the regret. Backward pass: regret flows BACKWARD through the network via chain rule of calculus, assigning blame (gradient) to each weight. Weight update: w = w - alpha times gradient. Adam optimizer: adaptive per-parameter learning rate -- the standard default (lr=0.001).

Dropout: 20-50% neurons disabled per step, forces redundant representations, scale by (1-p) at inference. Batch Normalization: normalizes layer inputs to mean=0, std=1, speeds training dramatically, allows higher learning rates. L2 weight decay: penalizes large weights. Early stopping: monitor validation loss, stop when it rises.

Learning rate (most critical), batch size (32-128 typical), number of layers, neurons per layer, activation functions, dropout rate. Tuning methods: random search (often better than grid for same budget), Bayesian optimization. Transfer learning: start from pretrained model -- dramatically helps with small datasets.

MSE (Mean Squared Error): regression, sensitive to outliers. MAE: regression, robust to outliers. Binary Cross-Entropy: binary classification with sigmoid output. Categorical Cross-Entropy: multi-class classification with softmax output. Loss is optimized during training (must be differentiable). Metric is what you report (can be anything -- accuracy, F1, AUC).

Zero initialization: catastrophic -- all neurons identical, all gradients identical, network never diversifies (symmetry problem). Xavier: scales by 1/sqrt(n_in) -- designed for sigmoid and tanh. He initialization: scales by 2/sqrt(n_in) -- designed for ReLU (accounts for half of neurons being zeroed). Batch Normalization reduces sensitivity to initialization.

Vanishing gradients: gradients shrink exponentially backward through many layers -- early layers barely update. Caused by sigmoid/tanh and many layers. Fix: ReLU, batch normalization, residual connections (ResNet skip connections -- gradient highway bypasses layers). Exploding gradients: weights blow up. Fix: gradient clipping (cap magnitude), careful initialization.

For each output position: Query (what am I looking for?), Key (what do I contain?), Value (what do I provide?). Score = Q dot K divided by sqrt(d_k), then softmax for weights, then weighted sum of Values. Self-attention: tokens attend to each other. Causal (decoder): only attends LEFT -- future tokens masked to enable autoregressive generation.

Batch Normalization: normalize across batch dimension for each feature -- great for CNNs, needs large batch, fails with small batches or variable-length sequences. Layer Normalization: normalize across feature dimension within each sample -- batch-size independent, works with variable lengths, standard for Transformers and RNNs. Both add learnable scale and shift parameters.

SGD: simple, often oscillates. Momentum: adds inertia, accelerates in consistent directions. RMSProp: adaptive per-parameter learning rate. Adam: combines momentum plus RMSProp -- most popular, works with defaults (lr=0.001, beta1=0.9, beta2=0.999). AdamW: proper weight decay fix -- better generalization, standard for fine-tuning transformers.

Each artificial neuron receives inputs (signals from other neurons), multiplies each by a weight (importance), sums them up, and fires if the total exceeds a threshold — controlled by an activation function. Like a real brain cell: weak signals from many friends can add up and trigger a response. The weights are what the network learns — adjusting them is how training happens.

ReLU (Rectified Linear Unit) is the most widely used activation function: output = max(0, x). If input is negative, output is zero (neuron silent). If positive, output equals input (neuron passes it through). Simple, fast, and avoids the vanishing gradient problem that plagued older functions like sigmoid. Think of ReLU as a bouncer — nothing negative gets past the door.

CNN (Convolutional Neural Network): designed for spatial data — images, detecting features regardless of position. RNN (Recurrent Neural Network): designed for sequential data — processes time-ordered inputs like text or audio, with memory of past steps. Transformer: attends to all positions simultaneously using attention, no sequential processing — this is what powers GPT, BERT, and all modern LLMs. Each architecture has a superpower matched to a data type.

During training, dropout randomly turns off a percentage of neurons (typically 20–50%) on each pass. This forces the network to learn redundant representations — it can't rely on any single neuron. Result: a more robust, generalizable network that doesn't overfit. At test time, all neurons are active. Think of it as randomly benching players in practice so the whole team builds skills, not just the stars.

In backpropagation, error signals flow backward through layers. Each layer multiplies the gradient by a small number (like 0.1). After 10 layers: 0.1^10 = 0.000000001. By the time the signal reaches early layers it's essentially zero — those layers stop learning. This is the vanishing gradient problem that crippled deep networks before ReLU and residual connections solved it. Modern architectures are specifically designed to keep gradients alive.

Every training step has three phases: Forward pass — feed input through all layers, compute the prediction, calculate the loss (how wrong were we?). Backward pass (backpropagation) — compute how much each weight contributed to the error, flowing gradients back through layers. Weight update — adjust each weight slightly in the direction that reduces error, scaled by the learning rate. Repeat millions of times.

Sigmoid: output 0–1, used for binary classification output layer, suffers vanishing gradient in deep nets. Tanh: output -1 to 1, zero-centered (better than sigmoid for hidden layers), still suffers vanishing gradient. ReLU: max(0,x), fastest to compute, avoids vanishing gradient in most cases, standard choice for hidden layers in deep networks. Softmax: converts logits to probabilities summing to 1 — used in multi-class output layers.

Initializing all weights to zero: every neuron computes the same gradient — symmetry never breaks, nothing learns. Random weights too large: activations explode. Too small: vanishing gradient. Xavier initialization (Glorot): scales weights by input/output size — designed for sigmoid/tanh. He initialization: scales larger — designed for ReLU. PyTorch applies He init by default for linear and conv layers. Match the initializer to the activation function.

Epochs: how many complete passes through the entire training set. More epochs = more learning, but risk overfitting. Batch size: how many samples to process before updating weights. Small batch (8–32): noisier updates, better generalization. Large batch (256–1024): faster, smoother, may overfit. Learning rate: how big each weight update step is. The single most important hyperparameter. Too high: diverges. Too low: never converges. Use a learning rate finder or start at 1e-3 with Adam.