Memory tricks that make computer vision and image AI click

Segmentation goes beyond bounding boxes — it classifies every single pixel. Semantic segmentation: color all pixels of the same class the same color (all cars = red, all roads = gray, all sky = blue). Instance segmentation: distinguish individual objects of the same class (car #1 = orange, car #2 = purple). Panoptic segmentation: unifies both into one complete picture — every pixel gets a category label AND individual objects are told apart, all in a single pass. Used in self-driving cars (understand the entire scene pixel by pixel), medical imaging (outline a tumor precisely), and satellite imagery analysis.

Early layers detect low-level features (edges, corners, color gradients). Middle layers detect textures, shapes, and patterns. Deep layers detect semantic concepts (faces, cars, dogs). This hierarchy mirrors the human visual cortex. AlexNet (2012): first deep CNN to win ImageNet -- crushed all competitors and launched the deep learning revolution in computer vision.

Classification: what is in this image. Detection: WHAT and WHERE -- bounding box plus class label for each object. Segmentation: which exact pixels belong to each object. YOLO: one-stage detector, divides image into grid, predicts boxes and classes simultaneously -- real-time speed. Faster R-CNN: two-stage, slower but more accurate. IoU: area of overlap divided by area of union -- threshold at 0.5 for acceptable detection.

U-Net: contracting path (encoder) compresses spatial dimensions while increasing channels -- captures what is in the image. Expanding path (decoder) restores spatial resolution -- recovers where. Skip connections: copy encoder feature maps directly to corresponding decoder layer -- preserve fine spatial detail lost during downsampling. Dominant for medical image segmentation. SAM (Meta 2023): foundation model -- segment any object with a click.

Geometric: horizontal/vertical flip, random crop, rotation, perspective warp. Color: brightness, contrast, saturation, hue jitter. Advanced: CutOut (mask random patches), MixUp (blend two images and labels), CutMix (paste patches from one image into another), AutoAugment (learned augmentation policy). Standard ImageNet recipe: RandomResizedCrop(224) + RandomHorizontalFlip + ColorJitter + Normalize.

ViT (Dosovitskiy et al., 2020): split image into 16x16 patches, flatten, project to embedding, add position embedding, run through standard Transformer encoder. Global context from layer 1 -- unlike CNNs that build context gradually. Needs more data than CNNs (no inductive bias for locality). SWIN Transformer: hierarchical ViT with shifted windows, efficient for high-resolution detection and segmentation.

VAE: encode image to distribution (mean and variance), sample, decode -- smooth latent space enables interpolation. GAN: Generator vs Discriminator, adversarial training, sharp results but mode collapse and instability. Diffusion: add noise over T steps, train to denoise, generate by starting from noise -- most stable training, best diversity, dominant for image generation (Stable Diffusion, DALL-E 3, Midjourney).

Pipeline: detect faces (MTCNN, RetinaFace), align using landmark points, encode face into compact vector (128 or 512 dimensions with ArcFace), compare via cosine similarity to known embeddings. Triplet loss: pull same-person embeddings together, push different people apart. Bias: higher error rates for dark-skinned women. EU AI Act bans real-time biometric surveillance in public spaces.

SAE Levels: L0 (none) to L5 (full autonomy any condition). L2: partial, human monitors (Tesla Autopilot). L4: high automation, limited geofenced area (Waymo One). Camera-only (Tesla) vs sensor fusion (Waymo). LiDAR: precise 3D depth, expensive. Camera: cheap, texture-rich, needs depth estimation. Radar: all-weather, velocity measurement. Long tail problem: rare edge cases too rare to appear in training data -- kills autonomous driving reliability.

Proven: diabetic retinopathy (matches ophthalmologist), CheXNet chest X-ray pneumonia detection, lymph node metastasis detection. Challenges: distribution shift (fails at hospitals not in training data), class imbalance (rare diseases), FDA approval required (510(k)). Grad-CAM: highlight influential image regions -- required for clinician trust. 3D U-Net: standard for CT and MRI volumetric segmentation.

Classification: Top-1 accuracy, Top-5 accuracy. Object Detection: mAP -- compute AP (area under precision-recall curve) for each class, average across classes. COCO mAP: average over IoU thresholds 0.5 to 0.95 -- much harder than Pascal VOC (IoU=0.5 only). Segmentation: mIoU (mean Intersection over Union across all semantic classes). FPS: frames per second -- real-time requires 30+ FPS.

Optical flow: 2D vector (dx, dy) per pixel between consecutive frames showing apparent motion. Classical: Lucas-Kanade (local patch matching), Horn-Schunck (global smoothness). Deep learning: FlowNet, PWC-Net, RAFT (current state of art). Applications: video compression (encode motion), action recognition, video stabilization, slow-motion generation (interpolate frames), object tracking.

Image classification answers "What is in this image?" — one label for the whole image (cat). Object detection answers "What is where?" — draw bounding boxes around each object AND label each one (cat at top-left, dog at bottom-right, person in center). Detection is classification + localization. YOLO (You Only Look Once) is the famous real-time object detector — it processes the entire image in one pass, making it fast enough for video.

Computer vision models need massive training data. Data augmentation artificially expands the dataset by applying transformations: horizontal flip, rotation, random crop, brightness/contrast change, adding noise. A single cat photo becomes 20 training examples. The model learns that a cat flipped upside down is still a cat — improving robustness. This is why modern CV models generalize well even when real data is limited. Augmentation happens on-the-fly during training.

After each convolution layer, the network produces feature maps — one per filter. Each feature map highlights where a particular pattern (edge, curve, texture) appears in the image. Early layers: simple features (horizontal edges, vertical edges). Middle layers: shapes (curves, corners). Deep layers: complex parts (eyes, wheels, faces). Visualizing feature maps shows what the network has learned to see — like giving it X-ray goggles tuned to different patterns.

Training a CNN from scratch needs millions of images. Instead: take ResNet, VGG, or EfficientNet pre-trained on ImageNet (1.2 million images, 1000 classes) — the early layers already detect edges, textures, and shapes. Freeze those layers. Replace the final classification layer with your own. Train only the last few layers on your small dataset. The borrowed eyes already know what visual features look like — you just teach them to recognize your specific objects.

Every CNN builds a visual hierarchy: Low-level features (edges, corners, colors, textures — detected by early layers, reusable across all tasks). Mid-level features (shapes, patterns, object parts — eyes, wheels, doors — detected by middle layers). High-level features (whole objects — faces, cars, buildings — detected by deep layers, task-specific). This hierarchy is why transfer learning works: the low and mid-level features trained on ImageNet transfer to almost any vision task.

IoU (Intersection over Union) measures bounding box quality: area where predicted and ground truth boxes overlap, divided by total area covered by both. IoU=1.0 = perfect match. IoU=0.5 = standard threshold for "correct detection." A detection is counted as correct only if IoU ≥ threshold AND class label matches. mAP (mean Average Precision) averages the AP score across all classes and IoU thresholds — the standard benchmark for object detectors (COCO, Pascal VOC).

Discriminative CV: given an image, output a label or bounding box. Classification (what?), detection (what and where?), segmentation (what at every pixel?). Generative CV: given a prompt or noise, create a new image. GANs, diffusion models, VAEs. The line is blurring: modern vision-language models (CLIP, DALL-E, Stable Diffusion) combine both — understanding AND generating images from the same representation space.

Object detectors predict hundreds of overlapping bounding boxes around each object. Without cleanup, you get 50 boxes around the same car. Non-Maximum Suppression fixes this: sort all boxes by confidence score, keep the highest-confidence box, delete any remaining boxes with IoU > threshold against the kept box, repeat. Result: one clean box per object. NMS is the final step in YOLO, Faster R-CNN, and every real-time detector.