Memory tricks that make tokenization, BERT and LLMs click

Tokenization: split into tokens. Normalization: lowercase, punctuation, contractions. Stop word removal: remove high-frequency low-information words. Stemming/Lemmatization: reduce to root form. Representation: convert to numerical vectors (BoW, TF-IDF, embeddings). Model: ML or DL algorithm. Evaluate: task-specific metric.

Bag of Words (BoW): word counts, ignores order and context, sparse. TF-IDF: weights rare domain terms higher, still sparse. Word2Vec/GloVe: dense semantic vectors, king - man + woman = queen, fixed per word. BERT / contextual embeddings: same word gets different embedding based on surrounding context. All modern NLP uses contextual embeddings.

BERT (Encoder-only, Bidirectional): sees entire sequence simultaneously, trained on masked language modeling, excellent for classification, NER, question answering, cannot generate text. GPT (Decoder-only, Autoregressive): sees only left context, trained on next-token prediction, excellent for text generation, creative writing, code, conversation. T5/BART: Encoder-Decoder -- best for translation and summarization.

BPE Algorithm: start with character vocabulary, count all character pairs, merge most frequent into new token, repeat until target vocabulary size. Result: common words as single tokens, rare words split into subword units. No true OOV words -- any word can be handled. GPT models: 50K-100K token vocabulary. Token count affects cost, speed, and context window utilization.

Zero-shot: just ask -- no examples. Few-shot: provide 2-5 examples of desired format. Chain-of-Thought (CoT): add Let's think step by step -- improves math, logic, multi-step reasoning. Role prompting: Act as an expert... -- calibrates vocabulary, depth, style. Structured output: Respond in JSON format. RAG: retrieve context to ground LLM in real facts and reduce hallucination.

BLEU: n-gram overlap between generated and reference translations. Range 0-1. Correlates with human judgment but misses semantic equivalents. ROUGE: n-gram overlap for summarization. ROUGE-1 (unigrams), ROUGE-2 (bigrams), ROUGE-L (longest common subsequence). Perplexity: how well LM predicts test set -- lower is better. BERTScore: uses embeddings for semantic similarity -- better than n-gram metrics.

History: Rule-based (1950s-80s), Statistical phrase-based (1990s-2010s), Neural (2014 -- seq2seq + attention), Transformer (2017 -- surpassed all previous). Modern LLMs competitive for most language pairs. Parallel corpora: same content in two languages needed for training. Back-translation: translate target to source synthetically, use as additional training data.

Sparse retrieval (BM25, TF-IDF): keyword matching -- fast, interpretable, handles exact terms (names, codes). Dense retrieval: semantic embedding similarity -- finds relevant documents even if keywords differ. Two-stage: BM25 narrows candidates (fast), dense re-ranks (accurate). RAG pipeline: embed documents, store in vector DB, retrieve top-K, inject into LLM prompt.

ASR pipeline: audio signal to features (mel spectrogram) to model to text. Traditional: acoustic model + pronunciation dictionary + language model. Modern end-to-end: CTC or attention encoder-decoder. Whisper (OpenAI 2022): trained on 680K hours of multilingual audio, near-human on English, open-source and free to run locally. WER (Word Error Rate): standard evaluation metric.

Coreference resolution identifies expressions referring to the same entity. Types: pronouns (he, she, it), noun phrases (the president), proper names (John). Challenge: requires world knowledge and reasoning. Example: The city council refused the protesters a permit because they feared violence -- they = city council, not protesters. Applications: reading comprehension, information extraction, summarization.

Greedy: always pick highest probability token -- deterministic, fast, repetitive. Sampling: randomly sample from distribution -- diverse but can be incoherent. Temperature: below 1 = sharper (conservative), above 1 = flatter (creative). Top-K: sample from top K tokens only. Top-P (nucleus): sample from smallest token set with cumulative probability above P -- adapts to distribution shape, most widely used. Repetition penalty: downweight already-generated tokens.

Before any NLP model reads text, it must be tokenized — split into tokens. Tokens can be words, subwords, or characters depending on the tokenizer. "unhappiness" might become ["un", "happy", "ness"] — three tokens. Why subwords? They handle rare words and new words gracefully. GPT uses byte-pair encoding (BPE). A typical paragraph becomes 100–150 tokens. Everything in an LLM is measured in tokens — context windows, costs, limits.

Word embeddings map each word to a point in high-dimensional space (e.g., 300 numbers). Similar words cluster together — "king" and "queen" are close. The magic: vector arithmetic captures meaning. King − Man + Woman ≈ Queen. Paris − France + Italy ≈ Rome. The model doesn't understand meaning — it learns that words appearing in similar contexts get similar coordinates. Word2Vec, GloVe, and modern contextual embeddings (BERT) all work on this principle.

Both BERT and GPT are Transformer-based but trained differently. BERT: trained by masking random words and predicting them using context from both left AND right — bidirectional. Best for understanding tasks (classification, named entity recognition, question answering). GPT: trained to predict the next word reading left to right — autoregressive. Best for generation tasks (writing, completing text, chatting). BERT is the reader; GPT is the writer.

NER identifies and classifies named entities in text: People (Barack Obama), Organizations (Google, WHO), Locations (Paris, Antarctica), Dates (March 12, 2024), and more. It's one of NLP's most practical tasks — used in news analysis, medical records, financial documents, and search engines. A sentence like "Apple announced in Cupertino that Tim Cook signed a deal with Samsung" yields: Apple (ORG), Cupertino (LOC), Tim Cook (PERSON), Samsung (ORG).

RAG gives LLMs access to a knowledge base at inference time. Instead of relying purely on memorized training data: Retrieve relevant documents from a database, Augment the prompt with that retrieved context, Generate an answer grounded in the retrieved facts. Result: far fewer hallucinations, up-to-date answers, and citations. Think of it as the difference between a closed-book exam (pure LLM) and an open-book exam (RAG). Most enterprise AI systems use RAG.

Every modern NLP system follows this pipeline: Tokenize (split raw text into tokens — subwords), Embed (map each token to a dense vector), Encode (run through transformer layers to build contextual representations — BERT-style), Decode (generate output tokens one at a time — GPT-style), Generate (apply sampling strategy — greedy, beam search, temperature). Understanding each stage lets you diagnose failures: garbled output could be tokenizer, poor coherence could be decoding strategy.

BLEU (Bilingual Evaluation Understudy): measures how many n-grams in the machine output appear in the reference translation — precision-oriented. Standard for machine translation. Range 0–1 (higher = better). ROUGE (Recall-Oriented Understudy for Gisting Evaluation): measures how many n-grams in the reference appear in the generated summary — recall-oriented. Standard for text summarization. ROUGE-1 (unigrams), ROUGE-2 (bigrams), ROUGE-L (longest common subsequence). Both compare to human reference outputs.

Keyword search (TF-IDF, BM25): finds documents containing the exact query words. Fails when synonyms are used or the concept is expressed differently. Semantic search: converts query and documents to dense vector embeddings, retrieves by cosine similarity in vector space. "Heart attack" retrieves "myocardial infarction" documents because their embeddings are close. Used in RAG systems for the retrieval step. FAISS and vector databases (Pinecone, Weaviate, Chroma) store and search billions of embeddings efficiently.

Prompting: write better instructions, add examples (few-shot), structure your query. Fast, cheap, no training needed. Sufficient for most tasks. Fine-tuning: update the model weights on your specific task data. Needed when: the task requires a consistent style/format the model struggles to follow via prompting, you have thousands of task-specific examples, or you need better performance than prompting achieves. Rule of thumb: always try prompting first. Fine-tune only if prompting hits a ceiling.