## **Quick Start: How Inference Works** - Your text → Tokenized → Embedded → Transformer Layers → Next token → Repeat - Example: "The cat sat" → [1, 262, 3847, 12934] → vectors → $n_{\text{layers}}$ × Transformer Layers → generates "on" → "the" → "mat"... - **In plain English**: Your prompt is split into pieces (tokens), converted to numbers, and processed through a massive neural network that predicts what comes next—one token at a time. ## **Part 0: Foundation - What Is LLM Inference?** ### **The Big Picture: Pattern Machines** | **Concept** | **Simple Explanation** | **Professional Terms** | | --- | --- | --- | | **What is LLM Inference?** | Using a trained model to predict the next token | Autoregressive generation using pre-trained transformer weights | | **How does it work?** | Process input through trained parameters to generate probability distribution | Forward pass, softmax sampling, token-by-token generation | | **Key limitation?** | Predicts patterns without understanding | Lack of semantic grounding or external verification → hallucinations | **Visual Metaphor**: *"Inference is like a master chef creating a dish from memory - they combine patterns they've learned, but can't taste what they're making. They need quality ingredients (prompt) and careful technique (sampling) for best results."* ### **Key Variables for Inference** Before diving into inference specifics, understand these critical variables: | Variable | Symbol | What It Controls | Typical Range | | --- | --- | --- | --- | | **Layers** | $n_{\text{layers}}$ | Depth of understanding | 12–96 | | **Model Dimension** | $d_{\text{model}}$ | Richness of context per token | 768–12,288 | | **Attention Heads** | $n_{\text{heads}}$ | Parallel attention mechanisms | 12–96 | | **Context Window** | $L_{\text{max}}$ | Max tokens model can process | 2k–128k | **Model-Specific Examples**: - **Llama 2 7B**: $n_{\text{layers}}$=32, $d_{\text{model}}$=4096, $n_{\text{heads}}$=32, $L_{\text{max}}$=4096 - **GPT-3 175B**: $n_{\text{layers}}$=96, $d_{\text{model}}$=12288, $n_{\text{heads}}$=96, $L_{\text{max}}$=2048 ## **Part 1: Input Processing - From Text to Numbers** ### **Tokenization - Splitting Text into Tokens** **Key Concept**: Tokenization converts raw text into a sequence of tokens using a *pre-trained vocabulary and rules*. These tokens are what the model actually reads. ``` Input: "The cat sat on the mat curiously." ↓ Load pre-trained vocabulary + tokenizer rules ↓ Pre-tokenize (split text) & apply subword segmentation ↓ Add special tokens & map to IDs ↓ Output: [1, 262, 3847, 12934, 284, 262, 8291, 18923, 13, 2] ``` **General Step-by-Step Process**: 1. **Normalization** **(depends on tokenizer)** Typical normalization may include: - Unicode cleanup - whitespace handling - optional lowercasing (some tokenizers do this, some don’t) *Note:* GPT-style tokenizers **do not lowercase**; SentencePiece tokenizers usually **do not lowercase either** unless configured to. 2. **Splitting** Split text based on whitespace and punctuation, producing initial chunks: `"The cat sat..." → ["The", "cat", "sat", "curiously", "."]` 3. **Subword Segmentation (BPE / WordPiece / Unigram)** Each chunk is broken into subwords using learned rules. Example (illustrative only): `"curiously" → ["cur", "ious", "ly"]` This ensures: - every word can be represented - unknown or rare words fall back to subwords 4. **Add Special Tokens** Most models prepend/append tokens like: `, , , , ` 5. **Map to Token IDs** Each subword is turned into an integer ID using the pre-trained vocabulary. **Model Comparison - Tokenization Differences**: | **Model** | **Vocab Size** | **Tokenization Method** | **Notable Features** | | --- | --- | --- | --- | | **Llama 1 / Llama 2 / Llama 3** | 32k | SentencePiece BPE | UTF-8 robust; no lowercasing | | **GPT-2 / GPT-3 / GPT-4** | ~50k–100k | Byte-level BPE | Operates on raw bytes; stable for any text | | **Claude** | ~100k (unofficial) | BPE variant | Very similar to GPT tokenizers | | **T5 / Flan-T5** | 32k | SentencePiece (Unigram) | Adds `` tokens | | **BERT / RoBERTa** | ~30k | WordPiece | Uses `` and `` | **Important Notes**: - Tokenizer rules are fixed after training - Different models require different tokenizers - Tokenization can be cached for faster repeated inference - Tokenizers may differ in normalization behavior (byte-level, unicode normalization, case sensitivity) ## **Part 2: Embeddings - From IDs to Vectors** ### **Token Embeddings** Convert discrete token IDs to continuous vectors that the model can process. **Math**: `X = Embedding(TokenIDs) ∈ ℝ^(seq_len × d_model)` **Process**: ```python Input IDs: [262, 3847, 12934] ↓ Embedding Matrix Lookup (W_E ∈ ℝ^(vocab_size × d_model)) ↓ Output Vectors: "the": [0.12, -0.28, 0.21, ..., 0.07] (d_model-dimensional vector) "cat": [-0.23, 0.14, 0.38, ..., -0.08] "sat": [0.31, 0.19, -0.12, ..., 0.11] ``` **Model-Specific Embedding Dimensions**: - **T5 Small**: d_model=512 (compact, fast) - **Llama 2 7B**: d_model=4096 (balanced) - **GPT-3 175B**: d_model=12288 (very expressive, memory-intensive) ### **Positional Encodings** Add position information so the model knows word order. **Math**: `X_final = X + PE(pos) ∈ ℝ^(seq_len × d_model)` **Basic Sinusoidal Positional Encoding (Original Transformer)** For **even** dimensions ($2i$): $$ PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right) $$ For **odd** dimensions ($2i+1$): $$ PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right) $$ **Rotary Position Embedding (RoPE) in Llama 2** ```python Example for sequence "The cat sat": Position 0: RoPE rotation 0° Position 1: RoPE rotation θ Position 2: RoPE rotation 2θ ... ``` **Position Encoding Types by Model** | **Model** | **Position Method** | **Key Advantage** | | --- | --- | --- | | **GPT-3** | Learned absolute positions | Simple, effective for short-to-medium context lengths | | **Llama 2** | RoPE (Rotary Position Embedding) | Handles long sequences; captures relative positions naturally | | **T5** | Relative position bias | Works well for encoder-decoder attention and variable-length sequences | | **Claude** | ALiBi (Attention with Linear Bias, estimated) | Excellent for very long contexts; linearly biases attention toward recent tokens | **Result After Embeddings**: `X_final ∈ ℝ^{seq_len × d_model}` ready for transformer layers # **Part 3: The Transformer Block - The Core Building Block** A transformer layer processes a sequence through two main operations: **multi-head attention** and a **feed-forward network**, with **pre-normalization** and **residual connections** for stability. ### Layer Architecture ```python Input: X ∈ ℝ^(seq_len × d_model) ↓ [Branch A: Keep Original X] -----------------------+ ↓ | [Branch B: Processing] | ↓ | [LayerNorm] → Normalize raw X | ↓ | [Multi-Head Attention] → Mix information | ↓ | [Residual Add] → X + Attention(LayerNorm(X)) <-----+ (Merge A & B) ↓ [Branch A: Keep Enhanced X] -----------------------+ ↓ | [Branch B: Processing] | ↓ | [LayerNorm] → Normalize enhanced X | ↓ | [Feed-Forward Network] → Process tokens | ↓ | [Residual Add] → X + FFN(LayerNorm(X)) <-----------+ (Merge A & B) ↓ Output: Enhanced X ∈ ℝ^(seq_len × d_model) ``` **Explanation**: - Each layer **refines token representations** while preserving original information. - The residual connections allow gradients to flow easily through very deep networks (50–100+ layers). ### **Pre-Norm vs Post-Norm** - **Pre-Norm (used by Llama 2, GPT-3, T5):** $X_{l+1} = X_l + \text{Sublayer}(\text{LayerNorm}(X_l))$ - **Post-Norm (original Transformer):** $X_{l+1} = \text{LayerNorm}(X_l + \text{Sublayer}(X_l))$ > Pre-norm is more **stable for deep networks** and reduces training instabilities. > ### **RMSNorm (Root Mean Square Normalization)** $$ \text{RMSNorm}(x) = \frac{x}{\sqrt{\text{mean}(x^2) + \epsilon}} \cdot \gamma $$ - Faster than standard LayerNorm (no mean subtraction) - Works well in **deep pre-norm networks** - γ is a learned scaling parameter ### **Why Residual Connections?** - **Enable gradient flow** in deep networks - **Preserve original token information** - **Allow training of very deep models** (50–100+ layers) > Without residuals, gradients can vanish and training very deep transformers becomes unstable. > ## **Part 4: Multi-Head Attention - The Information Blender** Multi-Head Attention is the mechanism that allows each token to “look at” other tokens in the sequence and gather relevant information. Think of it as a **dynamic, context-aware mixing of information**. ### **The Core: Scaled Dot-Product Attention** #### **Step 1: Create Q, K, V Projections** Given the input sequence embedding: $$ \mathbf{X} \in \mathbb{R}^{\text{seq\_len} \times d_{\text{model}}} $$ each token is linearly projected into three different spaces: - **Query (Q):** what the token is looking for - **Key (K):** what the token contains - **Value (V):** what the token shares $$ \mathbf{Q} = \mathbf{X}\mathbf{W}_Q,~\mathbf{K} = \mathbf{X}\mathbf{W}_K,~\mathbf{V} = \mathbf{X}\mathbf{W}_V $$ Each attention head has its own learned weight matrices: $$ \mathbf{W}_{Q}, \mathbf{W}_{K}, \mathbf{W}_{V} \in \mathbb{R}^{d_{\text{model}} \times d_{\text{head}}} $$ **Head Dimension:** $$ \\d_{\text{head}} = \frac{d_{\text{model}}}{\text{Number of Heads}} $$ - **Llama 2 (7B)**: 32 heads, d_model=4096 → d_head=4096/32=128 - **GPT-3 (6.7B)**: 32 heads × d_model=4096 → d_head=128 - **GPT-3 (175B)**: 96 heads × d_model=12288 → d_head=128 - Most modern models use d_head=64~128. > Each head processes information independently. Later, the heads are concatenated back together. > #### **Step 2: Compute Attention Scores** Each token asks: "How relevant is every other token to me?” $$ \text{Scores} = \mathbf{Q}\mathbf{K}^T \in \mathbb{R}^{\text{seq\_len} \times \text{seq\_len}} $$ Large score → token is more relevant. #### Step 3: Scale the Scores To prevent dot products from growing too large with dimension: $$ \text{Scores}_{\text{scaled}} = \frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d_{\text{head}}}} $$ This keeps the softmax stable and prevents gradients from vanishing. #### **Step 4: Causal Masking** For autoregressive models, each token can only attend to itself and previous tokens. Future positions are set to $−∞$ | **Query / Key** | **The** | **cat** | **sat** | **on** | | --- | --- | --- | --- | --- | | **The** | 2.5 | $−∞$ | $−∞$ | $−∞$ | | **cat** | 1.0 | 3.0 | $−∞$ | $−∞$ | | **sat** | 0.8 | 2.2 | 5.0 | $−∞$ | | **on** | 0.5 | 1.9 | 1.7 | 3.8 | #### **Step 5: Softmax: Turning Scores into Attention Weights** Softmax converts each row into a probability distribution: $$ \mathbf{A} = \text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d_{\text{head}}}}\right) $$ | **Query / Key** | **The** | **cat** | **sat** | **on** | | --- | --- | --- | --- | --- | | **The** | 1.00 | 0.00 | 0.00 | 0.00 | | **cat** | 0.12 | 0.88 | 0.00 | 0.00 | | **sat** | 0.01 | 0.06 | 0.93 | 0.00 | | **on** | 0.03 | 0.11 | 0.09 | 0.76 | #### **Step 6: Weighted Aggregation** Blend value vectors based on attention weights: $$ \text{Output}_i = \sum_{j} \text{AttentionWeight}_{i,j} \cdot \mathbf{V}_j $$ *Result:* A vector of size $d_{head}$ that contains the mixed context for that specific head. Example (for token “sat”): $$ \text{Output}_{\text{sat}} = (0.01 \times \mathbf{V}_{\text{The}}) + (0.06 \times \mathbf{V}_{\text{cat}}) + (0.93 \times \mathbf{V}_{\text{sat}}) $$ "sat" representation now contains context from all three tokens, weighted by relevance. The word "sat" now "knows" it's about a cat! ### The Merge (Multi-Head Integration) #### **Step 7:** Concatenation (Stitching) We take the output from all `n_head` heads and stack them side-by-side to recover the original width. $$ \text{Concat} = [\text{Head}_1, \text{Head}_2, \dots, \text{Head}_{n{\text{\_heads}}}] $$ *Result:* A vector of size $d_{model}$. #### **Step 8: Final Linear Projection** We run this stacked vector through one last weight matrix, the **Output Projection Matrix** ($\mathbf{W}_O$). $$ \text{MultiHeadOutput} = \text{Concat} \times \mathbf{W}_O $$ > **Why?** Concatenation just places the "Grammar" insight next to the "History" insight. This projection allows them to **mix** and exchange information before moving to the next layer. > #### **Step 9: Residual Connection** Finally, we add this result back to the original input $\mathbf{X}$: $$ \text{FinalOutput} = \mathbf{X} + \text{MultiHeadOutput} $$ This is called a **residual connection** or **skip connection**. ## **Part 5: Feed-Forward Network - The Pattern Expert** While Multi-Head Attention allows tokens to **"talk"** to each other, the Feed-Forward Network (FFN) allows each token to **"think"** and process that information individually. ### ReLU: The Classic Two-Layer Fold $$ \text{FFN}(x) = \text{ReLU}(x\mathbf{W}_1 + \mathbf{b}_1)\mathbf{W}_2 + \mathbf{b}_2 $$ **Where:** - $\mathbf{x} \in \mathbb{R}^{d_{\text{model}}}$ (a single token's vector) - $\mathbf{W}_1$ expands to a wider hidden dimension $d_{\text{ff}}$ (typically 2-4× wider) - $\mathbf{W}_2$ contracts back to $d_{\text{model}}$ The token's *relative position* to others doesn't change, but its **internal representation** is crumpled into a new shape that makes patterns linearly separable. #### **The Dying ReLU Problem** **Problem**: If a neuron's input is *always* negative during training, ReLU's gradient is **exactly zero**. The neuron "dies"—permanently stuck in the flat zero region, never learning again. In deep networks (50+ layers), this can kill 30% of capacity. ### **SwiGLU: The Gated Smoothing Solution** SwiGLU replaces ReLU's hard non-linearity with a smooth **Swish** gate, using **element-wise multiplication** between two parallel projections to dynamically regulate information flow. #### **Architecture: Two Parallel Paths** $$ \text{SwiGLU}(x) = \text{Swish}(x\mathbf{W}_{\text{gate}}) \odot (x\mathbf{W}_{\text{up}}) $$ **Where:** - $\odot$ represents **element-wise multiplication** - $\mathbf{W}_{\text{gate}}$ and $\mathbf{W}_{\text{up}}$ are the two parallel learnable projections - $\text{Swish}(z) = z \cdot \sigma(\beta z)$ is the smooth activation function **Path 1: The Gate** ("How *much* of each feature?") - $x\mathbf{W}_{\text{gate}}$: Learns a **scaling factor** for every hidden dimension - Swish: A smooth S-curve that can **partially** suppress (0.3×, 0.7×) instead of brutally zeroing **Path 2: The Value** ("What *data* am I scaling?") - $x\mathbf{W}_{\text{up}}$: Computes the **actual feature values** to be passed **Multiply**: `gate ⊙ value` = **smooth, selective amplification**. Each neuron decides *both* what to say and how loudly to say it. ### **Model-Specific FFN Dimensions & Activations**: - **Llama 2**: $d_{\text{ff}}$ = 11008 (~2.7× d_model), **SwiGLU** activation - **T5 Base**: $d_{\text{ff}}$ = 2048 (~2.7× d_model), **ReLU** activation - **GPT-3 models**: $d_{\text{ff}}$ = 4×d_model (16384 for 175B), **GeLU** activation **Why Different Activations?**: SwiGLU > GeLU > ReLU (older models). Each has different compute/memory tradeoffs. ![Activation Functions](https://vitalab.github.io/blog/images/activationfunctions/sc01.jpeg) ## **Part 6: The N-Layer Tower - Progressive Refinement** ### **Journey Through All Layers** **Layer 1-4: Syntax Detection** - **What happens**: Attention looks at local neighbors (±3 tokens). FFN memorizes patterns like "the word after 'the' is likely a noun." - **Result**: The vector for "sat" now carries a "past-tense verb" tag. **Layer 8-12: Semantic Assembly** - **What happens**: Attention spans full clauses ("cat" ↔ "sat" ↔ "mat"). FFN **composes** features into concepts: `feline + action + surface → "resting animal"`. - **Result**: "sat" now activates dimensions for "resting" and "feline context." **Layer 16-24: Pragmatic Inference** - **What happens**: Attention becomes sparse—"sat" only attends to "cat" and "on". FFN encodes world knowledge: "cats sit on soft surfaces when calm." - **Result**: The vector now lives near "domestic scene, low activity" in the model's latent space. **Layer 28-32+: Prediction-Ready Representation** - **What happens**: Final refinements. Attention cleans up noise. FFN sharpens the representation for next-token prediction. - **Result**: The vector is now optimized to compute token probabilities—ready to predict "the" or "." or "quietly." **Key Insight**: Each layer adds roughly the same amount of computation. Deeper = more nuanced understanding but slower generation. ## **Part 7: Output Generation - From Vector to Word** We have reached the final stage. The model has processed the input through all layers and produced a single, high-dimensional vector representing the **concept** of the next token. Now, we must convert that concept back into a specific word from the dictionary. ### **Step 7.1: Final Projection to Vocabulary** Convert the final layer's representation to logits (scores for each possible next token). ```python Input: Final Vector (h_final) for the last token position [0.89, -0.23, 1.12, ..., 0.45] ∈ ℝ^{d_model} Operation: Matrix Multiplication (Dot Product) Logits = h_final × W_vocab Where W_vocab is the Transpose of the Embedding Matrix. Shape: [d_model × vocab_size] Output: Raw Logits (Unnormalized Scores) [-2.1, 1.3, -0.5, 8.2, 0.8, ...] ∈ ℝ^{vocab_size} "apple" "boy" "cat" "mat" "dog" ``` > Larger vocab (more meaning per token) = fewer tokens per word = faster generation, but bigger embedding matrix > ### **Step 7.2: Sampling Strategies - Choosing the Next Token** Raw logits are hard to interpret (they can be negative or infinite). We push them through a pipeline to select the final token. #### **1. Temperature Scaling (The "Thermostat")** Before converting to probabilities, we adjust the logits using a hyperparameter: **Temperature ($T$)**. $$ P_i = \frac{\exp(logit_i / T)}{\sum \exp(logit_j / T)} $$ - **Low T (< 1.0):** Exaggerates differences. The "best" token gets a massive probability boost; unlikely tokens drop to near zero. (Logic/Code) - **High T (> 1.0):** Flattens differences. The gap between the "best" and "second best" shrinks, giving the underdog tokens a fighting chance. (Poetry/Brainstorming) #### **2. Softmax (Normalization)** This converts the scaled logits into a probability distribution that sums to 100% (1.0). - *Result:* `[0.001, 0.02, 0.85, 0.12, ...]` #### **3. Filtering Strategies (Top-k & Top-p)** To prevent the model from choosing complete garbage (the "tail" of the distribution), we truncate the options. - **Top-k (Hard Cutoff):** - *Logic:* "Only consider the top 50 candidates. Ignore the other 29,950." - *Pros:* Simple, guaranteed fixed computational cost. - *Cons:* In strict contexts, $k=50$ might include bad options. In broad contexts, $k=50$ might cut off good creative options. - **Top-p (Nucleus Sampling - Dynamic Cutoff):** - *Logic:* "Sort tokens by probability. Keep adding them to the list until their combined probability hits 90% (0.9)." - *Scenario A (The answer is obvious):* "The capital of France is..." -> Top-p might only keep 1 token ("Paris") because it holds 99% probability alone. - *Scenario B (Open ended):* "Once upon a..." -> Top-p might keep 100 tokens ("time", "midnight", "hill", etc.) because the distribution is flat. - *Verdict:* **Top-p is generally superior to Top-k** because it adapts to the uncertainty of the context. #### **4. Selection** Finally, the model rolls the dice on the remaining probability distribution to pick the Next Token. ## **Part 8: Quick Reference - Inference Guide** ### **Inference Configuration Cheat Sheet** | **Use Case** | **Temp** | **Top-p** | **Top-k** | **Repetition Penalty** | **Reasoning** | | --- | --- | --- | --- | --- | --- | | **Math / Coding** | 0.0 - 0.2 | 1.0 | 10 | 1.0 | **Determinism.** You want the single most likely answer every time. Creativity = Bugs. | | **Factual Q&A** | 0.2 - 0.4 | 0.9 | 20 | 1.05 | **Accuracy.** Slight flexibility helps natural phrasing, but hallucinations must be minimized. | | **Chat / Roleplay** | 0.7 - 0.8 | 0.9 | 40 | 1.1 | **Engagement.** High enough temp for natural variation, mild penalty to stop looping loops. | | **Creative Writing** | 0.9 - 1.1 | 0.95 | 100 | 1.15 | **Novelty.** Flatten the curve to allow rare words and unexpected plot twists. | ### **Common Pitfalls** 1. **Wrong Tokenizer**: Using GPT-4 tokenizer with Llama model = garbage output 2. **OOM Errors**: Batch size too large or sequence too long 3. **Bad Sampling**: Temperature too high = gibberish, too low = repetitive 4. **Format Mismatch**: Not using correct prompt format for instruction-tuned models 5. **No KV Cache**: Forgetting to cache = 10x slower generation