118: Visualizing Attention Patterns

Chapter Overview

We know that [[113-Multi-Head-Attention|Multi-Head Attention]] allows a model to focus on different parts of the input. But what do the individual attention heads actually learn to look at? By visualizing the attention weights, researchers have identified several common and interpretable patterns.

---

What are Attention Patterns?

An attention pattern is a visualization of the attention weights between all tokens in a sequence for a specific attention head. It's typically shown as a heatmap where a bright color indicates a high attention score between a "query" token (row) and a "key" token (column).

This allows us to peer inside the "black box" and understand what relationships a particular head has learned to prioritize.

Common Interpretable Patterns

Researchers have discovered that different heads in a trained Transformer often specialize in specific, human-understandable tasks.

%%{init: {
  'theme': 'base',
  'themeVariables': {
    'primaryColor': '#2563eb',
    'primaryTextColor': '#1e40af',
    'primaryBorderColor': '#3b82f6',
    'lineColor': '#6b7280',
    'secondaryColor': '#f1f5f9',
    'tertiaryColor': '#e2e8f0',
    'background': '#ffffff',
    'mainBkg': '#f8fafc',
    'secondBkg': '#e2e8f0',
    'tertiaryBkg': '#cbd5e1'
  }
}}%%

flowchart TD
    subgraph Input ["📝 Input Sentence"]
        direction TB
        Sentence["'The quick brown fox jumps over the lazy dog .'"]
    end

    subgraph Attention ["🔍 Specialized Attention Heads"]
        direction TB

        subgraph Head1 ["Head 1: Positional Attention"]
            H1_Desc["Focuses on adjacent tokens<br/>Creates sequential dependencies"]
            H1_Pattern["Pattern: Token → Previous Token"]
        end

        subgraph Head2 ["Head 2: Syntactic Attention"]
            H2_Desc["Links grammatical relationships<br/>Connects verbs with subjects/objects"]
            H2_Pattern["Pattern: 'jumps' → 'fox', 'over' → 'dog'"]
        end

        subgraph Head3 ["Head 3: Delimiter Attention"]
            H3_Desc["Aggregates sentence information<br/>All tokens attend to punctuation"]
            H3_Pattern["Pattern: All tokens → '.'"]
        end

        subgraph Head4 ["Head 4: Semantic Attention"]
            H4_Desc["Identifies conceptual relationships<br/>Links related meanings"]
            H4_Pattern["Pattern: 'fox' → 'dog', 'quick' → 'lazy'"]
        end
    end

    subgraph Output ["📊 Attention Visualization"]
        direction TB
        Heatmap["Heat maps showing attention weights<br/>Bright colors = High attention<br/>Dark colors = Low attention"]
    end

    Input --> Attention
    Attention --> Output

    %% Styling
    classDef inputStyle fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af
    classDef headStyle fill:#f0f9ff,stroke:#0ea5e9,stroke-width:2px,color:#0c4a6e
    classDef outputStyle fill:#ecfdf5,stroke:#10b981,stroke-width:2px,color:#047857
    classDef patternStyle fill:#fef3c7,stroke:#f59e0b,stroke-width:1px,color:#92400e

    class Input inputStyle
    class Head1,Head2,Head3,Head4 headStyle
    class Output outputStyle
    class H1_Pattern,H2_Pattern,H3_Pattern,H4_Pattern patternStyle

1. Positional Attention Heads

These heads learn to focus on tokens at specific relative positions, most commonly: - Previous token attention: Each token attends primarily to the token immediately before it - Next token attention: Each token looks ahead to the following token - Fixed offset attention: Consistent attention to tokens at a specific distance (e.g., 3 positions back)

Why this matters: Positional patterns help the model understand sequence order and local dependencies, which is crucial for language understanding.

2. Syntactic Attention Heads

These heads capture grammatical relationships: - Subject-verb connections: Verbs attend to their subjects - Verb-object links: Action words focus on what they act upon - Modifier relationships: Adjectives attend to the nouns they modify

Example: In "The quick brown fox jumps", a syntactic head might show strong attention from "jumps" back to "fox".

3. Delimiter Attention Heads

These heads use punctuation and separator tokens as information aggregators: - Sentence-ending punctuation: All tokens in a sentence attend to the final period - Comma attention: Tokens attend to commas that separate clauses - Special token focus: Strong attention to [CLS], [SEP], or other special tokens

Purpose: These patterns help the model aggregate information across the entire sequence.

4. Semantic Attention Heads

These heads identify meaningful content relationships: - Coreference resolution: Pronouns attend to their antecedents - Thematic similarity: Related concepts attend to each other - Long-range dependencies: Tokens attend to semantically related tokens far away

Example: In a passage about animals, words like "dog", "cat", and "pet" might show mutual attention patterns.

---

Practical Applications

1. Model Debugging

Attention visualizations help identify: - Heads that aren't learning useful patterns - Attention collapse (all heads learning similar patterns) - Unexpected or problematic attention behaviors

2. Model Interpretability

Understanding attention patterns helps: - Explain model predictions to users - Build trust in AI systems - Identify potential biases in attention

3. Architecture Design

Attention analysis informs: - Optimal number of attention heads - Head pruning strategies - Architectural improvements

---

Limitations and Considerations

Not Perfect Explanations

While attention patterns are intuitive, they have limitations: - Attention ≠ Importance: High attention doesn't always mean high importance for the final prediction - Indirect Effects: The model might use attention indirectly in ways not immediately apparent - Layer Interactions: Attention patterns in one layer affect all subsequent layers

Evolution During Training

Attention patterns change as the model learns: - Early training often shows random or uniform attention - Specialized patterns emerge as training progresses - Over-training can lead to attention collapse

---

Key Takeaways

1. Specialization: Different attention heads learn to focus on different types of relationships
2. Interpretability: Attention patterns provide valuable insights into model behavior
3. Debugging Tool: Visualizations help identify and fix attention-related issues
4. Not Perfect: Attention patterns are helpful but not complete explanations of model behavior

Understanding attention patterns bridges the gap between the mathematical mechanics of attention and the intuitive linguistic relationships that make language models effective. This knowledge is crucial for both researchers developing new architectures and practitioners working to understand and improve model performance.

---