What is Attention Mechanism? — AI Glossary

The attention mechanism is a technique that allows an AI model to focus on the most relevant parts of its input when producing each output. Instead of processing all input equally, attention dynamically weighs different parts based on their relevance — enabling models to understand context and relationships across long sequences of text or data.

Attention is the breakthrough idea that made the transformer architecture possible — and therefore is the foundation of every major language model, including GPT-4, Claude, and Gemini. The 2017 paper “Attention Is All You Need” introduced the transformer and demonstrated that attention alone, without recurrence, could power state-of-the-art natural language processing.

How Attention Works

Attention answers the question: “Given the current position I’m computing, which other positions in the input are most relevant?” For each output position, the mechanism computes a weighted sum of information from all input positions, where the weights reflect relevance.

The computation uses three vectors derived from each token:

• Query (Q) — “What am I looking for?”
• Key (K) — “What do I have to offer?”
• Value (V) — “What information should I pass along if selected?”

The attention score between a query and a key is their dot product (scaled). These scores are passed through a softmax function to produce a probability distribution — the attention weights — which are used to compute a weighted sum of the values. The result is a context-aware representation of each token.

Self-attention applies this mechanism within a single sequence — each token attends to all other tokens, learning their relationships. Cross-attention applies it between two sequences — common in encoder-decoder models where the decoder attends to the encoder’s output (e.g., in translation).

Multi-Head Attention

Transformers use multi-head attention: running several attention computations in parallel with different learned Q, K, V projections. Each “head” can focus on different types of relationships — one head might track syntactic dependencies, another might track semantic similarity, another might track coreference. The outputs of all heads are concatenated and projected to form the final representation.

GPT-4 uses 96 attention heads per layer across 120+ layers, with each head learning to recognize different linguistic and factual patterns. This is what allows large models to maintain coherent, contextually aware responses across long conversations.

Why Attention Matters

Before attention, language models used recurrent neural networks (RNNs) that processed text sequentially — one token at a time. This made it hard to relate tokens far apart in a sequence (the “long-range dependency” problem) and prevented parallelization during training.

Attention solves both problems: every token can directly attend to every other token regardless of distance, and all attention computations can run in parallel. This is why transformers scale so effectively on modern GPU hardware, enabling the training of models with hundreds of billions of parameters.

The context window of a language model is directly tied to attention: the model can attend to all tokens within its context window simultaneously. Extending context windows to millions of tokens is an active research area, with techniques like sliding window attention and linear attention variants making long-context processing practical.

Common Misconceptions

Misconception: Attention is the same as the transformer. Attention is a component; the transformer is an architecture built from attention layers, feed-forward layers, and normalization. There are also attention mechanisms used outside transformers.

Misconception: Attention means the model “focuses” like a human does. Attention weights are mathematically determined patterns in vector space. They don’t reflect conscious focus — they are a learned mechanism for combining information, some of which we can visualize as attention heatmaps but which don’t directly map to human intuition.

---

Key Takeaways

• Attention dynamically weighs different input positions based on their relevance to each output position.
• It uses Query, Key, and Value vectors to compute weighted combinations of input information.
• Multi-head attention runs multiple attention computations in parallel for richer representations.
• Attention enabled the transformer architecture, which powers all major modern language models.
• It solves the long-range dependency problem and enables fully parallel training.

Frequently Asked Questions

What is self-attention?

Self-attention applies the attention mechanism within a single sequence, where each token computes attention scores against all other tokens in the same sequence. This allows the model to build contextual representations where each token’s meaning is informed by its relationship to all other tokens in the input.

What is the computational complexity of attention?

Standard attention is O(n²) in sequence length — computing attention scores for all pairs of tokens. For a 4,000-token context, that’s 16 million pairwise computations. This quadratic scaling is why extending context windows is expensive and why efficient attention variants (FlashAttention, sparse attention) are important research areas.

What is FlashAttention?

FlashAttention is an efficient attention implementation that reduces GPU memory usage and increases speed by computing attention in tiles that fit in fast memory (SRAM) rather than slow memory (HBM). It produces identical results to standard attention but is 2–4x faster and uses far less memory, enabling longer context windows practically.

How does attention relate to the context window?

The context window defines how many tokens a model can attend to simultaneously. Larger context windows allow the model to reference earlier parts of a conversation or document, but increase attention computation quadratically. This is the core engineering challenge in building models with million-token context windows.

What is cross-attention?

Cross-attention applies the attention mechanism between two different sequences. In translation models, the decoder uses cross-attention to attend to the encoder’s representation of the source sentence when generating each target word. In image-text models, it allows the language decoder to attend to visual features from the image encoder.

---

Sources: Grokipedia — Attention Mechanism · arXiv: Attention Is All You Need (Vaswani et al., 2017) · The Illustrated Transformer (Jay Alammar)

You May Also Like

• What Is Artificial Intelligence
• AI Glossary
• AI for Dummies
• How Does AI Work
• Essential AI Skills

---

Get free AI tips daily → Subscribe to Beginners in AI

Sources

This article draws on official documentation, product pages, and industry reporting. Specific sources are linked inline throughout the text.

Last reviewed: April 2026