Skip to content
Indus AI Academy

How Transformers and Attention Actually Work

A clear, practical guide to how transformers work: the attention mechanism explained with a diagram, the real math, and working code you can run.

· IIT Delhi alumnus

· 12 min read

Abstract 3D visualization of colorful geometric blocks representing data flow in an AI model
Photo: Wes Cockx for Google DeepMind / Pexels

If you have ever wondered how transformers work well enough to power ChatGPT, Claude, Gemini and nearly every other modern language model, the honest answer fits in one sentence: they let every word in a sentence look directly at every other word, decide how relevant each one is, and blend their information together. That one mechanism, called attention, replaced a much clumsier approach that read text one word at a time, and it is worth understanding properly if you build with AI, evaluate AI tools at work, or just want to know what is actually happening behind the chat window.

This is not a surface-level analogy piece. We will build the intuition first, then go through the real mechanics: the query, key and value vectors, the formula that scores relevance, multi-head attention, and how attention gets stacked into a full transformer. You will see a diagram, the actual formula from the original paper, and a small working code example you can run yourself.

Key takeaways

  • Attention lets every word look at every other word directly, instead of reading text one token at a time like older recurrent models did.
  • Three vectors drive it: a query (what a word is looking for), a key (what each word offers), and a value (what each word contributes if chosen).
  • The core formula is simple: Attention(Q, K, V) = softmax(QKᵀ / √d_k) V — a weighted average, where the weights come from how well queries match keys.
  • Multiple attention heads run in parallel so the model can track several kinds of relationships (grammar, reference, topic) at once.
  • Attention has a real cost: compute and memory grow quadratically with input length, which is why "context window" limits and pricing exist.

The problem attention was built to solve

Before 2017, the best language models were recurrent neural networks (RNNs) and their more capable cousin, LSTMs. They read a sentence one word at a time, carrying forward a single "memory" vector that summarized everything seen so far. This created two real problems. First, information from early words faded as the sentence got longer, so models struggled to connect a pronoun to a noun many words back. Second, because each step depended on the previous one, you could not process a sentence in parallel — training was slow, even on fast hardware.

A Google research team addressed both problems in a paper called "Attention Is All You Need" (Vaswani et al., 2017), which introduced the Transformer architecture. Their core move was to drop recurrence entirely and let the model relate any two words directly, however far apart they are, using only attention. The paper reports its new model beat the previous best translation systems: 28.4 BLEU on English-to-German translation (more than 2 points above the prior state of the art, including ensembles) and a new single-model best of 41.8 BLEU on English-to-French, trained in 3.5 days on 8 GPUs — a fraction of what comparable models had needed. Nearly every large language model since, including GPT, Claude and Gemini, builds on this architecture.

The intuition: a sentence that reads itself

Picture the sentence: "The cat sat on the mat because it was tired." To understand what "it" refers to, you instinctively scan back through the sentence and decide "it" most likely means "the cat," not "the mat." You did this by comparing "it" against every other word and judging relevance.

Self-attention does the same thing mathematically, for every word, simultaneously. For each word, the model asks: given what I am (my query), how relevant is every other word in the sentence (their keys), and how much of their content (their values) should I absorb? IBM's explainer on self-attention describes this well: the mechanism produces "contextualized representations" — the vector for a word like "bank" ends up different depending on whether the sentence is about a river or a loan, because attention pulls in different surrounding words each time.

The real mechanics: queries, keys, values

Here is where the analogy becomes precise. Every word starts as an embedding — a vector of numbers learned during training that captures meaning. From that embedding, the model computes three new vectors using three separate learned weight matrices:

  • Query (Q): what this word is looking for in the rest of the sentence.
  • Key (K): what this word has to offer, used to be matched against other words' queries.
  • Value (V): the actual information this word contributes once it is selected.

To score how much one word should attend to another, the model takes the dot product of the first word's query with the second word's key, then scales it down by dividing by the square root of the key vector's dimension (written √d_k). This scaling matters more than it looks: as vector dimensions grow, raw dot products can become very large, which pushes the following softmax step into a region with tiny gradients and makes training unstable. Dividing by √d_k keeps the scores in a well-behaved range.

Those scaled scores go through a softmax function, which turns them into weights that sum to 1 — essentially, a percentage of attention paid to each word. Finally, the model computes a weighted sum of all the value vectors using those weights. The full operation, directly from the paper, is:

Attention(Q, K, V) = softmax(QKᵀ / √d_k) V
How self-attention turns a query word and the keys and values of a sentence into one context-aware vector, through scoring, softmax and a weighted sum
Self-attention in four steps, for one word in a three-word sentence

Seeing it in code

The clearest way to believe this is to run it. Here is self-attention, computed from scratch with nothing but basic linear algebra, for the toy sentence "The cat sat" (random weights stand in for the ones a real model would learn through training):

import numpy as np

np.random.seed(0)
tokens = ["The", "cat", "sat"]
d_model = 4  # tiny embedding size, just for the demo

X = np.random.randn(len(tokens), d_model)        # pretend embeddings
W_q = np.random.randn(d_model, d_model)          # learned, in a real model
W_k = np.random.randn(d_model, d_model)
W_v = np.random.randn(d_model, d_model)

Q, K, V = X @ W_q, X @ W_k, X @ W_v

def softmax(x):
    e = np.exp(x - np.max(x, axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

scores = Q @ K.T / np.sqrt(d_model)
weights = softmax(scores)
output = weights @ V
print(np.round(weights, 2))

Running this (Python 3.11, NumPy 2.4) prints a 3x3 matrix where each row sums to 1 — one row of attention weights per word, showing exactly how much that word attends to "The," "cat," and "sat":

[[0.   1.   0.  ]
 [0.06 0.04 0.9 ]
 [0.07 0.91 0.01]]

With real, trained weights instead of random ones, these rows stop being arbitrary and start reflecting actual grammatical and semantic relationships — which is exactly what training teaches the Q, K and V matrices to do.

Multi-head attention: more than one lens at once

A single attention calculation tends to settle on one dominant pattern. The paper's fix is multi-head attention: instead of computing attention once, the model computes it several times in parallel, each with its own learned Q, K and V projections, then concatenates the results. The base model in the original paper uses 8 heads, with a 512-dimensional embedding split into 64 dimensions per head (512 ÷ 8).

Each head can specialize. In the widely used walkthrough "The Illustrated Transformer", Jay Alammar shows a concrete example: for the sentence ending in "...because it was too tired," one attention head puts most of its weight on "the animal" when processing "it," while a different head attends more to "tired." Neither head alone tells the full story — together, they do. This is also why attention heads are one of the first things interpretability researchers inspect when trying to understand what a model has learned.

Putting word order back in: positional encoding

There is a catch with attention: it has no built-in sense of order. If you shuffled the words in a sentence, the attention calculation above would produce the same relationships, because it only compares vectors, not positions. Since word order matters for meaning (a language with subject-object-verb order, like Hindi or Japanese, depends on this just as much as English does), the Transformer adds a positional encoding vector to each word's embedding before attention runs. The original paper uses fixed sine and cosine functions at different frequencies for each dimension, which has the convenient property of letting the model generalize to sequences longer than it saw in training.

From one attention block to a full transformer

A single attention calculation is only part of the picture. The original Transformer stacks six encoder layers and six decoder layers, each combining a multi-head attention sub-layer with a simple feed-forward network, wrapped in residual connections and layer normalization that help gradients flow through deep stacks. Attention is used in three distinct ways inside this architecture:

Attention type Where it happens What it does
Encoder self-attention Inside the encoder Each input word attends to every other input word
Masked decoder self-attention Inside the decoder Each output word attends only to earlier output words, so the model can't "see the future" while generating
Encoder-decoder (cross) attention Between encoder and decoder The decoder attends to the encoder's output, to stay grounded in the input

The original paper was built for translation, so it needed both an encoder (to read the source sentence) and a decoder (to write the translation). Most of today's large language models, including the GPT family, are decoder-only: they drop the encoder and just stack masked self-attention and feed-forward layers, trained to predict the next token over and over. That single mechanism, repeated at scale, is most of what is happening when you chat with a modern AI system.

A quick jargon checklist

Use this as a reference the next time you read a technical explanation of a model:

  • Token: a chunk of text (often a word piece) the model processes as one unit.
  • Embedding: the vector of numbers representing a token's meaning before attention.
  • Attention head: one parallel attention calculation, specializing in some kind of relationship.
  • Context window: the maximum number of tokens a model can attend over at once.
  • Causal (masked) attention: attention restricted to only look backward, used for generating text left to right.

Where attention runs into real limits

Attention is powerful, but it is not free, and it is not magic. The cost of comparing every token to every other token grows quadratically with sequence length: double the input length, and the attention computation roughly quadruples. This is the direct, mechanical reason AI products advertise "context window" sizes and charge more for longer inputs — it is not an arbitrary business decision, it is a consequence of the architecture. It is also why a large share of recent AI research focuses on making attention cheaper for long sequences.

It is also worth being precise about what attention does not fix. Attention tells a model which parts of the input are relevant; it says nothing about whether the model's underlying knowledge is correct. A transformer can attend perfectly to the right part of a prompt and still generate a fluent, confidently wrong answer, because attention is a mechanism for combining information, not a mechanism for verifying it. That distinction matters anytime you are deciding how much to trust an AI-generated answer.

Why this is worth knowing, even if you don't train models

You do not need to implement attention to benefit from understanding it. If you use AI tools at work, this explains why giving a model better, more relevant context in your prompt genuinely changes its output — you are shaping what its attention has to work with. If you are evaluating AI products, it explains why "context window" and pricing by token count are not marketing fluff but direct consequences of how the architecture works. And if you want to go further — building retrieval systems, AI agents or your own small models — attention is the one concept everything else rests on. Our guide on how to build a RAG app from scratch and our explainer on what an AI generalist actually does both assume this foundation; if you want to see how researchers are now trying to look inside these attention-based models to understand their reasoning, our piece on AI interpretability research is a natural next read.

How to go deeper

Reading about attention gets you understanding; building with it gets you skill. If you want to go from "I understand how transformers work" to actually building retrieval systems, tool-using agents and small AI-powered apps, the AI Automation & Agents Bootcamp is built for exactly that jump. It is a live, online, lab-heavy program where you build real systems — including retrieval over your own documents and working agents — get your work reviewed, and finish with a portfolio project and a certificate anyone can verify, rather than just reading about the architecture.

If you are earlier in your journey and want a broader, practical AI skill set before specializing, the AI Generalist Certification covers prompting, automation and a first working agent, alongside live sessions you can join from anywhere.

Frequently asked questions

What is the difference between attention and self-attention?

Attention, in general, is any mechanism where one set of vectors (queries) is compared against another set (keys) to decide how much weight to give a third set (values). Self-attention is the specific case used inside transformers where the queries, keys and values all come from the same sequence — so a sentence attends to itself. Cross-attention, by contrast, has queries from one sequence (like a decoder) attending to keys and values from a different sequence (like an encoder's output).

Why do transformers need multiple attention heads instead of just one?

A single attention calculation tends to converge on one dominant pattern, such as grammatical agreement, and can miss other useful relationships happening in the same sentence. Running several attention calculations in parallel, each with its own learned projections, lets different heads specialize in different kinds of relationships — for example, one tracking what a pronoun refers to while another tracks sentence structure — and the results are combined afterward.

Do I need to know linear algebra to understand how transformers work?

Not to get a working understanding. The intuition — every word compares itself to every other word and blends in the relevant ones — doesn't require math. To follow the exact formula, it helps to know what a dot product, a matrix multiplication and a softmax function are, but you can learn those as you go; none of them are more advanced than what is taught in a first linear algebra course.

What is the difference between encoder-only, decoder-only and encoder-decoder transformers?

Encoder-only models (like the original BERT) read a full input at once and are typically used for understanding tasks such as classification. Decoder-only models (like the GPT family) generate text left to right, attending only to earlier tokens, which is why they're the dominant architecture for today's chat-style language models. Encoder-decoder models (like the original Transformer and T5) use both: an encoder to read an input sequence and a decoder to generate an output sequence, which suits tasks like translation and summarization.

Why can't transformers just handle unlimited amounts of text at once?

Because the attention calculation compares every token against every other token, its compute and memory cost grows quadratically with the length of the input. A context window twice as long needs roughly four times the computation for attention alone. That is the direct technical reason models have maximum context windows and why providers often charge more as inputs get longer, and it is an active area of AI research aimed at making long-context attention cheaper.

Found this useful?

Next cohort starts 2 Dec

AI Automation & Agents Bootcamp

Design, build and run AI agents and automations in production.

  • Mondays and Wednesdays, 7–10 PM IST
  • 6 weeks · 12 sessions · 36 live hours · Online
  • For: Operations, product and technical teams, developers and analysts
  • ₹44,999 per seat, taxes extra
Reserve your seat
Vivek Gupta

About the author

· IIT Delhi alumnus

Vivek Gupta is an IIT Delhi alumnus and PhD scholar, an AI researcher and serial entrepreneur with 14 years of building technology at MakeMyTrip, Goibibo, Wissen and Jubilant FoodWorks.

More articles by VivekConnect on LinkedIn

Discussion

No comments yet. Start the discussion.

Turn ideas into skills.

Live, hands-on AI programs for professionals and teams, with certificates issued by INDUS AI Private Limited.