// GLOSSARY · TRANSFORMER ARCHITECTURE

Transformer architecture

The neural-network design behind modern LLMs, from the 2017 paper 'Attention Is All You Need.' It reads a whole sequence in parallel using self-attention.

Last verified · 2026-10-23 · by Moe Ameen

Published 2026-09-22 · Updated

Short answer

The Transformer is the neural-network architecture behind nearly every modern large language model. Eight researchers introduced it in "Attention Is All You Need," posted on June 12, 2017 and presented at NIPS 2017. It drops recurrence and reads a whole sequence in parallel, using self-attention to weigh how much each token should affect every other token. GPT and BERT both build on it.

Key takeaways

  • The Transformer came from the 2017 paper "Attention Is All You Need," posted June 12, 2017.
  • It scored 28.4 BLEU on English-German and 41.8 on English-French after 3.5 days on eight GPUs.
  • Self-attention lets any token look at any other token directly, however far apart they sit.
  • BERT (2018) uses the encoder side; GPT models use the decoder side.
  • The architecture predicts likely tokens. It does not check facts, so drafts still need review.

What is the Transformer architecture?

The Transformer is the neural-network architecture that nearly every modern large language model is built on. It was introduced in the 2017 paper "Attention Is All You Need" by eight researchers at Google, and its central idea is in the title: instead of processing text word by word in order, a Transformer looks at the whole sequence at once and uses a mechanism called attention to weigh how much every token should be influenced by every other token. The "T" in GPT and BERT stands for Transformer, and it is the reason those models exist in their current form.

What made it a break from the past is that it dispensed with recurrence and convolutions — the two techniques that dominated sequence modeling before it. Earlier models like RNNs and LSTMs read a sentence left to right, one step at a time, carrying a running memory forward; that ordering made them slow to train and prone to losing track of long-range connections. A Transformer has no such loop. Every position is processed in parallel, and self-attention lets any word directly consult any other word in the input regardless of distance, so "it" at the end of a paragraph can look straight back at the noun it refers to near the start. Because there is no left-to-right loop to encode word order, the model adds positional encodings — signals that tell it where each token sits in the sequence.

The mechanical heart is scaled dot-product attention over three learned projections of each token: a query, a key, and a value. Loosely, each token's query is compared against every token's key to produce relevance weights, and those weights pull a blend of the values — a data-driven way of deciding what to pay attention to. The model runs several of these in parallel (multi-head attention) so different heads can track different kinds of relationships — grammar, reference, topic — at once. Around the attention layers sit feed-forward networks, residual connections, and layer normalization, stacked dozens of times deep. The original design was an encoder-decoder built for translation, but the two halves split into their own lineages: encoder-only models like BERT (built to understand text) and decoder-only models like the GPT and Claude families (built to generate it).

The quiet reason the Transformer took over is not just quality but parallelism. Removing the sequential bottleneck meant the architecture could be trained efficiently on modern GPUs across enormous datasets, and it kept improving as models, data, and compute grew rather than plateauing. That scalability is what turned a 2017 translation paper into the foundation of the entire generative-AI era — the same architecture, scaled up, powers today's chatbots, coding assistants, and image and video models.

When was the Transformer invented, and what came after it?

The Transformer was published on June 12, 2017, in the arXiv paper "Attention Is All You Need" (Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin), most of them at Google, and presented at the NeurIPS conference later that year. The immediate goal was machine translation, where the model set new state-of-the-art BLEU scores on English-to-German and English-to-French while training in a fraction of the time earlier architectures needed — the paper's headline results were 28.4 BLEU on English-to-German and 41.8 on English-to-French. Attention itself was not new; it had been bolted onto recurrent translation models since around 2014. The paper's radical move was to throw the recurrence away and keep only the attention.

From there the lineage is direct. In 2018 Google released BERT, an encoder-only Transformer that learned deep bidirectional representations of language and reset benchmarks across natural-language understanding. In parallel OpenAI built the GPT series — decoder-only Transformers trained to predict the next token — scaling from GPT-1 through GPT-2 and GPT-3 (175 billion parameters, described in 2020) and into the models that put generative AI in front of the public. The pattern that emerged, sometimes called the scaling era, was that the same architecture kept getting better as parameters, data, and compute increased, which is why the industry poured resources into scaling Transformers rather than replacing them. Nearly a decade on, the Transformer remains the dominant design; challengers like state-space models and, more recently, diffusion language models exist and win specific tradeoffs, but as of 2026 the strongest general models are still Transformers or close hybrids of one.

What does self-attention do in practice?

  • A translation model reads an entire source sentence at once and, via self-attention, lets the word being translated attend directly to the relevant source words no matter how far apart they sit — the task the original 2017 paper was built to win, and where it first beat the recurrent models it replaced.
  • Resolving a pronoun: in "the trophy did not fit in the suitcase because it was too big," attention lets "it" weight "trophy" over "suitcase," because the architecture can connect any two tokens directly rather than passing a fading memory down a chain.
  • Multi-head attention in action: on one pass through a sentence, one head tracks subject-verb agreement while another tracks which adjective modifies which noun and a third tracks the overall topic — parallel views of the same text that a single attention pass could not capture.
  • A creator never touches any of this directly, but every AI-drafted caption, blog article, and script they generate is a Transformer predicting tokens — the architecture is the engine under the 'write me a hook' button, not a feature they configure.

What do people get wrong about Transformers?

  • Confusing the Transformer (an architecture) with a specific model like GPT-4 or Claude. The Transformer is the blueprint; GPT and Claude are particular models built from that blueprint and trained on particular data. Many different models share the one architecture.
  • Assuming a Transformer reads left to right like we do. It processes the whole sequence in parallel and injects word order through positional encodings; the left-to-right feel of a chatbot comes from how decoder-only models generate output one token at a time, not from how the architecture reads input.
  • Thinking "attention" means the model consciously focuses. Attention is a weighting operation — comparing queries and keys to blend values — not awareness. It is a math routine for deciding which tokens influence which, nothing more.
  • Treating the Transformer as obsolete because newer designs exist. State-space models and diffusion language models are real and win specific tradeoffs, but as of 2026 the dominant general-purpose models are still Transformers or hybrids. It has not been replaced.
  • Believing the architecture guarantees correctness. A Transformer predicts likely tokens; it has no built-in fact-checker. Fluent, confident, and wrong is a failure mode of the whole family, which is why generated content still needs human review.

How much do content creators need to know about Transformers?

The Transformer is one of those rare ideas that is worth understanding at exactly one level of depth if you make content rather than build models: know that it reads everything at once and decides, through attention, what matters to what. That single property — any token can consult any other token directly — is why LLMs got good at holding a thought across a long prompt, and it is why context quality matters so much (see context engineering). You are not tuning the architecture; you are feeding it, and it rewards a clean, well-ordered input because attention is only as useful as the tokens you give it to attend to.

The practical mindset I would keep is that the architecture is a commodity and the operation around it is not. Every serious model — the ones drafting your captions, blogs, and scripts — is a Transformer, so the drafting layer is not where anyone wins anymore; the win is in what you do with the draft. That is deliberately the layer Kompozy owns and the reason it stays model-agnostic: it runs the same content operation on top of whichever Transformer is best today (Claude and OpenAI for copy right now) and could adopt a stronger one tomorrow without the workflow changing, because a Persona Brief governs the voice and the publishing engine handles publishing to 9 platforms plus blog and email. Understand the Transformer well enough to respect what it can and cannot do — it predicts, it does not know — and then spend your effort on the part it does not solve: turning a good prediction into finished, on-brand, scheduled content.

What does this look like in Kompozy?

A real five-slide carousel from the Kompozy workspace. Every caption and slide line in a format like this starts as text from a Transformer-based model.

Five-slide carousel generated in Kompozy, slide 1Five-slide carousel generated in Kompozy, slide 2Five-slide carousel generated in Kompozy, slide 3Five-slide carousel generated in Kompozy, slide 4Five-slide carousel generated in Kompozy, slide 5
Real output · Kompozy Carousel format · from the Kompozy workspace

Who is this for, and who should skip it?

Best for:

  • Creators and marketers who want to know what is under the "write me a hook" button.
  • Anyone deciding how much effort to put into clean, well-ordered prompts and context.
  • Readers who keep hearing "GPT" and "Transformer" and want the difference.

Not ideal for:

  • Engineers building or training models, who need the paper itself, not a glossary entry.
  • Readers shopping for a specific model. The architecture is shared across vendors, so it will not pick one for you.

Frequently asked questions

What is the Transformer architecture?

The Transformer is a neural-network architecture, introduced in the 2017 paper "Attention Is All You Need," that processes an entire sequence of text in parallel and uses a mechanism called attention to weigh how much each token should be influenced by every other token. It replaced the older recurrent and convolutional approaches to sequence modeling and is the foundation of modern large language models — the "T" in GPT and BERT stands for Transformer.

Who invented the Transformer and when?

It was introduced on June 12, 2017, in the paper "Attention Is All You Need" by eight researchers — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin — most of them at Google, and presented at the NeurIPS conference in 2017. It was built for machine translation and set new state-of-the-art results while training far faster than the recurrent models it replaced.

How is a Transformer different from an RNN or LSTM?

RNNs and LSTMs read a sequence one step at a time, carrying a running memory forward, which makes them slow to train and prone to losing long-range connections. A Transformer removes that loop: it processes every position in parallel and uses self-attention so any token can directly connect to any other token regardless of distance. Because there is no left-to-right loop, it adds positional encodings to represent word order.

What is self-attention?

Self-attention is the core operation of a Transformer. Each token is projected into a query, a key, and a value; every token's query is compared against every token's key to produce relevance weights, and those weights blend the values into a new representation. In effect it is a data-driven way of deciding which other words each word should pay attention to. Multi-head attention runs several of these in parallel so different heads can track grammar, reference, and topic at once.

Is the Transformer still used in 2026?

Yes — it remains the dominant architecture behind large language models nearly a decade after its introduction. Newer designs like state-space models and diffusion language models exist and win specific tradeoffs such as speed or long-context efficiency, but as of 2026 the strongest general-purpose models are still Transformers or close hybrids of one. The architecture has kept improving with scale rather than being replaced.

What is the difference between encoder-only, decoder-only, and encoder-decoder Transformers?

The original 2017 Transformer was an encoder-decoder built for translation. Those halves later split into their own families. Encoder-only models like BERT read text bidirectionally to understand it, which suits classification and search. Decoder-only models like the GPT and Claude families generate text one token at a time, which suits chat and content generation. Encoder-decoder designs are still used for tasks like translation and summarization that map one sequence to another.

How do we know this?

We checked every figure, date and platform rule on this page against the sources listed below on 2026-10-23, and rewrote or removed anything we could not confirm. Paper dates, authors and benchmark scores come from the arXiv and NeurIPS records. Kompozy plan and product details come from kompozy.io.

Sources

  1. arXiv — Attention Is All You Need (Vaswani et al., 2017) (read 2026-10-23)
  2. NeurIPS Proceedings — Attention is All you Need (NIPS 2017) (read 2026-10-23)
  3. Google Research — Transformer: A Novel Neural Network Architecture for Language Understanding (read 2026-10-23)
  4. arXiv — Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau et al., 2014) (read 2026-10-23)
  5. arXiv — BERT: Pre-training of Deep Bidirectional Transformers (Devlin et al., 2018) (read 2026-10-23)
  6. arXiv — Language Models are Few-Shot Learners (GPT-3, 2020) (read 2026-10-23)
  7. Kompozy — Home (read 2026-10-23)
  8. Kompozy — Pricing (read 2026-10-23)

Related terms

  • Diffusion language model — A language model that generates text by denoising a whole sequence in parallel — like an image model — instead of predicting one token at a time.
  • Context Engineering — Curating the whole set of tokens a model sees at inference — instructions, tools, references, memory, retrieved data — not just the wording of one prompt.
  • Model Context Protocol (MCP) — An open standard that lets an AI assistant connect to your files, tools, and data sources through one common interface — the "USB-C port for AI" that marketers use to give a model live access to their analytics, CRM, and CMS.
  • AI glossary (2026) — A plain-English reference to the AI terms creators actually run into in 2026 — LLM, token, prompt, hallucination, multimodal, agent, RAG, diffusion, fine-tuning, and inference — with what each one means for the person making content.
Related deep guides

← All terms · Get started →