Start a project
Foundations Reviewed 2 Oct 2026 · 22 sources

transformer

/trænsˈfɔːrmər/ noun Plural transformers · adjective transformer-based
Definition

A neural network architecture that processes a whole sequence at once, using attention to weigh how strongly each part, such as a word, relates to every other part.

Examples
  • OpenAI’s GPT models, whose T stands for transformer
  • Google’s BERT, applied to ranking search results in 2019
  • OpenAI’s Whisper, a speech recognition system
  • NVIDIA’s Nemotron 3 Ultra, a 2026 transformer and Mamba hybrid
A row of word tiles spells I arrived at the bank after crossing the river. Curved lines run from the tile bank to every other tile, and the line to river is the thickest.
Inkling · Lines of attention from one word to all the othersFig. T1
Earliest AI use
1997
Graph Transformer Networks: modules that turn one graph into another.
Current meaning
2017
Vaswani and colleagues name a translation model built on attention alone.
Coined by
Vaswani et al.
Co-author Jakob Uszkoreit came up with the name, Vaswani told NVIDIA.
§01

In plain words

In AI, a transformer is a neural network design that uses attention to link each part of its input to every other part; most large language models use one.

The leading language systems before 2017 mostly ran on recurrent networks. These read a sentence one word at a time, passing a running summary along. Linking two words far apart took many steps, and the work was hard to spread across many chips.1

A transformer takes in every word of a passage at once. For each word, it scores how much every other word matters to it. In “I arrived at the bank after crossing the river”, those scores can tie “bank” to “river”, marking it as a riverbank.1

The design comes from a 2017 paper, “Attention Is All You Need”, by 8 researchers, most of them at Google. They built it for translation. Their largest version trained in 3.5 days on 8 graphics chips, a small fraction of the cost of the best earlier systems.2 Ashish Vaswani has said the team nearly named it after attention, and that co-author Jakob Uszkoreit proposed Transformer.3

Adoption was fast. OpenAI built its GPT models on the design, and the T in GPT stands for transformer. Google built BERT on it and in 2019 began using BERT to rank search results. Researchers then applied transformers to images and to predicting protein structures.4 Most large language models are transformers.

The design has rivals. By late 2025 the business press was writing about a post-transformer era. Some 2026 models keep attention in only part of their layers.

The word is easy to misread. In everyday English, a transformer is an electrical device that raises or lowers voltage. In AI, the transformer is the design; a large language model is a system trained on text, usually built with that design.

§02

How it works

First the input is split into tokens, and each token becomes an embedding, a list of numbers that stands for its meaning. Because all tokens are read at once, the model also adds a signal for each token’s position.2

The tokens then pass through a stack of identical layers. In each layer, attention compares every token with every other token and scores each pair. The scores set how much each other token feeds into the token’s updated list of numbers. That weighted blend then passes through a small conventional network.1 Several sets of attention, called heads, run side by side. The 2017 model used 8 heads and stacked 6 layers to read its input.2 GPT-3 has 96 layers, each with 96 heads.5

When a transformer writes text, each token may look only at the tokens before it.2 After the last layer, the model scores every possible next token, and one is chosen.

The comparing is costly. For n tokens, attention makes n² pairs, so doubling the input roughly quadruples that work. Each model is trained with a fixed maximum length, its context window. Alternatives such as Mamba drop attention so that cost grows only in step with length.6 Hybrids mix the two. NVIDIA’s Nemotron 3 Ultra uses Mamba layers for long inputs and keeps transformer layers for exact recall.

Fig. T2 · Where one word looksIllustrative, not measured

I arrived at the bank after crossing the river

  1. river 48%
  2. crossing 14%
  3. arrived 10%
  4. bank (itself) 9%
  5. the (twice) 8%
  6. after 5%
  7. I 4%
  8. at 2%
Invented attention scores, in percent, for the word “bank” in the example sentence from Google’s 2017 post. Most weight goes to “river”, which tells the model this bank is a riverbank.
§03

Origin and first known use

From the verb transform, from Old French and from Latin transformare, “change in shape”. The noun has named an electrical device since 1882. Machine learning used it by 1997, when Léon Bottou, Yann LeCun and Yoshua Bengio proposed Graph Transformer Networks. The current sense dates from June 2017. Jakob Uszkoreit told Wired that he and two colleagues chose the name from the start, in their first design document. The mechanism transforms the information it takes in, and he had owned Transformers toys as a child. Ashish Vaswani recalled that the team nearly named it after attention. Llion Jones suggested the paper’s title, after the Beatles song “All You Need Is Love”.

  1. 1882 Transformer, the agent noun from the verb transform, comes to name an electrical device.
    Online Etymology Dictionary, “transformer” Etymonline →: source for 1882
  2. 1997 Bottou, Le Cun and Bengio propose Graph Transformer Networks, a machine learning method for systems built from modules that take graphs as input and produce graphs.
    Bottou, Le Cun and Bengio, “Global Training of Document Processing Systems using Graph Transformer Networks”, CVPR 1997 leon.bottou.org →: source for 1997
  3. 2014 Bahdanau, Cho and Bengio let a translation model search the source sentence for the parts relevant to each word it predicts, instead of one fixed-length summary.
    Bahdanau, Cho and Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate”, ICLR 2015 arXiv 1409.0473 →: source for 2014
  4. 2017 Vaswani and 7 co-authors propose the Transformer, built on attention alone, and report better translation quality with far less training time than earlier models.
    Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin, “Attention Is All You Need”, arXiv, June 2017; NIPS 2017 arXiv 1706.03762 →: source for 2017
  5. 2019 Google says BERT, a transformer-based model, will help its search engine understand 1 in 10 English-language searches in the US when ranking results.
    Pandu Nayak, “Understanding searches better than ever before”, Google, 25 October 2019 blog.google →: source for 2019
  6. 2020 Dosovitskiy and colleagues apply a pure transformer to images split into patches. Their Vision Transformer rivals leading convolutional networks at image classification with less training compute.
    Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale” arXiv 2010.11929 →: source for 2020
  7. 2023 Gu and Dao propose Mamba, an architecture without attention whose cost grows linearly with sequence length, as an alternative to the transformer.
    Gu and Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces” arXiv 2312.00752 →: source for 2023
  8. 2026 NVIDIA releases Nemotron 3 Ultra, an open hybrid model: Mamba layers handle long inputs, while transformer layers keep precise recall of specific facts.
    Chris Alexiuk and Chintan Patel, “NVIDIA Nemotron 3 Ultra Powers Faster, More Efficient Reasoning for Long-Running Agents”, NVIDIA Technical Blog, 4 June 2026 developer.nvidia.com →: source for 2026
§04

In the wild

10 sightings · 2017–2026 · oldest first
  1. Paper12 Jun 2017

    “We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.”

    Vaswani et al., arXiv preprint arXiv 1706.03762 →: Vaswani et al., arXiv preprint, 12 Jun 2017Archived →: copy of Vaswani et al., arXiv preprint, 12 Jun 2017
  2. Blog31 Aug 2017

    “…the Transformer requires less computation to train and is a much better fit for modern machine learning hardware…”

    Jakob Uszkoreit, Google Research blog research.google →: Jakob Uszkoreit, Google Research blog, 31 Aug 2017Archived →: copy of Jakob Uszkoreit, Google Research blog, 31 Aug 2017
  3. Paper11 Oct 2018

    “We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers.”

    Devlin et al., arXiv preprint arXiv 1810.04805 →: Devlin et al., arXiv preprint, 11 Oct 2018Archived →: copy of Devlin et al., arXiv preprint, 11 Oct 2018
  4. Blog14 Feb 2019

    “GPT‑2 is a large transformer-based language model with 1.5 billion parameters, trained on a dataset of 8 million web pages.”

    Alec Radford and colleagues, OpenAI blog openai.com →: Alec Radford and colleagues, OpenAI blog, 14 Feb 2019Archived →: copy of Alec Radford and colleagues, OpenAI blog, 14 Feb 2019
  5. Headline10 Mar 2022

    “Will Transformers Take Over Artificial Intelligence?”

    Quanta Magazine quantamagazine.org →: Quanta Magazine, 10 Mar 2022Archived →: copy of Quanta Magazine, 10 Mar 2022
  6. Blog21 Sep 2022

    “The Whisper architecture is a simple end-to-end approach, implemented as an encoder-decoder Transformer.”

    OpenAI, Whisper announcement openai.com →: OpenAI, Whisper announcement, 21 Sep 2022Archived →: copy of OpenAI, Whisper announcement, 21 Sep 2022
  7. Headline23 Jul 2023

    “Transformers: the Google scientists who pioneered an AI revolution”

    Financial Times ft.com →: Financial Times, 23 Jul 2023Archived →: copy of Financial Times, 23 Jul 2023
  8. Headline20 Mar 2024

    “8 Google Employees Invented Modern AI. Here’s the Inside Story”

    Wired wired.com →: Wired, 20 Mar 2024Archived →: copy of Wired, 20 Mar 2024
  9. Headline1 Dec 2025

    “An AI Startup Looks Toward the Post-Transformer Era”

    The Wall Street Journal wsj.com →: The Wall Street Journal, 1 Dec 2025Archived →: copy of The Wall Street Journal, 1 Dec 2025
  10. Headline18 Sep 2026

    “Beyond LLMs: A Post-Transformer World Emerges”

    Communications of the ACM cacm.acm.org →: Communications of the ACM, 18 Sep 2026
§05

Not to be confused with

  • Large language model

    A large language model is a trained system that predicts text. The transformer is the network design most such models use. Transformers also handle images and speech, and Mamba-based language models drop attention entirely.

    See large language model →

  • Recurrent neural network

    A recurrent network reads a sequence one step at a time, carrying a running summary forward. A transformer reads all positions at once and links any two directly through attention, which suits parallel hardware far better.

  • Attention mechanism

    Attention is one component: a way for a network to weigh parts of its input. The best translation systems used it alongside recurrent networks before 2017. The transformer is an architecture built on attention alone.

  • Electrical transformer

    A device that raises or lowers the voltage of an electric supply, a sense dating from 1882. The AI transformer is a software design that shares only the name.

§06

Questions

  • What is a transformer in AI?

    A transformer is a neural network design that reads a whole sequence at once. For each word or other unit, a step called attention scores how much every other unit matters, and uses the scores to update its meaning. Eight researchers, most of them at Google, introduced it for machine translation in 2017. Most large language models, including OpenAI’s GPT models, are transformers.

  • What does the T in GPT stand for?

    It stands for transformer: GPT is short for generative pre-trained transformer. OpenAI’s 2019 announcement of GPT-2 called that model transformer-based. Google’s BERT, which Google began using to rank search results in 2019, also carries the word in its full name: Bidirectional Encoder Representations from Transformers.

  • Who invented the transformer?

    Eight researchers introduced it in the 2017 paper “Attention Is All You Need”. They were Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser and Illia Polosukhin. Most of them worked at Google. Vaswani told NVIDIA that Uszkoreit came up with the name; Uszkoreit told Wired that he and two colleagues chose it from the start. The attention idea at its core was older and already used in translation systems.

  • What is the difference between a transformer and a large language model?

    A transformer is a design for a neural network. A large language model is a system trained on vast amounts of text to predict what comes next, and most are built as transformers. The design also works beyond language: the Vision Transformer reads images as patches, and OpenAI’s Whisper uses it to transcribe speech. Language models built on Mamba drop attention entirely.

  • Is the transformer being replaced?

    Not outright. Attention’s cost grows with the square of the input length, so researchers have built alternatives such as Mamba, which drops attention. By late 2025 the business press was writing about a post-transformer era. Some newer models are hybrids instead: NVIDIA’s Nemotron 3 Ultra, released in 2026, pairs Mamba layers for long inputs with transformer layers for precise recall.

§07

Watch

§09

Cite this entry

“Transformer.” SUPERR_dictionary, Superrduperr, 2 Oct. 2026, superrduperr.com/dictionary/transformer.

First published 24 Sep 2026 · Last reviewed 2 Oct 2026 · Report an error →