Start a project
Foundations Reviewed 24 Sep 2026 · 19 sources

pretraining

/ˌpriːˈtreɪ.nɪŋ/ noun Verb pretrain · adjective pretrained · also spelled pre-training
Definition

The first and usually costliest stage of training an AI model, in which it learns general patterns from a huge, mostly unlabeled dataset before being adapted to tasks.

Examples
  • OpenAI’s first GPT, trained for a month on over 7,000 unpublished books
  • BERT learning to fill in hidden words across Wikipedia
  • Meta’s Llama 3, trained on over 15 trillion tokens of public text
A tall stack of unlabeled pages pours through a funnel into a model. The model writes a guess for the next word on a card. A thin labeled folder waits beside it.
Inkling · A model learning from a vast pile of unlabeled textFig. P1
Current meaning
2018
OpenAI’s GPT: a language model trained on unlabeled text, then fine-tuned.
Also called
Pre-training
The hyphenated spelling, used in the GPT and BERT paper titles.
§01

In plain words

Pretraining is the first stage of training an AI model, in which it learns general patterns from vast unlabeled data before fine-tuning adapts it.

A large language model begins as billions of random numbers, its weights. In pretraining, it works through an enormous collection of text, guessing each next word. Every wrong guess nudges the weights. Over trillions of tokens, the small pieces of text a model reads, patterns of language and knowledge build up in them.

No one has to label this text, because the next word is its own answer key. OpenAI’s 2018 paper named the approach generative pre-training, the P in GPT. It pretrained a transformer on a large amount of unlabeled text, then fine-tuned it on much smaller labeled sets.1

Pretraining is also where most of the compute goes. DeepSeek reported that pretraining its V3 model used 2,664,000 of the 2,788,000 GPU hours in its training run.2 The later steps, fine-tuning and reinforcement learning from human feedback, cost far less. OpenAI put the compute for teaching GPT-3 to follow instructions at a fraction of what training GPT-3 took.3

Pretraining is different from inference, which is using the finished model and leaves its weights unchanged. It also sets the model’s knowledge cutoff, since the model learns little about events after its text was gathered. Ilya Sutskever, a co-author of the 2018 paper, said in 2025 that pretraining will eventually run out of data, because the data is finite.

§02

How it works

Pretraining starts with data. Builders collect text, such as web pages and books, and split it into tokens. Meta’s Llama 3, for example, was pretrained on more than 15 trillion tokens from public sources.

The training itself is one guessing game repeated at huge scale. The model reads a stretch of text and predicts the next token. The guess is compared with the real token, and every weight shifts slightly toward a better guess. Models in the GPT line learn this way. Google’s BERT instead hides some words and predicts them from the words on both sides.4 DeepSeek-V3 went through 14.8 trillion tokens this way, on 2,048 GPUs, in under two months.2

Scale follows measurable rules. In 2020 Kaplan and colleagues found that a model’s prediction error falls steadily as model size, data and compute grow.5 In 2022 Hoffmann and colleagues found that most large models had too little data for their size. Model size and data, they concluded, should grow in step.6

The result is a base model, a text predictor not yet trained to follow instructions. Later post-training, such as supervised fine-tuning and RLHF, adds that.

Fig. P2 · Where the training compute wentIn thousands of GPU hours
  1. Pretraining 2,664
  2. Context extension 119
  3. Post-training 5
DeepSeek’s reported figures for training DeepSeek-V3, in thousands of H800 GPU hours. They exclude earlier research and experiments.
§03

Origin and first known use

Formed from pre-, “before”, and train. Merriam-Webster dates the verb pretrain, “to train in advance”, to 1943 and lists both pretraining and pre-training. Neural network researchers were writing about pre-trained networks by 1993. In 2006 Hinton and Salakhutdinov put “pretraining” in quotation marks for training a deep network one layer at a time before fine-tuning it. Vision researchers then pretrained networks on one labeled task before fine-tuning them for another. From 2015 the word moved to language models trained on unlabeled text, and OpenAI’s 2018 paper on generative pre-training gave GPT its P.

  1. 1943 Merriam-Webster dates the first known use of pretrain, “to train in advance”, to 1943. It lists both pretraining and pre-training as forms.
    Merriam-Webster, “pretrain” merriam-webster.com →: source for 1943
  2. 1993 McRae and Hetherington pre-train a neural network on a random sample of data before a sequential learning task and report that catastrophic interference, new learning erasing old, disappears.
    McRae and Hetherington, “Catastrophic Interference is Eliminated in Pre-Trained Networks”, Proceedings of the 15th Annual Conference of the Cognitive Science Society, 1993, pages 723 to 728 wikipedia.org →: source for 1993
  3. 2006 Hinton and Salakhutdinov train a deep network one layer at a time, a step they call pretraining, then fine-tune the whole network with backpropagation.
    Hinton and Salakhutdinov, “Reducing the Dimensionality of Data with Neural Networks”, Science 313, 28 July 2006 toronto.edu →: source for 2006
  4. 2013 Girshick and colleagues show that when labeled data is scarce, supervised pre-training on another task, followed by fine-tuning, significantly improves object detection in images.
    Girshick, Donahue, Darrell and Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation” arXiv 1311.2524 →: source for 2013
  5. 2015 Dai and Le train a recurrent network to predict what comes next in unlabeled text, as a “pretraining” step before supervised learning.
    Dai and Le, “Semi-supervised Sequence Learning”, NIPS 2015 arXiv 1511.01432 →: source for 2015
  6. 2018 OpenAI pre-trains a transformer language model on over 7,000 unpublished books, then fine-tunes it for each task. “Generative pre-training” gives GPT its name.
    Radford, Narasimhan, Salimans and Sutskever, “Improving Language Understanding by Generative Pre-Training”, OpenAI, June 2018 openai.com →: source for 2018
  7. 2018 Google researchers pre-train BERT on unlabeled text using the words on both sides of each position, then fine-tune it to set records on eleven language tasks.
    Devlin, Chang, Lee and Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” arXiv 1810.04805 →: source for 2018
  8. 2025 Ilya Sutskever calls 2020 to 2025 the age of scaling and says pretraining will eventually run out of data, because the data is finite.
    Dwarkesh Patel, “Ilya Sutskever — We’re moving from the age of scaling to the age of research”, Dwarkesh Podcast, 25 November 2025 dwarkesh.com →: source for 2025
§04

In the wild

9 sightings · 2006–2025 · oldest first
  1. Paper28 Jul 2006

    “We introduce this “pretraining” procedure for binary data, generalize it to real-valued data, and show that it works well for a variety of data sets.”

    Hinton and Salakhutdinov, Science 2006 toronto.edu →: Hinton and Salakhutdinov, Science 2006, 28 Jul 2006Archived →: copy of Hinton and Salakhutdinov, Science 2006, 28 Jul 2006
  2. Blog11 Jun 2018

    “Our approach is a combination of two existing ideas: transformers and unsupervised pre-training.”

    Alec Radford, OpenAI blog openai.com →: Alec Radford, OpenAI blog, 11 Jun 2018Archived →: copy of Alec Radford, OpenAI blog, 11 Jun 2018
  3. Blog2 Nov 2018

    “…techniques for training general purpose language representation models using the enormous amount of unannotated text on the web (known as pre-training).”

    Jacob Devlin and Ming-Wei Chang, Google AI blog research.google →: Jacob Devlin and Ming-Wei Chang, Google AI blog, 2 Nov 2018Archived →: copy of Jacob Devlin and Ming-Wei Chang, Google AI blog, 2 Nov 2018
  4. Paper28 May 2020

    “…substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.”

    Brown et al., arXiv preprint arXiv 2005.14165 →: Brown et al., arXiv preprint, 28 May 2020Archived →: copy of Brown et al., arXiv preprint, 28 May 2020
  5. Paper15 Mar 2023

    “GPT-4 is a Transformer-based model pre-trained to predict the next token in a document.”

    OpenAI, GPT-4 Technical Report arXiv 2303.08774 →: OpenAI, GPT-4 Technical Report, 15 Mar 2023
  6. Blog18 Apr 2024

    “Llama 3 is pretrained on over 15T tokens that were all collected from publicly available sources.”

    Meta, Llama 3 announcement ai.meta.com →: Meta, Llama 3 announcement, 18 Apr 2024Archived →: copy of Meta, Llama 3 announcement, 18 Apr 2024
  7. Paper27 Dec 2024

    “We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages…”

    DeepSeek-AI, DeepSeek-V3 Technical Report arXiv 2412.19437 →: DeepSeek-AI, DeepSeek-V3 Technical Report, 27 Dec 2024Archived →: copy of DeepSeek-AI, DeepSeek-V3 Technical Report, 27 Dec 2024
  8. Blog27 Feb 2025

    “GPT‑4.5 is a step forward in scaling up pre-training and post-training.”

    OpenAI, GPT-4.5 announcement openai.com →: OpenAI, GPT-4.5 announcement, 27 Feb 2025Archived →: copy of OpenAI, GPT-4.5 announcement, 27 Feb 2025
  9. Transcript25 Nov 2025

    “At some point though, pre-training will run out of data. The data is very clearly finite.”

    Ilya Sutskever, on the Dwarkesh Podcast dwarkesh.com →: Ilya Sutskever, on the Dwarkesh Podcast, 25 Nov 2025Archived →: copy of Ilya Sutskever, on the Dwarkesh Podcast, 25 Nov 2025
§05

Not to be confused with

  • Fine-tuning

    Later, smaller training that adapts a pretrained model to a task, subject or style, often with thousands of labeled examples. Pretraining comes first, costs far more and builds general ability from a huge unlabeled collection.

    See fine-tuning →

  • Inference

    Running a finished model to produce an answer. Pretraining changes the model’s weights; inference leaves them unchanged. A model is pretrained once, over weeks or months, and then answers many requests.

    See inference →

  • Post-training, such as RLHF

    Training that follows pretraining and teaches a model to follow instructions, using human feedback. It is far smaller: OpenAI’s InstructGPT needed 60 petaflops/s-days of compute for it, against 3,640 to pretrain GPT-3.

    See reinforcement learning from human feedback →
§06

Questions

  • What is pretraining in AI?

    Pretraining is the first stage of building a model such as a large language model. The model starts with random weights and learns by predicting the next word, or a hidden word, across a huge collection of unlabeled text. The result is a base model with broad knowledge of language. Later stages, such as fine-tuning and RLHF, adapt it to tasks and to following instructions.

  • What is the difference between pretraining and fine-tuning?

    Pretraining comes first. It trains a model from random weights on a huge, mostly unlabeled collection of text, and it uses most of the compute. Fine-tuning comes later. It continues training the pretrained model on a much smaller set of examples, often labeled, to suit a task or a style. OpenAI’s 2018 GPT paper paired the two: generative pre-training, then fine-tuning for each task.

  • How much does it cost to pretrain a large language model?

    DeepSeek reported that pretraining DeepSeek-V3 on 14.8 trillion tokens took under two months on 2,048 GPUs. That came to 2,664,000 GPU hours, or $5.328 million at an assumed rental price of $2 an hour, excluding earlier research. Post-training the same model took 5,000 GPU hours. OpenAI’s first GPT, in 2018, needed a month on 8 GPUs.

  • What does the P in GPT stand for?

    Pre-trained. GPT stands for generative pre-trained transformer. The name traces to OpenAI’s 2018 paper “Improving Language Understanding by Generative Pre-Training”, by Radford, Narasimhan, Salimans and Sutskever. It pre-trained a transformer language model on over 7,000 unpublished books, then fine-tuned it on much smaller labeled datasets for each task.

  • Will pretraining run out of data?

    Ilya Sutskever, a co-author of OpenAI’s 2018 GPT paper, expects so. In a November 2025 interview he called 2020 to 2025 the age of scaling. He said pretraining will eventually run out of data, because the data is finite. Scaling research had shown the need for data: in 2022 Hoffmann and colleagues found that training data should grow in step with model size.

§07

Watch

§09

Cite this entry

“Pretraining.” SUPERR_dictionary, Superrduperr, 24 Sept. 2026, superrduperr.com/dictionary/pretraining.

First published 24 Sep 2026 · Last reviewed 24 Sep 2026 · Report an error →