Start a project
Foundations Reviewed 24 Sep 2026 · 17 sources

inference

/ˈɪn.fər.əns/ noun Plural inferences · verb infer · adjective inference-time
Definition

The process of running a trained AI model on new input to produce an output, such as a label or a written reply, without changing the model’s weights.

Examples
  • A chatbot writing its reply, one token at a time
  • Speech recognition turning a spoken search into text
  • A phone app labeling the objects in a photo
  • A reasoning model working through a math problem before answering
Two scenes side by side. On the left, a model studies a tall stack of books while the dials on its side turn. On the right, the same model, its dials locked, takes question cards from a queue and hands back answer cards.
Inkling · The finished model at work, its learned settings lockedFig. I1
Earliest AI use
1956
Ray Solomonoff’s “inductive inference machine”, learning from examples.
Current meaning
2014
Running a trained network on new data, as in the GoogLeNet paper.
Opposite
Training
The stage in which a model learns from data and its weights change.
Also called
Test time
As in test-time compute, the extra work a model does at inference.
§01

In plain words

In AI, inference is the stage in which a trained model is put to use, turning new inputs into outputs such as answers or predictions.

Building an AI model and using it are separate stages. In training, a model learns from data, and its weights, the numbers that store what it has learned, keep changing. For large models, training starts with pretraining. Inference comes after. The finished model receives a new input, such as a question or a photo, and computes an output. Its weights stay fixed. Every reply from a chatbot is produced this way.

Training is a large cost paid once per model. Inference is paid on every request, so its total grows with use. Google designed the first generation of its Tensor Processing Unit, a machine learning chip, for inference. Google gave an example of the scale. If people used voice search for three minutes a day, running speech recognition on its existing processors would have meant doubling its data centers.1

For a large language model, providers charge for inference by the token, the unit of text a model reads and writes. Prices for a given level of ability have fallen quickly. In Epoch AI’s data, they dropped by between 9 and 900 times a year, depending on the task.2 Reasoning models spend extra computation at inference, writing long chains of working before they reply. Researchers call this test-time compute. Each request also uses energy. NIST describes running generative models as resource-intensive, and in 2025 Google measured the median text prompt to its Gemini assistant at 0.24 watt-hours.

The word is older than AI, and its other senses can confuse. In logic, an inference is a conclusion drawn from premises. In statistics, it is a generalization from a sample to a whole population. Google’s machine learning glossary notes that the statistical meaning differs from the one used for models.3

§02

How it works

Inference begins when an input arrives, such as an image or a prompt. A language model first splits a prompt into tokens and turns them into numbers. The model then runs a forward pass. The numbers flow through its layers and combine with the weights at each one. Out comes a result, such as a label or a score for every possible next token. Training runs the same forward pass, then compares the result with a known answer and adjusts the weights. Inference stops after the forward pass.4

A large language model answers in two phases. It first processes the whole prompt, a step limited mainly by computing power. It then generates the reply one token per pass, adding each new token to the input, a step limited mainly by memory.5 The model also stores intermediate results for every earlier token, in what is called the key-value cache.

Serving systems run many users’ requests together in batches. Bigger batches produce more tokens per second, but every request’s cache takes memory, which caps the batch size.6 Smaller batches return each answer sooner.4 Chips built for inference can compute with fewer bits: Google’s first TPU used 8-bit arithmetic. Storing weights with fewer bits is called quantization.

Fig. I2 · One prompt, two phasesIllustrative, not measured

What is the capital of Australia?

Split
The prompt becomes 8 tokens, each turned into numbers
Process the prompt
All 8 tokens pass through the model together
Generate
One new token per pass, each added to the input: The, capital, is, Canberra
Stop
An end token closes the reply, and the tokens become text
A language model answering a short prompt. The prompt, the token count and the reply are invented to show the steps.
§03

Origin and first known use

From Medieval Latin inferentia, based on Latin inferre, “bring into; conclude, deduce”, inference entered English in the 1590s for the act of drawing a conclusion. AI took the word up early. In 1956 Ray Solomonoff described an “inductive inference machine” that learned methods from worked examples. There, inference meant drawing general rules from examples. Expert systems of the 1970s put hand-written rules to use with an “inference engine”. From the mid-2010s, deep learning researchers used inference for running a trained network on new data, as distinct from training it. Chips and benchmarks built for that stage followed.

  1. 1590s Enters English as the action of inferring, from Medieval Latin inferentia, from Latin inferre, “bring into; conclude, deduce”.
    Online Etymology Dictionary, “inference” Etymonline →: source for 1590s
  2. 1956 Ray Solomonoff writes “An Inductive Inference Machine” during the Dartmouth summer workshop on AI. It describes a machine designed to learn to work math problems from worked examples.
    Ray Solomonoff, “An Inductive Inference Machine”, dated 14 August 1956, author’s annotated version raysolomonoff.com →: source for 1956
  3. 1977 Edward Feigenbaum describes expert systems that capture an expert’s craft in rules, applied by an inference engine. He calls PUFF and MYCIN backward-chaining inference systems.
    Edward A. Feigenbaum, “The Art of Artificial Intelligence: Themes and Case Studies of Knowledge Engineering”, IJCAI 1977 ijcai.org →: source for 1977
  4. 2014 Szegedy and colleagues design GoogLeNet, an image recognition network, within a budget of 1.5 billion multiply-adds at inference time. The aim is real-world use at reasonable cost.
    Szegedy et al., “Going Deeper with Convolutions”, arXiv, 17 September 2014 arXiv 1409.4842 →: source for 2014
  5. 2017 Google’s Tensor Processing Unit, in its data centers since 2015, runs neural network inference with 8-bit arithmetic, 15 to 30 times faster than contemporary chips.
    Jouppi et al., “In-Datacenter Performance Analysis of a Tensor Processing Unit”, ISCA 2017 arXiv 1704.04760 →: source for 2017
  6. 2019 Reddi and 49 co-authors publish MLPerf Inference, a benchmark for systems that run trained models. Its first round drew more than 600 measurements from 14 organizations.
    Reddi et al., “MLPerf Inference Benchmark”, arXiv, 6 November 2019 arXiv 1911.02549 →: source for 2019
  7. 2024 Snell and colleagues study test-time compute, extra computation at inference. On problems it can partly solve, a smaller model using it can beat one 14 times larger.
    Snell, Lee, Xu and Kumar, “Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters” arXiv 2408.03314 →: source for 2024
  8. 2025 Google measures its Gemini assistant in production. The median text prompt uses 0.24 watt-hours, less than nine seconds of television, after a 33-fold cut in a year.
    Elsworth et al., “Measuring the environmental impact of delivering AI at Google Scale” arXiv 2508.15734 →: source for 2025
§04

In the wild

8 sightings · 2014–2025 · oldest first
  1. Paper17 Sep 2014

    “The network was designed with computational efficiency and practicality in mind, so that inference can be run on individual devices…”

    Szegedy et al., arXiv preprint arXiv 1409.4842 →: Szegedy et al., arXiv preprint, 17 Sep 2014Archived →: copy of Szegedy et al., arXiv preprint, 17 Sep 2014
  2. Blog11 Nov 2015

    “Then, the network is deployed to run inference, using its previously trained parameters to classify, recognize and process unknown inputs.”

    Michael Andersch, NVIDIA Developer blog developer.nvidia.com →: Michael Andersch, NVIDIA Developer blog, 11 Nov 2015Archived →: copy of Michael Andersch, NVIDIA Developer blog, 11 Nov 2015
  3. Blog5 Apr 2017

    “This first generation of TPUs targeted inference (the use of an already trained model, as opposed to the training phase of a model…)”

    Norm Jouppi, Google Cloud blog cloud.google.com →: Norm Jouppi, Google Cloud blog, 5 Apr 2017Archived →: copy of Norm Jouppi, Google Cloud blog, 5 Apr 2017
  4. Paper6 Nov 2019

    “Driven by ML applications, the number of different ML inference systems has exploded. Over 100 organizations are building ML inference chips…”

    Reddi et al., arXiv preprint arXiv 1911.02549 →: Reddi et al., arXiv preprint, 6 Nov 2019Archived →: copy of Reddi et al., arXiv preprint, 6 Nov 2019
  5. Docs26 Jul 2024

    “Training, maintaining, and operating (running inference on) GAI systems are resource-intensive activities, with potentially large energy and environmental footprints.”

    NIST, Generative AI Profile (NIST AI 600-1) nist.gov →: NIST, Generative AI Profile (NIST AI 600-1), 26 Jul 2024Archived →: copy of NIST, Generative AI Profile (NIST AI 600-1), 26 Jul 2024
  6. Paper6 Aug 2024

    “In this paper, we study the scaling of inference-time computation in LLMs…”

    Snell et al., arXiv preprint arXiv 2408.03314 →: Snell et al., arXiv preprint, 6 Aug 2024Archived →: copy of Snell et al., arXiv preprint, 6 Aug 2024
  7. Blog12 Nov 2024

    “The cost of LLM inference has dropped by a factor of 1,000 in 3 years.”

    Guido Appenzeller, a16z blog a16z.com →: Guido Appenzeller, a16z blog, 12 Nov 2024Archived →: copy of Guido Appenzeller, a16z blog, 12 Nov 2024
  8. Paper21 Aug 2025

    “…a comprehensive methodology for measuring the energy usage, carbon emissions, and water consumption of AI inference workloads in a large-scale, AI production environment.”

    Elsworth et al., Google, arXiv preprint arXiv 2508.15734 →: Elsworth et al., Google, arXiv preprint, 21 Aug 2025Archived →: copy of Elsworth et al., Google, arXiv preprint, 21 Aug 2025
§05

Not to be confused with

  • Training or pretraining

    Training, which begins with pretraining, sets a model’s weights by learning from data, often over weeks or months. Inference runs the finished model on each new request and leaves those weights unchanged.

    See pretraining →

  • Statistical inference

    Drawing general conclusions about a population from sample data, with a stated degree of certainty. Machine learning inference is narrower: running an already trained model on a new input to get its output.

  • Reasoning

    In logic, an inference is a step from premises to a conclusion. Machine learning inference is a computation, whatever the task. Reasoning models write out step-by-step working, and that text is itself produced during inference.

    See reasoning model →

  • Inference engine

    In the expert systems of the 1970s, the program that applied hand-written if-then rules to the facts of a case. Machine learning inference runs a model whose weights were learned from data instead.

§06

Questions

  • What is inference in AI?

    Inference is the stage in which a trained AI model is used. The model receives a new input, such as a question or a photo, and computes an output while its weights stay fixed. It follows training, in which those weights are set. Every chatbot reply is produced by inference, and large language model providers charge for it by the token.

  • What is the difference between training and inference?

    Training builds the model. Examples pass through it, each result is compared with a known answer, and the weights are adjusted. Inference uses the finished model. The same forward calculation runs, but nothing is adjusted. Training is a large cost paid once per model, while inference is paid on every request. Google designed its first Tensor Processing Unit chips for inference.

  • How much does AI inference cost?

    Providers of large language models charge for inference by the token, the unit of text a model reads and writes. Prices for a given level of ability have fallen quickly. Epoch AI found declines of between 9 and 900 times a year, depending on the task. An a16z analysis estimated that the cost of LLM inference dropped by a factor of 1,000 in three years.

  • What is test-time compute?

    Test-time compute is extra computation a model spends during inference to produce a better answer, for example by writing out its working before replying. Reasoning models depend on it. In 2024, Charlie Snell and colleagues found that test-time compute, well used, could let a smaller model outperform one 14 times larger. That held on problems the smaller model could already partly solve.

  • How much energy does AI inference use?

    It varies with the model and the task. In 2025 Google measured its Gemini assistant in production. The median text prompt used 0.24 watt-hours, less energy than watching nine seconds of television. That was 33 times less than a year earlier. NIST describes running generative AI systems as resource-intensive, with potentially large energy footprints.

§07

Watch

§09

Cite this entry

“Inference.” SUPERR_dictionary, Superrduperr, 24 Sept. 2026, superrduperr.com/dictionary/inference.

First published 24 Sep 2026 · Last reviewed 24 Sep 2026 · Report an error →