Start a project
Practice Reviewed 24 Sep 2026 · 19 sources

benchmark

/ˈbɛntʃˌmɑrk/ noun Plural benchmarks · verb benchmark · noun benchmarking
Definition

A standard test for AI models, made of fixed tasks and an agreed scoring method, so that different models can be compared on equal terms.

Examples
  • MMLU, multiple-choice questions on 57 subjects from math to law
  • SWE-bench, 2,294 real GitHub issues for a model to resolve
  • ImageNet’s yearly challenge to recognize objects in millions of photos
  • Humanity’s Last Exam, 2,500 questions written by subject experts
Three boxes labeled as models each receive the same stack of numbered test cards. Beside them, one tall surveyor’s staff carries three notches at different heights, one for each model’s score.
Inkling · The same test for every model, scored on one scaleFig. B1
Earliest AI use
1990
Earliest found: NIST’s DARPA speech recognition benchmark test.
Current meaning
2018
GLUE tests one general-purpose model across many language tasks.
§01

In plain words

In AI, a benchmark is a standard test, the same tasks scored the same way for every model, used to compare models and track progress over time.1

A benchmark gives every model the same exam. The tasks and the scoring rule are set in advance, so a score means the same thing whoever reports it. MMLU, for example, asks multiple-choice questions on 57 subjects, from elementary mathematics to law. When it appeared in 2020, most models scored close to chance.2 SWE-bench gives a model real issues reported in popular Python projects on GitHub and checks whether it can resolve them.

Benchmark scores are how model makers announce progress. When Google introduced Gemini in December 2023, it said its largest version beat the previous best results on 30 of 32 widely used academic benchmarks.

A benchmark stops being useful once the best models score near its maximum, a state known as saturation. Within about a year of the release of GLUE, a set of language understanding tasks, models had passed the level of non-expert humans on it. Its successor, SuperGLUE, used harder tasks. After models passed 90% on MMLU, researchers built Humanity’s Last Exam from questions written by subject experts.3

Scores can also mislead. Test questions can leak into the data a model learns from, and developers can tune a model to one test. Benchmarks are often confused with evals, a wider practice that includes private tests a team writes for its own product. A benchmark is public and fixed, so many developers can run the same one.

§02

How it works

A benchmark fixes three things before any model takes it: the test items, the correct answers and a scoring rule. Items are often multiple-choice or short-answer, so a program can do the grading. Humanity’s Last Exam was built this way.3 Each model answers every item under the same conditions, and its score is usually the share it gets right. Results often go on a leaderboard, a ranked table of models.

Several things can distort the number. Test questions published online can end up in a model’s pretraining data, so a high score may partly reflect memory. This is called contamination. When researchers wrote fresh grade-school math problems, several model families scored lower on them, a sign of overfitting.4 Developers can also test many private versions of a model on a public leaderboard and publish only the best. A 2025 study counted 27 private variants that Meta tested on Chatbot Arena before releasing Llama 4.5 Meta’s launch post cited the LMArena score of an experimental chat version, not the model it released.6

§03

Origin and first known use

From bench plus mark. In surveying, a bench mark was a fixed point of reference for measuring height, where an angle-iron support, the bench, held the leveling staff. Merriam-Webster dates the surveying sense to 1813 and the Online Etymology Dictionary to 1838. The figurative sense, a standard to judge others by, follows in 1884. Computing adopted the word for standard tests used to compare machines. By 1990 speech recognition researchers used it for shared test sets on which rival systems were scored. GLUE, in 2018, applied it to general-purpose language models tested across many tasks.

  1. 1838 Recorded as the surveyors’ term for a fixed point of reference, where an angle-iron bench held the leveling staff. The figurative sense follows in 1884.
    Online Etymology Dictionary, “benchmark” Etymonline →: source for 1838
  2. 1988 Workstation vendors found SPEC, the Standard Performance Evaluation Corporation, to create realistic, standardized tests for comparing computer performance.
    Standard Performance Evaluation Corporation, overview page on spec.org spec.org →: source for 1988
  3. 1990 NIST reports the June 1990 DARPA Resource Management Benchmark Test, which scored speech recognizers from several labs on shared test sets. Earliest AI use found.
    Pallett, Fiscus and Garofolo, “DARPA Resource Management Benchmark Test Results June 1990”, Speech and Natural Language workshop, Hidden Valley, Pennsylvania, June 1990 aclanthology.org →: source for 1990
  4. 2010 The ImageNet Large Scale Visual Recognition Challenge is first held. Run every year, it had drawn entrants from more than 50 institutions by 2014.
    Russakovsky et al., “ImageNet Large Scale Visual Recognition Challenge”, arXiv 2014 arXiv 1409.0575 →: source for 2010
  5. 2018 Wang and colleagues introduce GLUE, which tests one model across a diverse range of language understanding tasks rather than a single task. The current sense starts here.
    Wang et al., “GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding”, arXiv 2018 arXiv 1804.07461 →: source for 2018
  6. 2019 SuperGLUE follows GLUE after models pass non-expert humans on it within about a year. It adds harder tasks, a software toolkit and a public leaderboard.
    Wang et al., “SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems”, NeurIPS 2019 arXiv 1905.00537 →: source for 2019
  7. 2023 SWE-bench asks models to resolve 2,294 real GitHub issues from 12 popular Python projects. The best model at launch, Claude 2, solves 1.96% of them.
    Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?”, arXiv 2023 arXiv 2310.06770 →: source for 2023
  8. 2025 The Center for AI Safety and Scale AI release Humanity’s Last Exam, written by subject experts, after models pass 90% on benchmarks such as MMLU.
    Phan et al., “Humanity’s Last Exam”, arXiv 2501.14249, January 2025; published in Nature in 2026 lastexam.ai →: source for 2025
§04

In the wild

8 sightings · 2014–2026 · oldest first
  1. Paper1 Sep 2014

    “The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images.”

    Russakovsky et al., arXiv preprint arXiv 1409.0575 →: Russakovsky et al., arXiv preprint, 1 Sep 2014Archived →: copy of Russakovsky et al., arXiv preprint, 1 Sep 2014
  2. Paper2 May 2019

    “…performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research.”

    Wang et al., NeurIPS 2019 arXiv 1905.00537 →: Wang et al., NeurIPS 2019, 2 May 2019Archived →: copy of Wang et al., NeurIPS 2019, 2 May 2019
  3. Blog7 Apr 2021

    “Researchers are often incentivized to benchmark their methods on a handful of popular datasets that have been well established in the field…”

    Joaquin Vanschoren and Serena Yeung, NeurIPS blog blog.neurips.cc →: Joaquin Vanschoren and Serena Yeung, NeurIPS blog, 7 Apr 2021Archived →: copy of Joaquin Vanschoren and Serena Yeung, NeurIPS blog, 7 Apr 2021
  4. Blog6 Dec 2023

    “Gemini Ultra’s performance exceeds current state-of-the-art results on 30 of the 32 widely-used academic benchmarks used in large language model (LLM) research and development.”

    Sundar Pichai and Demis Hassabis, Google blog blog.google →: Sundar Pichai and Demis Hassabis, Google blog, 6 Dec 2023Archived →: copy of Sundar Pichai and Demis Hassabis, Google blog, 6 Dec 2023
  5. Docs29 Jan 2024

    “LLM benchmarks are standardized frameworks for assessing the performance of large language models (LLMs).”

    Rina Diane Caballar and Cole Stryker, IBM Think explainer ibm.com →: Rina Diane Caballar and Cole Stryker, IBM Think explainer, 29 Jan 2024Archived →: copy of Rina Diane Caballar and Cole Stryker, IBM Think explainer, 29 Jan 2024
  6. Headline6 Apr 2025

    “Meta’s benchmarks for its new AI models are a bit misleading”

    TechCrunch techcrunch.com →: TechCrunch, 6 Apr 2025Archived →: copy of TechCrunch, 6 Apr 2025
  7. Paper29 Apr 2025

    “As benchmarks play an increasingly central role, they also grow more susceptible to distortion.”

    Singh et al., arXiv preprint arXiv 2504.20879 →: Singh et al., arXiv preprint, 29 Apr 2025Archived →: copy of Singh et al., arXiv preprint, 29 Apr 2025
  8. Paper28 Jan 2026

    “…benchmarks are not keeping pace in difficulty: LLMs now achieve more than 90% accuracy on popular benchmarks such as Measuring Massive Multitask Language Understanding…”

    Center for AI Safety, Scale AI and HLE Contributors Consortium, Nature nature.com →: Center for AI Safety, Scale AI and HLE Contributors Consortium, Nature, 28 Jan 2026Archived →: copy of Center for AI Safety, Scale AI and HLE Contributors Consortium, Nature, 28 Jan 2026
§05

Not to be confused with

  • Evals

    A benchmark is fixed and public, so many developers can run it and compare results. Evals are tests chosen for one model, product or risk, often private and rerun after each change. A benchmark can be one of them.

    See evals →

  • Leaderboard

    A leaderboard is a ranked table of results, not the test itself. Many rank scores on one or more benchmarks. LMArena instead ranks models by which of two answers people prefer.

  • Speed benchmark

    In computing, a benchmark times how fast hardware or software runs a standard workload, as SPEC’s tests have since 1988. An AI benchmark of the kind defined here scores what a model gets right.

§06

Questions

  • What is a benchmark in AI?

    A benchmark is a standard test for AI models. It fixes a set of tasks, the correct answers and a scoring rule, so every model is measured the same way. MMLU, for example, asks multiple-choice questions on 57 subjects, and SWE-bench asks models to resolve real GitHub issues. Model makers report benchmark scores to show how their models compare with others.

  • What is the difference between a benchmark and evals?

    A benchmark is a fixed, public test that many developers run, so their models can be compared on equal terms. Evals are the wider practice of testing a model or product. They include private tests a team writes for its own tasks or for a specific risk, rerun after each change. A public benchmark can be one of a team’s evals.

  • What is benchmark saturation?

    A benchmark is saturated when the best models score near its maximum, so it can no longer show which is better. It has happened repeatedly. Within about a year of GLUE’s release in 2018, models had passed non-expert humans on it, and the harder SuperGLUE followed. After models passed 90% on MMLU, researchers built Humanity’s Last Exam from questions written by subject experts.

  • What is benchmark contamination?

    Contamination happens when test questions, or text very like them, leak into the data a model is trained on. A high score may then partly reflect memory rather than skill. In 2024 researchers wrote new grade-school math problems and tested leading models on them. Accuracy fell, and several model families showed signs of systematic overfitting.

  • Can AI benchmark results be gamed?

    Yes, in several ways. Test questions can leak into training data, and a model can be tuned to one test. A developer can also try many private versions on a leaderboard and publish the best. A 2025 study counted 27 private variants that Meta tested on Chatbot Arena before releasing Llama 4. Meta’s launch post cited the LMArena score of an experimental chat version.

§07

Watch

§09

Cite this entry

“Benchmark.” SUPERR_dictionary, Superrduperr, 24 Sept. 2026, superrduperr.com/dictionary/benchmark.

First published 24 Sep 2026 · Last reviewed 24 Sep 2026 · Report an error →