Start a project
Foundations Reviewed 24 Sep 2026 · 17 sources

token

/ˈtoʊ.kən/ noun Plural tokens · verb tokenize · nouns tokenizer, tokenization
Definition

A chunk of text, often a word or part of a word, that a language model reads and writes, and by which its use is measured and priced.

Examples
  • A common word such as “great”, together with the space before it
  • A piece of a rare word, such as “ik” in “tiktoken”
  • A punctuation mark such as an exclamation point
A paper strip printed with the words tiktoken is great, followed by an exclamation point, is cut into six uneven pieces: t, ik, token, is, great and the exclamation point. Each piece drops into a numbered slot on a model.
Inkling · One sentence cut into the six pieces a model readsFig. T1
Current meaning
2015
Translation research: rare words split into pieces, counted as tokens.
Also called
Wordpiece
Google’s term, from its WordPiece model for voice search, 2012.
§01

In plain words

In AI, a token is the basic unit of text a language model reads and writes, usually a word or part of one; model limits and prices are counted in tokens.

Large language models do not read text letter by letter or word by word. A program called a tokenizer first cuts the text into tokens, pieces drawn from its vocabulary. A frequent word is usually a single token, often together with the space before it. A rare word or a name is split into several pieces. Merriam-Webster lists this computing sense of the word.1

Tokens are how the use of a model is measured. For English, OpenAI estimates a token at about four characters, or three quarters of a word, so 100 tokens make about 75 words.2 Google puts 100 tokens at 60 to 80 English words.3 The context window, the most a model can handle at once, is also set in tokens.

Prices follow the same unit. In 2026, Anthropic listed one model at $4 per million tokens read and $20 per million written. It also says its newer tokenizer produces about 30% more tokens for the same text.4 Reasoning models spend extra tokens on steps the user never sees, and OpenAI bills them as output.2

Counts also differ by language. A passage can take many times more tokens in one language than in another. The gap raises the cost for speakers of those languages, slows replies and leaves less room in the context window.5

The word has other meanings in technology. A crypto token is a unit of a digital currency, and an access token is a credential that lets a program use a service. Neither is a piece of text for a model to read.

§02

How it works

A tokenizer learns its vocabulary from a large sample of text. One common method, byte pair encoding, starts from single characters or bytes. It finds the pair of neighboring symbols that occurs most often, merges it into a new symbol, and repeats. Frequent words and word parts end up as single entries. GPT-2’s vocabulary held 50,257.

Each entry has an ID number. The tokenizer turns a prompt into a list of these IDs. The model turns each ID into an embedding, a list of numbers it can compute with. It then predicts the next token, one at a time, and the tokenizer turns the chosen IDs back into text.

Because the model receives IDs rather than letters or digits, some easy tasks become hard. A tokenizer might store “380” as one token and “381” as two, which hides how the digits relate. Reversing a word or solving an anagram is hard for the same reason.6

Multimodal models cut images and sound into tokens too. Google counts a small image as 258 tokens and each second of audio as 32.3

Fig. T2 · One sentence, six tokensIn characters

tiktoken is great!

  1. t 1
  2. ik 2
  3. token 5
  4. ␣is 3
  5. ␣great 6
  6. ! 1
OpenAI’s Cookbook shows cl100k_base, the encoding used by GPT-4, splitting this 18-character sentence into 6 tokens. Values count the characters in each token; ␣ marks a space the token carries.
§03

Origin and first known use

From Old English tacen, “sign, symbol, evidence, portent”; by the 1590s it also meant a coin-like piece of stamped metal. In 1906 the philosopher Charles Sanders Peirce used token for one occurrence of a word, and type for the word itself. Twenty uses of “the” on a page are twenty tokens of one type. Language processing took up the term, and by 1992 identifying tokens was called its first step. From 2015, translation researchers split rare words into learned pieces and counted text in them. A token came to mean a piece that may be smaller than a word.

  1. 1590s From Old English tacen, “sign, symbol, evidence, portent”, token has also come to mean a coin-like piece of stamped metal by the 1590s.
    Online Etymology Dictionary, “token” etymonline.com →: source for 1590s
  2. 1906 Charles Sanders Peirce calls one occurrence of a word a token, and the word itself a type. Twenty uses of “the” are twenty tokens of one type.
    Charles Sanders Peirce, The Monist 16 (1906), as quoted in David Liebesman, “Types and Tokens”, Stanford Encyclopedia of Philosophy, 2026 plato.stanford.edu →: source for 1906
  3. 1992 Webster and Kit call tokenization the first step of natural language processing: identifying tokens, the basic units that later processing does not break down.
    Jonathan J. Webster and Chunyu Kit, “Tokenization as the Initial Phase in NLP”, COLING 1992 aclanthology.org →: source for 1992
  4. 2012 Google’s Schuster and Nakajima build a WordPiece model for Japanese and Korean voice search, learning word units from text by repeatedly combining two smaller units.
    Mike Schuster and Kaisuke Nakajima, “Japanese and Korean Voice Search”, ICASSP 2012 research.google.com →: source for 2012
  5. 2015 Sennrich, Haddow and Birch split rare words into subword units with byte pair encoding, which repeatedly merges the most frequent pair of symbols. They count the result in tokens.
    Rico Sennrich, Barry Haddow and Alexandra Birch, “Neural Machine Translation of Rare Words with Subword Units”, ACL 2016 arXiv 1508.07909 →: source for 2015
  6. 2019 OpenAI’s GPT-2 runs byte pair encoding on raw bytes, so any text can be encoded, and expands its vocabulary to 50,257 entries.
    Radford et al., “Language Models are Unsupervised Multitask Learners”, OpenAI, 14 February 2019 cdn.openai.com →: source for 2019
  7. 2022 OpenAI’s guide to its open-source tokenizer, tiktoken, notes that usage is priced by token. A count shows whether text fits a model and what a call costs.
    Ted Sanders, “How to count tokens with tiktoken”, OpenAI Cookbook, 16 December 2022 developers.openai.com →: source for 2022
  8. 2023 Petrov and colleagues show that the same text, translated, can take up to 15 times as many tokens in one language as in another.
    Aleksandar Petrov, Emanuele La Malfa, Philip H.S. Torr and Adel Bibi, “Language Model Tokenizers Introduce Unfairness Between Languages”, NeurIPS 2023 arXiv 2305.15425 →: source for 2023
§04

In the wild

8 sightings · 2015–2026 · oldest first
  1. Paper31 Aug 2015

    “We compare different segmentation algorithms on the basis of the number of tokens, which affects efficiency…”

    Sennrich et al., ACL 2016 arXiv 1508.07909 →: Sennrich et al., ACL 2016, 31 Aug 2015Archived →: copy of Sennrich et al., ACL 2016, 31 Aug 2015
  2. Paper11 Oct 2018

    “We use WordPiece embeddings (Wu et al., 2016) with a 30,000 token vocabulary.”

    Devlin et al., Google AI Language arXiv 1810.04805 →: Devlin et al., Google AI Language, 11 Oct 2018Archived →: copy of Devlin et al., Google AI Language, 11 Oct 2018
  3. Docs16 Dec 2022

    “Splitting text strings into tokens is useful because GPT models see text in the form of tokens.”

    Ted Sanders, OpenAI Cookbook developers.openai.com →: Ted Sanders, OpenAI Cookbook, 16 Dec 2022Archived →: copy of Ted Sanders, OpenAI Cookbook, 16 Dec 2022
  4. Paper17 May 2023

    “The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases.”

    Petrov et al., NeurIPS 2023 arXiv 2305.15425 →: Petrov et al., NeurIPS 2023, 17 May 2023
  5. Blog8 Jun 2023

    “Large language models … work in terms of tokens. They take text, convert it into tokens (integers), then predict which tokens should come next.”

    Simon Willison, on his blog simonwillison.net →: Simon Willison, on his blog, 8 Jun 2023Archived →: copy of Simon Willison, on his blog, 8 Jun 2023
  6. Headline6 Jul 2024

    “Tokens are a big reason today’s generative AI falls short”

    TechCrunch techcrunch.com →: TechCrunch, 6 Jul 2024Archived →: copy of TechCrunch, 6 Jul 2024
  7. Court record23 Jun 2025

    “…all characters were grouped into short sequences and translated into corresponding number sequences or “tokens” according to an Anthropic-made dictionary.”

    Judge William Alsup, order on fair use in Bartz v. Anthropic Opinion →: Judge William Alsup, order on fair use in Bartz v. Anthropic, 23 Jun 2025Archived →: copy of Judge William Alsup, order on fair use in Bartz v. Anthropic, 23 Jun 2025
  8. Docs23 Sep 2026

    “Gemini and other generative AI models process input and output at a granularity called a token.”

    Google, Gemini API documentation ai.google.dev →: Google, Gemini API documentation, 23 Sep 2026Archived →: copy of Google, Gemini API documentation, 23 Sep 2026
§05

Not to be confused with

  • Word

    A word is a unit of language; a token is whatever unit a tokenizer produces. Common English words are often single tokens, and rare words split into several. OpenAI estimates a token at about three quarters of an English word.

  • Crypto token

    A unit of a cryptocurrency or other digital asset recorded on a blockchain, which can be owned and traded. An AI token is a piece of text a model processes; it is counted, not owned.

  • Access token

    A credential, like an API key, that proves a program may use a service. It is sent with each request to a model provider, but as proof of permission, not as text for the model to read.

  • Context window

    The maximum number of tokens a model can handle in one request, counting its input and its reply. The token is the unit of measure; the context window is a limit stated in that unit.

    See context window →
§06

Questions

  • What is a token in AI?

    A token is the unit of text that a language model reads and writes. It may be a whole word, a piece of a longer word, a punctuation mark or a word with the space before it. A program called a tokenizer splits text into tokens before the model sees it. The limits and prices of models are counted in tokens.

  • How many words are in 1,000 tokens?

    About 750 in English, by OpenAI’s estimate of three quarters of a word per token. Google puts 100 tokens at 60 to 80 English words for its Gemini models. The figures are averages. The same passage can need many times more tokens in another language. Models also count differently: Anthropic says its newer tokenizer produces about 30% more tokens for the same text.

  • How does token-based pricing work for AI models?

    Providers charge for the tokens a model reads and the tokens it writes, and the two can be priced differently. Anthropic quotes prices per million tokens: in 2026 it listed one model at $4 per million input tokens and $20 per million output tokens. Reasoning models also produce tokens the user never sees, and OpenAI bills those as output tokens.

  • Why do some languages use more tokens than English?

    The difference comes from the tokenizer, which splits the same passage into more pieces in some languages than in others. Petrov and colleagues found gaps of up to 15 times, even in tokenizers built for many languages. Speakers of those languages pay more for the same content, wait longer for replies and can fit less into a model’s context window.

  • Why do language models struggle to reverse words or do arithmetic?

    A model receives tokens as ID numbers, not as letters or digits. A word arrives whole or in a few chunks, so the model does not see its letters one by one. That makes reversing words and solving anagrams hard. Numbers can be split unevenly: a tokenizer might store “380” as one token and “381” as two. That hides how the digits relate, which makes arithmetic harder.

§07

Watch

§09

Cite this entry

“Token.” SUPERR_dictionary, Superrduperr, 24 Sept. 2026, superrduperr.com/dictionary/token.

First published 24 Sep 2026 · Last reviewed 24 Sep 2026 · Report an error →