Start a project
Foundations Reviewed 24 Sep 2026 · 19 sources

embedding

/ɪmˈbɛd.ɪŋ/ noun Plural embeddings · verb embed
Definition

A list of numbers, learned by a model, that represents a word, sentence, image or other content so that items with similar meanings get similar numbers.

Examples
  • A 1,536-number vector for each article on a blog
  • GPT-3’s 12,288 numbers for each token in its vocabulary
  • A photo of a dog, placed near the word dog by CLIP
Three word cards, labeled king, queen and apple, each turn into a strip of numbers and a point on a map. The points for king and queen sit close together; the point for apple sits far away.
Inkling · Words as points, with similar meanings close togetherFig. E1
Current meaning
2008
A neural network learns each word’s embedding; similar words cluster.
Also called
Word vector
A name for a word’s embedding, used in the 2013 word2vec paper.
Opposite
One-hot encoding
A long vector with one slot per item; it captures no similarity.
§01

In plain words

In AI, an embedding is a list of numbers that stands for a word, text or image, placed so that similar meanings sit close together.

Computers can compare numbers but not meanings. An embedding closes that gap by turning a piece of content into a list of numbers. The list has a fixed length set by the model that makes it, such as 300, 1,000 or 1,536 numbers.1 The values are learned from large amounts of data, not assigned by hand.

Each list works like a location in a space with many dimensions. Content with similar meaning lands close together, so distance becomes a measure of similarity. A food embedding can show that a hot dog is more like a shawarma than a salad. One-hot codes, which give each item its own slot, cannot show that.2

Semantic search relies on this: a query finds documents by meaning rather than by exact words. The same comparison drives related-content features, clustering and the retrieval step of retrieval-augmented generation. OpenAI says embeddings power knowledge retrieval in ChatGPT and in many RAG developer tools.3

Embeddings also absorb patterns from their training text, prejudice included. In a 2016 study, an embedding trained on Google News articles completed the analogy man is to computer programmer as woman is to homemaker.4

An embedding is often confused with a vector database. The embedding is the list of numbers for one item. A vector database stores many of them and quickly finds the ones nearest to a query. A token is different again: it is a chunk of text that a large language model reads. The model turns each token into an embedding before any other step.

§02

How it works

Embeddings rest on an idea from 1950s linguistics: words used in similar contexts tend to have similar meanings.5 Training puts the idea to work. A model reads large amounts of text and adjusts each word’s numbers until they help predict nearby words. Words that keep similar company end up with similar numbers.

Word2vec, described by Google researchers in 2013, learned word vectors from 1.6 billion words in under a day. Simple arithmetic works on vectors like these: king minus man plus woman lands nearest to queen.

Later models embed whole sentences or passages. Sentence-BERT, from 2019, produces one vector per sentence and compares vectors by cosine similarity, a measure of the angle between them.

A search system uses an embedding model twice. It embeds every document once and stores the results. At query time it embeds the question and returns the documents whose vectors sit closest.

Inside a transformer, the first matrix is the embedding matrix, with one column of numbers for each token in its vocabulary. GPT-3’s matrix has 50,257 columns of 12,288 numbers. They start as random values and are learned in training.6 The matrix is therefore part of the model’s weights.

§03

Origin and first known use

From embed, which entered English in 1778 as a geological word for fossils lodged in rock. Mathematics gave it a technical sense: a mapping from one space or structure to another. Jurafsky and Martin trace the AI sense to that mapping, although the meaning has shifted. Latent semantic analysis researchers wrote in 1997 of embedding word and passage data in a space of chosen size. In 2008 Collobert and Weston used it for the vector a neural network learns for each word. By 2019 it also covered vectors for whole sentences, and models such as CLIP embed images in the same space as text.

  1. 1778 Embed enters English as a geological term, for fossils lying in a bed of surrounding rock.
    Online Etymology Dictionary, “embed” Etymonline →: source for 1778
  2. 1997 Latent semantic analysis researchers write of embedding word and passage data in spaces of a chosen size. Jurafsky and Martin call it that community’s likely first use.
    Landauer, Laham, Rehder and Schreiner, “How Well Can Passage Meaning be Derived without Using Word Order? A Comparison of Latent Semantic Analysis and Humans”, Cognitive Science Society, 1997 escholarship.org →: source for 1997
  3. 2003 Bengio and colleagues train a neural network that learns a vector for each word while learning to predict the next word. The paper calls it a distributed representation.
    Bengio, Ducharme, Vincent and Jauvin, “A Neural Probabilistic Language Model”, Journal of Machine Learning Research 3, 2003 jmlr.org →: source for 2003
  4. 2008 Collobert and Weston’s network embeds each word in a space through a lookup table. The learned embedding, they report, clusters words with similar meanings.
    Collobert and Weston, “A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning”, ICML 2008 collobert.com →: source for 2008
  5. 2013 Mikolov and colleagues learn word vectors from 1.6 billion words in under a day. Simple arithmetic works on such vectors: king minus man plus woman lands nearest queen.
    Mikolov, Chen, Corrado and Dean, “Efficient Estimation of Word Representations in Vector Space”, Google arXiv 1301.3781 →: source for 2013
  6. 2016 Bolukbasi and colleagues show that word embeddings trained on Google News articles carry gender stereotypes, and propose ways to reduce them.
    Bolukbasi, Chang, Zou, Saligrama and Kalai, “Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings” arXiv 1607.06520 →: source for 2016
  7. 2019 Sentence-BERT embeds whole sentences for comparison by cosine similarity, cutting a search for the most similar pair among 10,000 sentences from about 65 hours to about 5 seconds.
    Reimers and Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks”, EMNLP 2019 arXiv 1908.10084 →: source for 2019
  8. 2022 OpenAI adds an embeddings endpoint to its API, with models for text and code, for tasks such as semantic search, clustering, topic modeling and classification.
    Neelakantan, Weng, Power and Jang, “Introducing text and code embeddings”, OpenAI, 25 January 2022 openai.com →: source for 2022
§04

In the wild

8 sightings · 2014–2025 · oldest first
  1. Blog7 Jul 2014

    “A word embedding … is a paramaterized function mapping words in some language to high-dimensional vectors (perhaps 200 to 500 dimensions).”

    Christopher Olah, on his blog colah.github.io →: Christopher Olah, on his blog, 7 Jul 2014Archived →: copy of Christopher Olah, on his blog, 7 Jul 2014
  2. Paper21 Jul 2016

    “We show that even word embeddings trained on Google News articles exhibit female/male gender stereotypes to a disturbing extent.”

    Bolukbasi et al., arXiv preprint arXiv 1607.06520 →: Bolukbasi et al., arXiv preprint, 21 Jul 2016Archived →: copy of Bolukbasi et al., arXiv preprint, 21 Jul 2016
  3. Paper12 Jun 2017

    “Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors…”

    Vaswani et al., arXiv preprint arXiv 1706.03762 →: Vaswani et al., arXiv preprint, 12 Jun 2017Archived →: copy of Vaswani et al., arXiv preprint, 12 Jun 2017
  4. Paper27 Aug 2019

    “…derive semantically meaningful sentence embeddings that can be compared using cosine-similarity.”

    Reimers and Gurevych, EMNLP 2019 arXiv 1908.10084 →: Reimers and Gurevych, EMNLP 2019, 27 Aug 2019Archived →: copy of Reimers and Gurevych, EMNLP 2019, 27 Aug 2019
  5. Blog25 Jan 2022

    “Embeddings are numerical representations of concepts converted to number sequences, which make it easy for computers to understand the relationships between those concepts.”

    Neelakantan, Weng, Power and Jang, OpenAI blog openai.com →: Neelakantan, Weng, Power and Jang, OpenAI blog, 25 Jan 2022Archived →: copy of Neelakantan, Weng, Power and Jang, OpenAI blog, 25 Jan 2022
  6. Blog23 Oct 2023

    “Embeddings are a really neat trick that often come wrapped in a pile of intimidating jargon.”

    Simon Willison, on his blog simonwillison.net →: Simon Willison, on his blog, 23 Oct 2023Archived →: copy of Simon Willison, on his blog, 23 Oct 2023
  7. Headline7 Mar 2025

    “Google debuts a new Gemini-based text embedding model”

    TechCrunch techcrunch.com →: TechCrunch, 7 Mar 2025Archived →: copy of TechCrunch, 7 Mar 2025
  8. Docs25 Aug 2025

    “An embedding is a vector representation of data in embedding space.”

    Google, Machine Learning Crash Course developers.google.com →: Google, Machine Learning Crash Course, 25 Aug 2025Archived →: copy of Google, Machine Learning Crash Course, 25 Aug 2025
§05

Not to be confused with

  • Vector database

    A vector database is software that stores many embeddings and quickly finds the ones nearest to a query. The embedding is the list of numbers for one item; the database holds and searches such lists.

    See vector database →

  • Token

    A token is a chunk of text, often part of a word, that a language model reads as one unit. The embedding is the list of numbers the model turns that token into.

    See token →

  • Weights

    Weights are all the numbers a model learns in training. A language model’s embedding matrix is one part of them. An embedding is the vector for one input, looked up in that matrix or computed by an embedding model.

    See weights →

  • Embedded system

    A computer built into another device, such as a car or a thermostat. The name shares the root embed but has no link to embeddings, which are lists of numbers standing for content.

§06

Questions

  • What is an embedding in AI?

    An embedding is a list of numbers that a model produces to represent a piece of content. The content can be a word, a sentence, a document or an image. The length is fixed by the model, often hundreds or thousands of numbers. Content with similar meaning gets similar numbers, so the distance between two embeddings works as a measure of similarity. Semantic search and retrieval-augmented generation both depend on this.

  • What is the difference between an embedding and a vector database?

    An embedding is the list of numbers that represents one item, such as a paragraph or a photo. A vector database is software that stores many embeddings and quickly finds the ones closest to a query’s embedding. In retrieval-augmented generation, an embedding model turns documents and questions into embeddings, and a vector database returns the documents nearest to each question.

  • How are embeddings created?

    A model learns them from large amounts of data. For words, training adjusts each word’s numbers until they help predict nearby words, so words used in similar contexts end up close together. Inside a transformer, the embedding matrix starts as random values and is learned in training; GPT-3’s holds 12,288 numbers for each of 50,257 tokens. Models such as Sentence-BERT produce one vector for a whole sentence.

  • What is the king minus man plus woman example?

    It is a test of whether word embeddings capture relationships between words. Take the vector for king, subtract the vector for man and add the vector for woman. The word vector nearest to the result is queen. The example appears in the 2013 word2vec paper by Google researchers, whose method learned word vectors from 1.6 billion words in under a day.

  • Can embeddings be biased?

    Yes. Embeddings learn from human-written text and absorb its patterns, prejudice included. In 2016, Bolukbasi and colleagues showed that word embeddings trained on Google News articles carried gender stereotypes. Their embedding completed the analogy man is to computer programmer as woman is to homemaker. The same paper proposed ways to reduce such stereotypes in an embedding.

§07

Watch

§09

Cite this entry

“Embedding.” SUPERR_dictionary, Superrduperr, 24 Sept. 2026, superrduperr.com/dictionary/embedding.

First published 24 Sep 2026 · Last reviewed 24 Sep 2026 · Report an error →