multimodal
Of an AI model or system, able to take in or produce more than one kind of data, such as text, images, audio or video.
- A chatbot answering questions about an uploaded photo
- A voice assistant that hears a question and answers aloud
- A model describing what happens in a video clip
- A 1989 map display taking spoken commands and mouse pointing
- Earliest AI use
- 1989
- Earliest found: CUBRICON, an interface mixing speech, maps and pointing.
- Current meaning
- 2023
- GPT-4 launches as a large multimodal model, reading images and text.
- Opposite
- Unimodal
- Limited to one kind of data, such as a text-only model.
In plain words
In AI, multimodal describes a model that can work with several kinds of data, such as text, images, audio and video, rather than just one.
A modality is a kind of data, or a way of experiencing something: text, images, speech, video. Researchers call a problem multimodal when it involves more than one modality, and a model limited to one unimodal.1 A multimodal model can answer a question about a photo, or hear a spoken question and answer aloud.
Many AI systems take in a single kind of data: a spam filter reads text, and a photo classifier looks at images. In March 2023 OpenAI introduced GPT-4 as a large multimodal model that accepted images as well as text. Research papers call such models multimodal large language models, or MLLMs. In December Google launched Gemini as natively multimodal: trained on several kinds of data from the start, not assembled from separate parts.2 Meta described its open-weight Llama 4 models the same way in 2025.
Google’s Gemini demo drew criticism. TechCrunch reported that the video showed very different interactions from those used to make it. Google had prompted the model with still frames and text.3 Images can also carry misleading text. OpenAI showed that its CLIP model could be fooled into absurd classifications by text in an image, an attack needing only pen and paper.4
Multimodal is different from multi-model, which means using several separate models. A system can also handle several kinds of data by chaining separate models, each built for one task. ChatGPT’s voice feature worked this way before GPT-4o, and the model writing its replies never heard the audio.5
How it works
A model computes with numbers, so every input is converted first. Text is split into tokens. Some models turn images into tokens as well; Meta calls these vision tokens.
Designs differ in when the kinds of data meet. One design joins models trained separately. Flamingo, from 2022, bridged a pretrained vision model and a pretrained language model, so it could take images and text mixed in one sequence.6 Google described building from separate components stitched together as the standard approach before Gemini.2 The other design trains one model on several kinds of data from the start, which Google calls natively multimodal. Meta’s version, early fusion, feeds text and vision tokens into one model.
Some models learn to match two kinds of data rather than to generate them. OpenAI’s CLIP learned from text and image pairs found on the internet. It labels a new photo by predicting which caption, such as a photo of a dog, belongs with it.
Other systems chain separate models. Before GPT-4o, ChatGPT’s Voice Mode used three: one transcribed speech, GPT-3.5 or GPT-4 wrote a reply in text, and a third read it aloud. OpenAI said the main model could not observe tone, multiple speakers or background noise. GPT-4o replaced the chain with one neural network.5
Origin and first known use
From multi-, “many”, and modal, from mode. Merriam-Webster dates the first known use to 1899, in a general sense covering several modes, modalities or maxima. In machine learning a modality is the way something happens or is experienced, such as seeing or hearing. The earliest AI use found is from 1989, when researchers building CUBRICON, a system combining speech, text, maps and pointing, wrote of multi-modal communication. In 2011 Ngiam and colleagues used the word in the title of a deep learning paper on learning from audio and video together. Since 2023 OpenAI, Google and Meta have launched large models as multimodal, and researchers abbreviate multimodal large language model as MLLM.
- 1899 Merriam-Webster dates the first known use to 1899, in a general sense covering several modes, modalities or maxima, as in a distribution with several peaks.Merriam-Webster, “multimodal” merriam-webster.com →: source for 1899
- 1989 Neal and colleagues describe CUBRICON, an interface combining speech, text, maps and pointing, and call such communication multi-modal. Earliest AI use found.Neal, Thielman, Dobes, Haller and Shapiro, “Natural Language with Integrated Deictic and Graphic Gestures”, DARPA Speech and Natural Language Workshop, October 1989 aclanthology.org →: source for 1989
- 2011 Ngiam and colleagues at Stanford and the University of Michigan train deep networks to learn features from several modalities, such as audio and video, together.Ngiam, Khosla, Kim, Nam, Lee and Ng, “Multimodal Deep Learning”, ICML 2011 stanford.edu →: source for 2011
- 2021 OpenAI’s CLIP learns from text and image pairs on the internet. It labels a photo by predicting which caption, such as a photo of a dog, belongs with it.Radford, Sutskever, Kim, Krueger and Agarwal, “CLIP: Connecting text and images”, OpenAI, 5 January 2021 openai.com →: source for 2021
- 2023 OpenAI introduces GPT-4 as a large multimodal model that takes images and text and replies in text. Image input stays a research preview at launch.OpenAI, GPT-4 research announcement, 14 March 2023 openai.com →: source for 2023
- 2023 Google launches Gemini as natively multimodal: pre-trained from the start on different modalities, and able to combine text, code, audio, images and video.Sundar Pichai and Demis Hassabis, “Introducing Gemini: our largest and most capable AI model”, Google, 6 December 2023 blog.google →: source for 2023
- 2024 OpenAI’s GPT-4o accepts any mix of text, audio, image and video and processes it in one neural network, replacing a three-model voice pipeline.OpenAI, GPT-4o announcement, 13 May 2024 openai.com →: source for 2024
- 2025 Meta introduces Llama 4 Scout and Maverick, open-weight models that are natively multimodal. They feed text and vision tokens into one backbone, a design Meta calls early fusion.Meta, “The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation”, 5 April 2025 ai.meta.com →: source for 2025
In the wild
-
“Similar multi-modal communication can facilitate human interaction with modern sophisticated information processing and decision-aiding computer systems.”
Neal et al., DARPA Speech and Natural Language Workshop aclanthology.org →: Neal et al., DARPA Speech and Natural Language Workshop, 15 Oct 1989Archived →: copy of Neal et al., DARPA Speech and Natural Language Workshop, 15 Oct 1989 -
“Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors.”
Baltrušaitis et al., arXiv preprint arXiv 1705.09406 →: Baltrušaitis et al., arXiv preprint, 26 May 2017Archived →: copy of Baltrušaitis et al., arXiv preprint, 26 May 2017 -
“…Flamingo models can be trained on large-scale multimodal web corpora containing arbitrarily interleaved text and images…”
-
“GPT‑4 is a large multimodal model (accepting image and text inputs, emitting text outputs)…”
-
“Recently, Multimodal Large Language Model (MLLM) represented by GPT-4V has been a new rising research hotspot…”
-
“We designed Gemini to be natively multimodal, pre-trained from the start on different modalities.”
Sundar Pichai and Demis Hassabis, Google blog blog.google →: Sundar Pichai and Demis Hassabis, Google blog, 6 Dec 2023Archived →: copy of Sundar Pichai and Demis Hassabis, Google blog, 6 Dec 2023 -
“Google’s best Gemini demo was faked”
-
“OpenAI says natively multimodal GPT-4o eats text, visuals, sound – and emits the same”
-
“We’re introducing Llama 4 Scout and Llama 4 Maverick, the first open-weight natively multimodal models…”
Meta AI, Llama 4 announcement ai.meta.com →: Meta AI, Llama 4 announcement, 5 Apr 2025Archived →: copy of Meta AI, Llama 4 announcement, 5 Apr 2025 -
“The Gemini API supports prompting with text, image, audio, and video data, also known as multimodal prompting.”
Google, Gemini API documentation ai.google.dev →: Google, Gemini API documentation, 23 Sep 2026Archived →: copy of Google, Gemini API documentation, 23 Sep 2026
Not to be confused with
-
Multi-model
Multi-model describes a system that uses several separate models, for example to route different requests. Multimodal describes the kinds of data a model or system handles. A single model can be multimodal, and a multi-model system can be text-only.
-
Text-to-image model
A text-to-image model turns a description into a picture: one kind of data in, another out. Researchers count this translation as multimodal. Launches such as GPT-4’s used the word for a model reading text and images together.
-
Pipeline of separate models
A system can handle speech by chaining separate models: one transcribes, one writes a text reply, one reads it aloud. The model writing the reply never hears the audio; OpenAI said its chain lost tone and background noise.
-
Multimodal distribution
In statistics, a multimodal distribution has more than one peak, or most frequent value. The phrase describes the shape of a set of numbers, not the kinds of data a model takes in.
Questions
What does multimodal mean in AI?
It describes an AI model or system that works with more than one kind of data, such as text, images, audio or video. Each kind of data is called a modality, and a model limited to one is unimodal. OpenAI introduced GPT-4 in 2023 as a large multimodal model because it accepted images as well as text.
What is the difference between multimodal and multi-model?
Multimodal is about kinds of data: a multimodal model handles more than one, such as text and images. Multi-model is about the number of models: a multi-model system uses several separate ones. The two can overlap. Before GPT-4o, ChatGPT’s Voice Mode chained three separate models to transcribe speech, write a reply and read it aloud. The system handled audio, but its main model read and wrote only text.
What does natively multimodal mean?
Google used the phrase when it launched Gemini in December 2023, for a model pre-trained from the start on several kinds of data. It said the standard approach until then had been to train separate components for each modality and stitch them together. Meta called its 2025 Llama 4 models natively multimodal too. They feed text and vision tokens into one model, a design Meta calls early fusion.
Is a text-to-image generator multimodal?
By the research definition, yes. A 2017 survey of multimodal machine learning lists translation, mapping data from one modality to another, among the field’s five core challenges. The word is also used for models that accept several kinds of input in one prompt. Google’s Gemini API, for example, takes text, images, audio and video, which it calls multimodal prompting.
When was multimodal first used in AI?
The earliest AI use found is from 1989. Researchers building CUBRICON, a system that combined speech, text, maps and pointing, wrote of multi-modal communication. In 2011 researchers at Stanford and the University of Michigan published Multimodal Deep Learning, on networks that learn from audio and video together. In 2023 OpenAI called GPT-4 a large multimodal model, and Google launched Gemini as natively multimodal.
Watch
- Lecture 1.1 - Introduction (CMU Multimodal Machine Learning course, Fall 2022)The first lecture of Carnegie Mellon’s Fall 2022 course on multimodal machine learning, introducing the field.
- [CVPR2023 Tutorial Talk] Large Multimodal Models: Towards Building and Surpassing Multimodal GPT-4Chunyuan Li’s CVPR 2023 tutorial talk on how large multimodal models are built, with GPT-4 as the reference point.
Cite this entry
“Multimodal.” SUPERR_dictionary, Superrduperr, 24 Sept. 2026, superrduperr.com/
Superrduperr. (2026, September 24). Multimodal. In SUPERR_dictionary. https://superrduperr.com/
Superrduperr. “Multimodal.” SUPERR_dictionary. September 24, 2026. https://superrduperr.com/