Start a project
Safety Reviewed 24 Sep 2026 · 19 sources

red teaming

/ˈrɛd ˈtiː.mɪŋ/ noun Verb red team · nouns red team, red teamer · also spelled red-teaming
Definition

Authorized testing in which people or other models deliberately attack an AI system, as an adversary would, to find harmful outputs and security flaws so they can be fixed.

Examples
  • More than 50 outside experts probing GPT-4 before its release
  • One language model writing test questions to draw offensive replies from another
  • About 2,200 DEF CON attendees trying to make chatbots misbehave
A tester wearing a red armband feeds question cards into a model box on a test bench. Harmful replies are pinned to a board marked fix before release. The door to the public stays shut.
Inkling · Attacks on purpose, with every failure pinned up for fixingFig. R1
Earliest AI use
2018
A report on malicious uses of AI urges borrowing it from computer security.
Current meaning
2022
Probing language models for harmful replies, not only for security holes.
Opposite
Blue team
The defenders. In 1960s US military exercises, blue was the United States.
§01

In plain words

In AI, red teaming is testing in which people or models attack a system on purpose, as an adversary would, to find harmful outputs and flaws to fix.

Red teaming borrows a military habit: to find the weak points in a plan, someone is asked to play the enemy. In AI, the enemy is played by testers who set out to make a model or an AI product misbehave. They look for offensive or biased replies, leaks of private data, help with biological, chemical or cyber attacks, and ways around the model’s safety training. Each failure they find is one the developer can try to fix before release.

Labs red team their own models and also bring in outsiders. OpenAI engaged more than 50 outside experts, in fields from chemistry to law, to probe GPT-4 before its launch.1 In 2023 it opened a standing network of outside red teamers. Some red teaming is automated, with one large language model writing the attacks on another. That same year, about 2,200 people at the DEF CON hacking conference tried to make chatbots from Google, Meta, OpenAI and others misbehave.

The term reached policy in 2023 as well. Seven companies made voluntary commitments to the White House that included red-teaming their models, and an executive order defined AI red-teaming. That order was revoked in January 2025.

Red teaming is often confused with evals, fixed tests scored the same way every time. Red teaming is an open-ended search for failures that no test covers yet, and what it finds can become new evals.2 A jailbreak is one kind of attack that red teamers try. Critics say the term is used too loosely. A 2024 review found AI red-teaming efforts differing in purpose, in what they tested, in who took part and in what their findings led to.3

§02

How it works

An exercise begins with a scope: the system to test, the harms to look for and the people who will look. Testers may be staff, hired specialists or members of the public. Each attack is a prompt, or a whole conversation, meant to draw out a failure. Testers record which attacks worked.

Those records can be large. Anthropic released 38,961 attacks from its own red teaming. It found that models trained with reinforcement learning from human feedback grew harder to red team as they grew larger.4

Results vary from one try to the next. A generative model is probabilistic, so an attack that fails once can succeed later. Red teamers therefore repeat their attempts.5

Automation adds scale. OpenAI reports that automated methods produce attacks at a larger scale but have struggled to make them tactically diverse. It pairs them with human red teamers.6

Findings then feed fixes, such as further training or guardrails placed around the model. Red teamers go back to test whether those fixes hold.2

§03

Origin and first known use

Red stood for the enemy. In US military exercises as early as the 1960s, a red team modeled the Soviet Union’s behavior against a blue team representing the United States. Computer security took up the practice, and AI researchers borrowed it from there: a 2018 report on malicious uses of AI called for exploring it. In 2022 the term moved to language models, as labs used people and other models to draw out harmful replies. In 2023 it entered US policy through voluntary company commitments and an executive order, since revoked.

  1. 1960s The US military applies the term red team to modeling the Soviet Union’s behavior, set against a blue team representing the United States.
    Feffer et al., “Red-Teaming for Generative AI: Silver Bullet or Security Theater?”, AIES 2024, citing Micah Zenko, Red Team: How to Succeed by Thinking Like the Enemy (2015) arXiv 2401.15897 →: source for 1960s
  2. 2018 Brundage and colleagues’ report on malicious uses of AI calls for exploring red teaming, a practice to import from computer security. Earliest AI use found.
    Brundage et al., “The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation”, February 2018 arXiv 1802.07228 →: source for 2018
  3. 2020 Brundage and colleagues urge AI developers to run red team exercises, which they define as structured hunts for flaws and vulnerabilities, often by people adopting an attacker’s mindset.
    Brundage et al., “Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims”, April 2020 arXiv 2004.07213 →: source for 2020
  4. 2022 Perez and colleagues use one language model to write test questions for another, finding tens of thousands of offensive replies and leaked private data. The current sense starts here.
    Perez et al., “Red Teaming Language Models with Language Models”, EMNLP 2022 arXiv 2202.03286 →: source for 2022
  5. 2023 Seven companies, including Amazon, Anthropic, Google, Meta, Microsoft and OpenAI, make voluntary commitments to the White House that include internal and external red-teaming of their models.
    The White House, “FACT SHEET: Biden-Harris Administration Secures Voluntary Commitments from Leading Artificial Intelligence Companies to Manage the Risks Posed by AI”, 21 July 2023 bidenwhitehouse.archives.gov →: source for 2023
  6. 2023 About 2,200 people at the DEF CON hacking conference take part in a public red teaming exercise on models from Anthropic, Google, Meta, OpenAI and other companies.
    Sven Cattell, “Generative Red Team Recap”, AI Village, 12 October 2023 aivillage.org →: source for 2023
  7. 2023 Executive Order 14110 defines AI red-teaming and directs the Commerce Secretary to require developers of dual-use foundation models to report their red-team test results.
    Executive Order 14110, “Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence”, signed 30 October 2023, Federal Register, 1 November 2023 federalregister.gov →: source for 2023
  8. 2025 Executive Order 14148, signed on 20 January 2025, revokes Executive Order 14110, which had defined AI red-teaming.
    Executive Order 14148, “Initial Rescissions of Harmful Executive Orders and Actions”, signed 20 January 2025, Federal Register, 28 January 2025 federalregister.gov →: source for 2025
§04

In the wild

10 sightings · 2018–2025 · oldest first
  1. Paper20 Feb 2018

    “…we highlight the need to explore and potentially implement red teaming, formal verification, responsible disclosure of AI vulnerabilities…”

    Brundage et al., arXiv preprint arXiv 1802.07228 →: Brundage et al., arXiv preprint, 20 Feb 2018Archived →: copy of Brundage et al., arXiv preprint, 20 Feb 2018
  2. Paper7 Feb 2022

    “…we automatically find cases where a target LM behaves in a harmful way, by generating test cases (“red teaming”) using another LM.”

    Perez et al., EMNLP 2022 arXiv 2202.03286 →: Perez et al., EMNLP 2022, 7 Feb 2022Archived →: copy of Perez et al., EMNLP 2022, 7 Feb 2022
  3. Blog19 Jul 2023

    “The term came from the military, and described activities where a designated team would play an adversarial role (the “Red Team”) against the “home” team.”

    Daniel Fabian, head of Google Red Teams, on Google’s blog blog.google →: Daniel Fabian, head of Google Red Teams, on Google’s blog, 19 Jul 2023Archived →: copy of Daniel Fabian, head of Google Red Teams, on Google’s blog, 19 Jul 2023
  4. Docs21 Jul 2023

    “Commit to internal and external red-teaming of models or systems in areas including misuse, societal risks, and national security concerns…”

    The White House, voluntary commitments by seven AI companies bidenwhitehouse.archives.gov →: The White House, voluntary commitments by seven AI companies, 21 Jul 2023Archived →: copy of The White House, voluntary commitments by seven AI companies, 21 Jul 2023
  5. Blog7 Aug 2023

    “Unlike traditional security red teaming, which mostly focuses on only malicious adversaries, AI red teaming considers broader set of personas and failures.”

    Ram Shankar Siva Kumar, Microsoft AI Red Team, on the Microsoft Security blog microsoft.com →: Ram Shankar Siva Kumar, Microsoft AI Red Team, on the Microsoft Security blog, 7 Aug 2023Archived →: copy of Ram Shankar Siva Kumar, Microsoft AI Red Team, on the Microsoft Security blog, 7 Aug 2023
  6. Headline19 Sep 2023

    “OpenAI launches a red teaming network to make its models more robust”

    TechCrunch techcrunch.com →: TechCrunch, 19 Sep 2023Archived →: copy of TechCrunch, 19 Sep 2023
  7. Docs1 Nov 2023

    “The term “AI red-teaming” means a structured testing effort to find flaws and vulnerabilities in an AI system…”

    Executive Order 14110, section 3(d), in the Federal Register federalregister.gov →: Executive Order 14110, section 3(d), in the Federal Register, 1 Nov 2023Archived →: copy of Executive Order 14110, section 3(d), in the Federal Register, 1 Nov 2023
  8. Paper29 Jan 2024

    “…gestures towards red-teaming (based on public definitions) as a panacea for every possible risk verge on security theater.”

    Feffer et al., AIES 2024 arXiv 2401.15897 →: Feffer et al., AIES 2024, 29 Jan 2024Archived →: copy of Feffer et al., AIES 2024, 29 Jan 2024
  9. Blog12 Jun 2024

    “The lack of standardized practices for AI red teaming further complicates the situation.”

    Anthropic, post on red teaming methods anthropic.com →: Anthropic, post on red teaming methods, 12 Jun 2024Archived →: copy of Anthropic, post on red teaming methods, 12 Jun 2024
  10. Paper15 Jan 2025

    “…LLM red teaming is defined as a limit-seeking, non-malicious, manual activity, which depends highly on a team-effort and an alchemist mindset.”

    Inie, Stray and Derczynski, PLOS ONE plos.org →: Inie, Stray and Derczynski, PLOS ONE, 15 Jan 2025Archived →: copy of Inie, Stray and Derczynski, PLOS ONE, 15 Jan 2025
§05

Not to be confused with

  • Evals

    Evals are fixed sets of test cases, scored the same way each time so results can be compared across versions. Red teaming is an open-ended hunt for failures no test covers yet, and its findings can become new evals.

    See evals →

  • Jailbreak

    A jailbreak is one kind of attack: a prompt that gets a model past its safety training, whoever writes it. Red teaming is authorized testing that tries jailbreaks among many other attacks, to find weaknesses to fix.

    See jailbreak →

  • Penetration testing

    A penetration test is a security exercise that tries to break into computer systems to find exploitable holes. AI red teaming borrows the attacker’s role but also hunts for harmful or biased output, which involves no break-in.

§06

Questions

  • What is red teaming in AI?

    Red teaming is authorized testing in which people, or other models, attack an AI system on purpose to find its failures. Testers look for offensive or biased replies, leaks of private data, help with biological, chemical or cyber attacks, and ways around safety training. Developers then try to fix what they find. OpenAI, for example, engaged more than 50 outside experts to test GPT-4 before its launch.

  • Where does the term red teaming come from?

    The term comes from the military. In US exercises as early as the 1960s, a red team modeled the Soviet Union’s behavior, while a blue team represented the United States. Computer security later adopted the practice. A 2018 report on malicious uses of AI urged researchers to borrow it. From 2022, labs applied it to the harmful replies of language models.

  • What is the difference between red teaming and evals?

    Evals are fixed sets of test cases, scored the same way each time so that versions of a model can be compared. Red teaming is an open-ended search for failures that no existing test covers. The two work together. OpenAI says red teaming shows where automated evaluations are worth building and supplies the data to pilot them.

  • Can AI red teaming be automated?

    Partly. In 2022 Perez and colleagues had one language model write test questions for another and found tens of thousands of offensive replies from a chatbot. OpenAI reports that automated methods produce attacks at a larger scale but have struggled to make them tactically diverse. It pairs them with human red teamers. A 2025 study based on interviews with practitioners defines LLM red teaming as a manual activity.

  • What did the US executive order say about AI red-teaming?

    Executive Order 14110 was signed in October 2023. It defined AI red-teaming as structured testing to find an AI system’s flaws and vulnerabilities, often in a controlled setting with its developers. It also directed the Commerce Secretary to require developers of dual-use foundation models to report their red-team test results. Executive Order 14148 revoked the order on 20 January 2025.

§07

Watch

§09

Cite this entry

“Red teaming.” SUPERR_dictionary, Superrduperr, 24 Sept. 2026, superrduperr.com/dictionary/red-teaming.

First published 24 Sep 2026 · Last reviewed 24 Sep 2026 · Report an error →