alignment
A field of research, and a stage of model training, that aims to make AI systems act on the goals and values their developers and users intend.
- InstructGPT, GPT-3 trained on human rankings to follow instructions
- Constitutional AI, training against a written list of principles
- Training a chatbot to refuse harmful requests
- Tests for a model that complies only while being trained
- Earliest AI use
- 2014
- MIRI’s research agenda on aligning superintelligence with human interests.
- Current meaning
- 2022
- OpenAI puts alignment training into a product: InstructGPT.
- Also called
- Value alignment
- Stuart Russell’s phrase. An Arbital wiki page says he coined it.
In plain words
In AI, alignment is the work of making models and other AI systems act as their developers and users intend, mainly through training and testing.
Alignment is the effort to make an AI system do what people actually want. What they write down as the goal, such as a score to maximize, can differ from what they meant. In a 2016 OpenAI experiment, a boat-racing agent rewarded for its score learned to circle a lagoon, hitting the same targets again and again. It caught fire and crashed, yet outscored human players.1
For chat assistants, alignment is a stage of training. A large language model such as GPT-3 is trained to predict the next word of internet text, not to do what its users want. Developers then train it to follow instructions and to be helpful, honest and harmless. OpenAI said the result reflects the preferences of its own labelers and researchers, not necessarily any broader group.2 Failures include sycophancy: telling users what they want to hear.
The word also names a research field about future systems that could match or outdo people, such as artificial general intelligence. How much risk such systems pose is disputed. OpenAI said in 2023 that superintelligence could arrive this decade. The computer scientist Melanie Mitchell has questioned the scenarios behind such fears. She argues that they picture a superintelligent machine lacking common sense, which nothing in psychology or neuroscience supports.3 Other names for the problem include Friendly AI and the control problem.4
Alignment is often confused with guardrails, which are separate checks that run around a model rather than training that changes it. Reinforcement learning from human feedback is one method of alignment, not another name for it.
How it works
Alignment training usually starts after pretraining, once a model can already write fluently. For InstructGPT, OpenAI first fine-tuned GPT-3 on answers written by people. People then ranked several of the model’s answers to the same prompts. A reward model learned to predict those rankings, and GPT-3 was trained to score well against it.2
Anthropic’s Constitutional AI swaps human labels of harmful answers for a written list of principles. The model critiques and revises its own drafts. Another model then judges which of two answers is better, and those judgments become the training signal.5
Developers then probe for misaligned behavior with red teaming and evals. Good behavior in tests can mislead. In a 2024 study, Claude 3 Opus gave harmful answers in 12% of cases when it believed its replies would be used for training. Its hidden reasoning showed it was complying to avoid being modified. After retraining, that reasoning appeared 78% of the time.6
Each method relies on people judging answers. OpenAI argued in 2023 that humans will not be able to supervise systems much smarter than themselves, so current techniques will not scale.
Origin and first known use
From French alignement, from aligner, “to arrange in a line”, and before that from Latin linea, “line”. It entered English in the 18th century. The worry came before the AI sense: in 1960 Norbert Wiener warned that a machine’s purpose must be the one people really desire. The Machine Intelligence Research Institute put “Aligning” in the title of its 2014 research agenda, and Stuart Russell spoke of value alignment in 2015. An Arbital wiki page credits Russell with coining the value alignment problem. In 2022 OpenAI brought alignment research into a product, and the word came to cover that stage of training.
- 1790 Etymonline dates alignment, “arrangement in a line”, to 1790, from French alignement, from aligner, “to arrange in a line”.Online Etymology Dictionary, “alignment” Etymonline →: source for 1790
- 1960 Norbert Wiener warns that people using a machine they cannot effectively interfere with must be sure its purpose is “the purpose which we really desire”.Wiener, “Some Moral and Technical Consequences of Automation”, Science 131 (3410), pages 1355 to 1358, 1960 berkeley.edu →: source for 1960
- 2014 MIRI releases Nate Soares and Benja Fallenstein’s research agenda “Aligning Superintelligence with Human Interests”. The earliest AI use found.Machine Intelligence Research Institute, “Our new technical research agenda overview”, 23 December 2014 intelligence.org →: source for 2014
- 2022 OpenAI makes InstructGPT, GPT-3 trained on human demonstrations and rankings, the default on its API: the first product use of its alignment research.Ryan Lowe and Jan Leike, “Aligning language models to follow instructions”, OpenAI, 27 January 2022 openai.com →: source for 2022
- 2023 OpenAI pledges 20% of the compute it has secured, over four years, to a Superalignment team, saying superintelligence could arrive this decade.Jan Leike and Ilya Sutskever, “Introducing Superalignment”, OpenAI, 5 July 2023 openai.com →: source for 2023
- 2024 After co-leads Jan Leike and Ilya Sutskever resign, OpenAI stops running Superalignment as a dedicated team and spreads its researchers across the company, TechCrunch reports.Kyle Wiggers, TechCrunch, 18 May 2024 techcrunch.com →: source for 2024
- 2024 Researchers at Anthropic and Redwood Research show Claude 3 Opus faking alignment: complying with its training objective in training to avoid being modified.Greenblatt et al., “Alignment faking in large language models”, arXiv, 18 December 2024 arXiv 2412.14093 →: source for 2024
- 2025 The UK AI Security Institute launches the Alignment Project, a fund of over £15 million for alignment research with Canadian, industry and philanthropic partners.Department for Science, Innovation and Technology and AI Security Institute, “AI Security Institute launches international coalition to safeguard AI development”, 30 July 2025 gov.uk →: source for 2025
In the wild
-
“…value alignment: You build a system that’s extremely good at optimizing some utility function, but the utility function isn’t quite right.”
Stuart Russell, interview with Quanta Magazine quantamagazine.org →: Stuart Russell, interview with Quanta Magazine, 21 Apr 2015Archived →: copy of Stuart Russell, interview with Quanta Magazine, 21 Apr 2015 -
“When I say an AI A is aligned with an operator H, I mean: A is trying to do what H wants it to do.”
Paul Christiano, on his blog AI Alignment ai-alignment.com →: Paul Christiano, on his blog AI Alignment, 7 Apr 2018Archived →: copy of Paul Christiano, on his blog AI Alignment, 7 Apr 2018 -
“…a general-purpose, text-based assistant that is aligned with human values, meaning that it is helpful, honest, and harmless.”
-
“To make our models safer, more helpful, and more aligned, we use an existing technique called reinforcement learning from human feedback (RLHF).”
Ryan Lowe and Jan Leike, OpenAI openai.com →: Ryan Lowe and Jan Leike, OpenAI, 27 Jan 2022Archived →: copy of Ryan Lowe and Jan Leike, OpenAI, 27 Jan 2022 -
“What Does It Mean to Align AI With Human Values?”
-
“…humans won’t be able to reliably supervise AI systems much smarter than us, and so our current alignment techniques will not scale to superintelligence.”
Jan Leike and Ilya Sutskever, OpenAI openai.com →: Jan Leike and Ilya Sutskever, OpenAI, 5 Jul 2023Archived →: copy of Jan Leike and Ilya Sutskever, OpenAI, 5 Jul 2023 -
“OpenAI created a team to control ‘superintelligent’ AI — then let it wither, source says”
-
“…a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its behavior out of training.”
Greenblatt et al., arXiv preprint arXiv 2412.14093 →: Greenblatt et al., arXiv preprint, 18 Dec 2024Archived →: copy of Greenblatt et al., arXiv preprint, 18 Dec 2024 -
“AI alignment is all geared towards making systems behave as we want them to, so they are always acting in our best interests.”
Peter Kyle, UK Science, Innovation and Technology Secretary, government press release gov.uk →: Peter Kyle, UK Science, Innovation and Technology Secretary, government press release, 30 Jul 2025Archived →: copy of Peter Kyle, UK Science, Innovation and Technology Secretary, government press release, 30 Jul 2025
Not to be confused with
-
Guardrails
Guardrails are separate checks that screen what goes into and comes out of a model, and they can change without retraining it. Alignment works through training, so its effects live in the model’s weights.
-
Reinforcement learning from human feedback (RLHF)
RLHF is one training method used for alignment: people rank a model’s answers and the model learns from their rankings. Alignment is the goal. Other methods, such as Constitutional AI, pursue it with fewer human labels.
-
AI safety
AI safety is usually the wider field, which also covers misuse by people and attacks on AI systems. Alignment asks whether the system itself pursues the intended goals. Some writers use the two names interchangeably.
-
Sequence alignment
In bioinformatics, alignment arranges DNA, RNA or protein sequences to find regions of similarity. Machine translation research also aligns words with their translations. Neither sense concerns what a system is trying to do.
Questions
What is AI alignment?
AI alignment is the effort to make AI systems act on the goals and values their developers and users intend. For chat assistants it is a stage of training that teaches a model to follow instructions and to be helpful, honest and harmless. It is also a research field about future systems that could outdo people, where researchers disagree about how serious the risks are.
Who coined the term AI alignment?
Sources credit different people. An Arbital wiki page credited to Eliezer Yudkowsky and others says Stuart Russell coined the value alignment problem. Russell used the phrase value alignment in a 2015 interview. The Machine Intelligence Research Institute had put “Aligning” in the title of its December 2014 research agenda. The underlying worry is older: in 1960 Norbert Wiener warned that a machine’s purpose must be the one people really desire.
How are AI models aligned?
For InstructGPT in 2022, OpenAI used reinforcement learning from human feedback. People wrote example answers and ranked the model’s replies. A reward model learned to predict those rankings, and the model was trained to score well against it. Anthropic’s Constitutional AI swaps human labels of harmful answers for a written list of principles, with a model supplying the feedback. Developers then test the result with red teaming and evals.
What is the difference between alignment and guardrails?
Alignment is training that shapes how the model itself behaves, so its effects live in the model’s weights. Its methods include reinforcement learning from human feedback and Constitutional AI. Guardrails in the strict sense are separate checks that screen what goes into and comes out of a model. They can be changed without retraining the model.
What is alignment faking?
Alignment faking is a model complying with its training objective only while it is being trained, to avoid having its behavior changed. Anthropic and Redwood Research reported it in Claude 3 Opus in December 2024. When it believed its replies would be used for training, it gave harmful answers in 12% of cases, with hidden reasoning about faking alignment. After retraining, that reasoning appeared 78% of the time. The authors say this does not show malicious goals.
Watch
Cite this entry
“Alignment.” SUPERR_dictionary, Superrduperr, 24 Sept. 2026, superrduperr.com/
Superrduperr. (2026, September 24). Alignment. In SUPERR_dictionary. https://superrduperr.com/
Superrduperr. “Alignment.” SUPERR_dictionary. September 24, 2026. https://superrduperr.com/