Start a project
Techniques Reviewed 24 Sep 2026 · 19 sources

reinforcement learning from human feedback

RLHF /ˌriː.ənˈfɔːrs.mənt ˈlɝː.nɪŋ frəm ˈhjuː.mən ˈfiːdˌbæk/ noun Abbreviation RLHF · variant reinforcement learning with human feedback
Definition

A technique that trains an AI model toward the answers people prefer, using a reward model fitted to their comparisons of its outputs as the score to maximize.

Examples
  • ChatGPT, trained on AI trainers’ rankings of alternative replies
  • A simulated robot taught a backflip from about 900 human choices
  • A summarizer tuned toward the summaries people preferred
A person holds two answer cards from a model and ticks the better one. The tick passes to a small judge holding a score dial, and the dial’s reading turns a knob on the model.
Inkling · People’s picks train a judge that then tunes the modelFig. R1
Earliest AI use
2017
Simulated robots and Atari agents trained on people’s pairwise choices.
Current meaning
2022
Language models: OpenAI’s core method for InstructGPT, a tuned GPT-3.
§01

In plain words

Reinforcement learning from human feedback (RLHF) is a way to train an AI model toward the answers people prefer, using their rankings of its outputs.

A large language model fresh from pretraining predicts text well but is not yet a helpful assistant. Reinforcement learning from human feedback is one of the training steps that close the gap. People compare several of the model’s answers to the same prompt and rank them. The model is then trained to produce more answers like the ones ranked highest.

The approach began outside language. In 2017, researchers at OpenAI and DeepMind taught a simulated robot to do a backflip without writing a goal function for it. A person was shown two attempts at a time and picked the better one. About 900 such choices were enough.1

OpenAI then applied the method to language models. In January 2022 it made InstructGPT, a version of GPT-3 trained this way, the default in its API. About 40 contractors supplied the example answers and rankings.2 ChatGPT followed in November 2022. Anthropic, DeepMind and Meta have used the technique or close variants.

The model learns whatever the judges reward, including their biases. OpenAI said early ChatGPT was often excessively verbose because its trainers had favored longer answers that seemed more comprehensive.3 In 2025 it said a reward built from users’ thumbs-up ratings can favor agreeable answers, a failure known as sycophancy.4 A 2023 survey called RLHF the central method for fine-tuning leading language models and set out its open problems.

RLHF is a form of fine-tuning, but it learns from people’s judgments of the model’s own answers, not from example answers written for it. It also differs from human in the loop designs, in which a person checks particular outputs while a system runs.

§02

How it works

RLHF usually starts from a model already fine-tuned on example answers written by people, as with InstructGPT and ChatGPT. Then come three steps. First, the model writes several answers to each prompt, and people rank them.2 Second, a separate network, the reward model, learns from those rankings to give any answer a score. Anthropic calls it a preference model.

Third comes reinforcement learning, in which a system improves by trial and error to earn a reward. The model writes answers, the reward model scores them, and the model’s weights shift toward higher-scoring answers. OpenAI used an algorithm called Proximal Policy Optimization for this step.

The reward model only imitates the judges, and the judges can be gamed. In 2019, summarizers trained this way scored well by copying whole sentences, perhaps exploiting labelers’ simple heuristics. Pushing too hard against a reward model also backfires: in a 2020 study, its scores came to run opposite to people’s preferences. A penalty for drifting far from the starting model keeps outputs close to what the reward model saw in training.5

Newer methods change the recipe. Direct preference optimization trains on the comparisons directly, with no separate reward model or reinforcement learning, and matched or beat RLHF in its authors’ tests.6 Anthropic’s Constitutional AI keeps reinforcement learning but has a model, guided by a list of principles, make the comparisons.

Fig. R2 · From rankings to trainingIllustrative, not measured

Write a two-sentence reply to a customer whose order arrived late.

Collect comparisons
The model drafts four replies, and a person ranks them from best to worst
Train a reward model
A second network learns to give any reply a score that predicts the ranking
Optimize the model
The model writes new replies and shifts toward those the reward model scores highly
The three core steps of RLHF, as used for InstructGPT and ChatGPT after supervised fine-tuning. The prompt, the replies and the ranking are invented.
§03

Origin and first known use

The name joins reinforcement learning, learning by trial and error to earn a reward, with human feedback, people’s judgments of a system’s behavior. Earlier systems, such as Knox and Stone’s TAMER of 2008, learned from rewards that people gave directly. In 2017, researchers at OpenAI and DeepMind titled their method deep reinforcement learning from human preferences. It was applied to language models from 2019. By January 2022, OpenAI called it reinforcement learning from human feedback (RLHF), a method it had helped pioneer. DeepMind wrote RLHP, for human preferences, that March. Meta’s Llama 3 model card uses the variant reinforcement learning with human feedback.

  1. 2008 Knox and Stone’s TAMER lets a person train a Tetris-playing agent with reward signals. The agent models the person’s reward and picks actions it predicts will earn most.
    Knox and Stone, “TAMER: Training an Agent Manually via Evaluative Reinforcement”, IEEE ICDL 2008 utexas.edu →: source for 2008
  2. 2017 Researchers at OpenAI and DeepMind train agents on a reward learned from people’s choices between two short stretches of behavior, in Atari games and simulated robot locomotion.
    Christiano, Leike, Brown, Martic, Legg and Amodei, “Deep reinforcement learning from human preferences”, arXiv, 12 June 2017 arXiv 1706.03741 →: source for 2017
  3. 2019 Ziegler and colleagues apply the method to pretrained language models. Their summarizers copy whole sentences and score well with labelers, perhaps by exploiting the labelers’ simple heuristics.
    Ziegler et al., “Fine-Tuning Language Models from Human Preferences” arXiv 1909.08593 →: source for 2019
  4. 2020 Stiennon and colleagues train summarizers against a reward model built from people’s comparisons. Their summaries of Reddit posts outperform the human-written reference summaries.
    Stiennon et al., “Learning to summarize from human feedback”, NeurIPS 2020 arXiv 2009.01325 →: source for 2020
  5. 2022 OpenAI fine-tunes GPT-3 on labelers’ example answers, then on their rankings, with a technique it calls reinforcement learning from human feedback (RLHF). InstructGPT becomes its API default.
    OpenAI, “Aligning language models to follow instructions”, 27 January 2022 openai.com →: source for 2022
  6. 2022 OpenAI launches ChatGPT, trained with RLHF like InstructGPT: trainers rank alternative replies, a reward model learns the rankings, and Proximal Policy Optimization fine-tunes the chatbot against it.
    OpenAI, “Introducing ChatGPT”, 30 November 2022 openai.com →: source for 2022
  7. 2022 Anthropic trains a harmless assistant without human labels of harm. A model, guided by a list of principles, judges which of two answers is better: RL from AI feedback (RLAIF).
    Bai et al., “Constitutional AI: Harmlessness from AI Feedback” arXiv 2212.08073 →: source for 2022
  8. 2023 Rafailov and colleagues propose direct preference optimization, which trains on people’s comparisons with a simple classification loss, skipping the separate reward model and reinforcement learning.
    Rafailov et al., “Direct Preference Optimization: Your Language Model is Secretly a Reward Model” arXiv 2305.18290 →: source for 2023
§04

In the wild

9 sightings · 2022–2024 · oldest first
  1. Blog27 Jan 2022

    “To train InstructGPT models, our core technique is reinforcement learning from human feedback (RLHF), a method we helped pioneer in our earlier alignment research.”

    OpenAI, InstructGPT announcement openai.com →: OpenAI, InstructGPT announcement, 27 Jan 2022Archived →: copy of OpenAI, InstructGPT announcement, 27 Jan 2022
  2. Paper21 Mar 2022

    “…we use reinforcement learning from human preferences (RLHP) to train “open-book” QA models that generate answers whilst also citing specific evidence for their claims…”

    Menick et al., DeepMind arXiv 2203.11147 →: Menick et al., DeepMind, 21 Mar 2022Archived →: copy of Menick et al., DeepMind, 21 Mar 2022
  3. Paper12 Apr 2022

    “We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants.”

    Bai et al., Anthropic arXiv 2204.05862 →: Bai et al., Anthropic, 12 Apr 2022Archived →: copy of Bai et al., Anthropic, 12 Apr 2022
  4. Blog30 Nov 2022

    “We trained this model using Reinforcement Learning from Human Feedback (RLHF), using the same methods as InstructGPT…”

    OpenAI, ChatGPT launch post openai.com →: OpenAI, ChatGPT launch post, 30 Nov 2022Archived →: copy of OpenAI, ChatGPT launch post, 30 Nov 2022
  5. Blog9 Dec 2022

    “That’s the idea of Reinforcement Learning from Human Feedback (RLHF); use methods from reinforcement learning to directly optimize a language model with human feedback.”

    Nathan Lambert and colleagues, Hugging Face blog huggingface.co →: Nathan Lambert and colleagues, Hugging Face blog, 9 Dec 2022Archived →: copy of Nathan Lambert and colleagues, Hugging Face blog, 9 Dec 2022
  6. Paper29 May 2023

    “…RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences…”

    Rafailov et al., arXiv preprint arXiv 2305.18290 →: Rafailov et al., arXiv preprint, 29 May 2023Archived →: copy of Rafailov et al., arXiv preprint, 29 May 2023
  7. Paper27 Jul 2023

    “RLHF has emerged as the central method used to finetune state-of-the-art large language models… there has been relatively little public work systematizing its flaws.”

    Casper et al., arXiv preprint arXiv 2307.15217 →: Casper et al., arXiv preprint, 27 Jul 2023Archived →: copy of Casper et al., arXiv preprint, 27 Jul 2023
  8. Transcript31 Aug 2023

    “…systems are being aligned today… using reinforcement learning from human feedback (RLHF)… But this assumes that the human knows exactly how the task works…”

    Jan Leike, OpenAI’s head of alignment research, interview with IEEE Spectrum spectrum.ieee.org →: Jan Leike, OpenAI’s head of alignment research, interview with IEEE Spectrum, 31 Aug 2023Archived →: copy of Jan Leike, OpenAI’s head of alignment research, interview with IEEE Spectrum, 31 Aug 2023
  9. Docs18 Apr 2024

    “The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety.”

    Meta, Llama 3 model card github.com →: Meta, Llama 3 model card, 18 Apr 2024Archived →: copy of Meta, Llama 3 model card, 18 Apr 2024
§05

Not to be confused with

  • Fine-tuning

    Supervised fine-tuning trains a model to copy example answers written by people. RLHF, itself a form of fine-tuning, trains it on people’s choices between its own answers. InstructGPT used supervised fine-tuning first, then RLHF.

    See fine-tuning →

  • Human in the loop

    Human in the loop puts a person in the path of particular outputs or actions while a system runs. In RLHF, people shape the model’s general behavior during training; they do not approve its answers in use.

    See human in the loop →

  • Reinforcement learning

    The broader method, in which a system learns by trial and error to earn a reward, such as a game score. In RLHF the reward comes from a model trained on people’s preferences, not from the task itself.

  • Alignment

    Alignment is the goal of making a model act as its developers and users intend. RLHF is one training method for reaching it. Other methods, such as Anthropic’s Constitutional AI, pursue it with far fewer human labels.

    See alignment →
§06

Questions

  • What does RLHF stand for in AI?

    RLHF stands for reinforcement learning from human feedback. People compare a model’s answers to the same prompt and rank them. A reward model learns to predict those rankings, and the model is then trained by reinforcement learning to write answers that score highly. OpenAI used it for InstructGPT and ChatGPT. Anthropic, DeepMind and Meta have used it or close variants.

  • How was ChatGPT trained with RLHF?

    OpenAI said in November 2022 that it trained ChatGPT with RLHF, using the same methods as InstructGPT. The model was first fine-tuned on example answers written by people. AI trainers then ranked alternative replies, a reward model learned their rankings, and an algorithm called Proximal Policy Optimization fine-tuned the chatbot against it. OpenAI said the trainers’ preference for longer answers made ChatGPT often excessively verbose.

  • What is the difference between RLHF and fine-tuning?

    RLHF is itself a kind of fine-tuning. Supervised fine-tuning trains a model on example answers that people wrote. RLHF trains it on people’s choices between answers the model produced, through a reward model that scores new answers. The two are usually combined. InstructGPT and ChatGPT were fine-tuned on examples first and then trained with RLHF, and Meta describes its tuned Llama 3 models the same way.

  • What are the problems with RLHF?

    The model learns whatever the judges reward, including their biases. OpenAI traced early ChatGPT’s verbosity to trainers preferring longer answers, and in 2025 said thumbs-up ratings can favor agreeable answers. Optimizing too hard against a reward model can make its scores run opposite to people’s preferences. OpenAI’s Jan Leike noted that the method assumes the judge knows exactly how the task works. A 2023 survey set out many more open problems.

  • What are the alternatives to RLHF?

    Two alternatives change the recipe. Direct preference optimization, proposed in 2023, trains on the same human comparisons with a simple classification loss, skipping the reward model and reinforcement learning. Its authors report it matching or beating RLHF in their tests. Anthropic’s Constitutional AI keeps reinforcement learning but has a model, guided by a list of principles, make the comparisons. Anthropic calls this reinforcement learning from AI feedback.

§07

Watch

§09

Cite this entry

“Reinforcement learning from human feedback.” SUPERR_dictionary, Superrduperr, 24 Sept. 2026, superrduperr.com/dictionary/reinforcement-learning-from-human-feedback.

First published 24 Sep 2026 · Last reviewed 24 Sep 2026 · Report an error →