How RLHF Works: The Three-Stage Pipeline That Turns Human Preferences Into a Fine-Tuned Language Model

Reinforcement Learning from Human Feedback (RLHF) is a multi-stage pipeline that trains language models by incorporating human preferences. It involves supervised fine-tuning, training a reward model on human judgments, and then optimizing the language model using reinforcement learning.

David Katzman

November 6, 2026 · 7 min read

Reinforcement Learning from Human Feedback, or RLHF, is not a single training step. It is a multi-model pipeline in which a pretrained language model generates candidate responses, humans rank those responses, a separate reward model learns to score them numerically, and a reinforcement learning algorithm then uses that score to reshape the language model itself. The counterintuitive core of the method is that the humans never grade the final model directly — their judgments are distilled into an intermediate model that can hand out rewards automatically, thousands of times, without a person in the loop.

Understanding Reinforcement Learning from Human Feedback (RLHF)

RLHF exists to solve a problem that conventional machine learning objectives handle badly: many desirable behaviors are subjective or hard to quantify. As Zhaolin Gao of Carnegie Mellon University's Machine Learning Department explains in a technical tutorial, RLHF aligns AI systems with human preferences by training them on feedback from people rather than predefined reward functions, because manually coding every desirable behavior is often infeasible for complex tasks. It matters most where success is hard to define — generating helpful and safe text responses being the canonical example.

The technique's deceptive difficulty comes from its architecture. Nathan Lambert and colleagues at Hugging Face, in an early and widely cited walkthrough of the method, describe RLHF as "a challenging concept because it involves a multiple-model training process and different stages of deployment." They break it into three core steps: starting from a pretrained language model, gathering data and training a reward model, and fine-tuning the language model with reinforcement learning. Each stage produces a distinct artifact — a tuned starting model, a scoring model, and finally an aligned policy — and confusing those artifacts is the most common conceptual error.

Learning from human preferences is also older than the current wave of chatbots. Chip Huyen notes that the approach has been around for over a decade, with OpenAI exploring it back when its main bet was robotics. What changed is scale: the same preference-learning machinery now sits on top of models pretrained on enormous corpora.

Stage 1: Supervised Fine-Tuning and Human Preference Data Collection

The pipeline begins with a language model that has already been pretrained with classical objectives — next-token prediction over massive text corpora. Pretraining is by far the most expensive phase; Huyen cites the InstructGPT model as an example, where pretraining consumed roughly 98 percent of overall compute and data resources. Fine-tuning stages, in this framing, unlock capabilities the pretrained model already has but that users cannot reliably reach through prompting alone.

Before any reinforcement learning happens, the model typically undergoes supervised fine-tuning (SFT): it is trained on high-quality prompt-response pairs, often written or curated by humans, so it can follow instructions in the first place. AWS's explainer of the technique describes this as fine-tuning a commercial pretrained model so its responses to predetermined prompts can be compared against human responses collected beforehand.

SFT is not strictly mandatory — Huyen points out that you can run RLHF directly on top of the pretrained model, skipping the SFT phase. Empirically, however, combining all three steps gives the best performance. Once the SFT model exists, it is used to generate multiple candidate responses to the same prompt, and human labelers compare those outputs — typically ranking them or selecting which of a pair is better. These comparisons, not absolute grades, are the raw material for the next stage.

Stage 2: Training the Reward Model with Human Preferences

Human preference data is qualitative: "response A is better than response B." Reinforcement learning needs a number. The reward model is the bridge. As IBM's technical overview puts it, a reward model is needed to translate human preference into a numerical reward signal, and designing an effective one is crucial because no straightforward mathematical formula can define subjective human values. Once trained on enough direct feedback from human evaluators, the reward model mimics how humans allocate rewards to different kinds of responses — allowing training to continue offline, without a human in the loop.

The standard mathematical engine for this translation is the Bradley-Terry model. Nathan Lambert's RLHF Book gives the core definition: the model defines the probability that a judge prefers item i over item j as the ratio of i's latent strength to the sum of both strengths. Each item is assumed to have a latent quality, and observed preferences are treated as a noisy reflection of those underlying strengths. An ETH Zurich research note describes the standard pipeline concretely: given a dataset of pairwise comparisons, the reward model's parameters are estimated by log-likelihood maximization, and the estimated reward function is then used to train the LLM with standard RL algorithms such as PPO.

This noise-tolerant design is what makes the system workable with real labelers. Because the Bradley-Terry model embraces uncertainty explicitly, it can recover a consistent underlying ranking even when individual comparisons contradict each other — disagreement among annotators becomes part of the model rather than a defect to eliminate. A pairwise framework paper on arXiv likewise notes that traditional RLHF pipelines train the reward model to assign scalar rewards by fitting a Bradley-Terry model on pairwise human preference comparisons. A useful side effect, as the Reinforced newsletter's Alex Nikulkov observes, is that a trained reward model can also rank multiple generations and pick the best one — the "best-of-n" technique — or serve for offline evaluation, provided it is trained on data that does not overlap with the LLM's training data.

The Three Stages of the RLHF Pipeline

The table below lays out the full pipeline as a diagnostic framework: each stage has a distinct purpose, its own key components, a defined data flow, and a concrete outcome. Reading down the "Data Flow" column shows why RLHF is a pipeline rather than a monolithic step — each stage consumes the previous stage's output.

The three-stage RLHF pipeline, from pretrained model to aligned policy
Stage	Purpose	Key Components	Data Flow	Outcome
1. Supervised fine-tuning	Give the pretrained model basic instruction-following ability	Pretrained LM; human-written prompt-response pairs; human labelers comparing generated responses	High-quality pairs train the model; the tuned model then generates multiple candidate responses for humans to rank	An SFT model plus a dataset of pairwise human preferences
2. Reward model training	Convert qualitative preferences into a numerical reward signal	A separate reward model; Bradley-Terry model fitted to pairwise comparisons via log-likelihood maximization	Pairwise preference data trains the reward model to output scalar scores for any response	A frozen scoring model that can evaluate responses offline, without humans in the loop
3. Policy optimization	Fine-tune the language model's policy against the learned reward	The SFT model as policy; reward model as reward function; an RL algorithm such as PPO	Policy generates responses, reward model scores them, RL algorithm updates the policy; the cycle repeats across iterations	An aligned language model whose outputs the reward model — and, by proxy, humans — score highly

The three-stage RLHF pipeline, from pretrained model to aligned policy

Stage

Purpose

Key Components

Data Flow

Outcome

1. Supervised fine-tuning

Give the pretrained model basic instruction-following ability

Pretrained LM; human-written prompt-response pairs; human labelers comparing generated responses

High-quality pairs train the model; the tuned model then generates multiple candidate responses for humans to rank

An SFT model plus a dataset of pairwise human preferences

2. Reward model training

Convert qualitative preferences into a numerical reward signal

A separate reward model; Bradley-Terry model fitted to pairwise comparisons via log-likelihood maximization

Pairwise preference data trains the reward model to output scalar scores for any response

A frozen scoring model that can evaluate responses offline, without humans in the loop

3. Policy optimization

Fine-tune the language model's policy against the learned reward

The SFT model as policy; reward model as reward function; an RL algorithm such as PPO

Policy generates responses, reward model scores them, RL algorithm updates the policy; the cycle repeats across iterations

An aligned language model whose outputs the reward model — and, by proxy, humans — score highly

Two qualifications belong beside this framework. First, SFT is optional in principle but empirically part of the best-performing combination. Second, the reward model is a proxy, not an oracle: it encodes the preferences of the specific labelers whose comparisons trained it, which is why reward-model quality is widely treated as the crucial design decision in the pipeline.

Key Open-Source Tools for RLHF Implementation

Understanding the pipeline is one thing; running it is another. The Hugging Face authors identify three primary open-source PyTorch repositories that grew out of practical RLHF work. Transformers Reinforcement Learning (TRL) is designed to fine-tune pretrained language models in the Hugging Face ecosystem using PPO — the natural starting point for developers already working in that ecosystem. TRLX, a fork of TRL built by CarperAI, extends the approach to larger models and supports both online and offline training; at the time of that writing, its API was capable of production-ready RLHF with PPO and Implicit Language Q-Learning (ILQL) at the scales required for LLM deployment, on the order of 33 billion parameters. Reinforcement Learning for Language models (RL4LMs) rounds out the set as an additional active framework.

These libraries matter because the pipeline's stages have very different engineering demands: data generation and reward-model inference are batch-scoring problems, while policy optimization involves the tight interplay of a generating model, a scoring model, and an update rule. A framework that handles that plumbing lets a developer focus on the parts that actually determine alignment quality — the preference data and the reward model.

How RLHF Works: The Three-Stage Pipeline That Turns Human Preferences Into a Fine-Tuned Language Model

Understanding Reinforcement Learning from Human Feedback (RLHF)

Stage 1: Supervised Fine-Tuning and Human Preference Data Collection

Tags

Stage 2: Training the Reward Model with Human Preferences

Stage 3: Policy Optimization via Reinforcement Learning

The Three Stages of the RLHF Pipeline

Key Open-Source Tools for RLHF Implementation

Implementing RLHF: Next Steps for Developers

Sources

Related articles

AI Agents Finally Made Formal Verification Practical — and Signal's Core Code Is the Proof

How AI Accelerates Scientific Discovery: Three Stages of a Faster Research Loop

How AI Models Accelerate Target Identification and Lead Optimization in Early Drug Discovery

Neuro-Symbolic AI Explained: How Hybrid Systems Combine Pattern Recognition with Logical Reasoning