Language models & chatbots
RLHF is a training step where human raters compare a model's answers and the model is then adjusted to produce the kind of replies people prefer.
A pretrained model can write anything, including rude, rambling, or dangerous text. RLHF teaches it manners and helpfulness. Human raters are shown two or more replies to the same prompt and pick the better one. Those choices train a second model, a reward model, that learns to predict what people would prefer.
Then reinforcement learning kicks in. The main model generates replies, the reward model scores them, and the main model is nudged toward higher-scoring behavior, thousands of times. The result answers questions directly, follows instructions, and declines harmful requests, because those behaviors earned higher ratings.
RLHF was central to InstructGPT and ChatGPT in 2022 and is now standard. It has side effects: models can become too agreeable or overly cautious, because raters rewarded those traits. Newer variants use AI feedback or skip the reward model, but the core idea, learning from preferences, remains.
A rater sees two answers to 'how do I fix a flat tire,' one clear and one confusing, and clicks the clear one; millions of such clicks shape the model.