Reward Function

A reward function is the scoring rule that tells a reinforcement-learning system how good an action, output, or completed episode is. In language-model training, it converts generated behavior into a reward signal used to update an agent policy or model.

Reward functions can be:

• verifiable, such as tests passing, an answer matching ground truth, or a proof checker accepting a result;
• learned, such as a reward model trained from human preferences;
• rubric-based, where several criteria are scored separately and combined;
• sparse, with only a final task reward; or
• dense, with intermediate rewards at multiple steps.

The reward function is central to Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR). In RLHF, the reward is often predicted by a learned reward model. In RLVR, the reward is computed by an external verifier.

A reward function is not the same as an evaluation. Evaluation measures system behavior, often for reporting or regression testing. A reward function is used as an optimization target during training or search. The same metric can sometimes serve both roles, but the incentives are different.

The main risk is reward hacking. A model may find behavior that maximizes the reward while violating the intended task. For example, code may pass visible tests while being brittle, insecure, or incomplete. A helpful-looking answer may satisfy a weak judge while fabricating facts.

Good reward design requires clear task specification, hidden test cases, adversarial checks, calibration against human judgment when relevant, and independent post-training evaluation.

Hugging Face's agent glossary describes reward as the score used by a trainer to update a model's weights during agent training.