Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty

reinforcement learning with calibration rewards — training a reasoning model to output a confidence score alongside its answer, rewarded by a Brier score

Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, Jacob Andreas · ICLR 2026 · 2026

In one sentence

RLCR adds a Brier-score term to the binary correctness reward of reasoning RL, so models learn to emit a calibrated confidence estimate alongside their answer without sacrificing task accuracy.

Abstract

When language models (LMs) are trained via reinforcement learning (RL) to generate natural language"reasoning chains", their performance improves on a variety of difficult question answering tasks. Today, almost all successful applications of RL for reasoning use binary reward functions that evaluate the correctness of LM outputs. Because such reward functions do not penalize guessing or low-confidence outputs, they often have the unintended side-effect of degrading calibration and increasing the rate at which LMs generate incorrect responses (or"hallucinate") in other problem domains. This paper describes RLCR (Reinforcement Learning with Calibration Rewards), an approach to training reasoning models that jointly improves accuracy and calibrated confidence estimation. During RLCR, LMs generate both predictions and numerical confidence estimates after reasoning. They are trained to optimize a reward function that augments a binary correctness score with a Brier score -- a scoring rule for confidence estimates that incentivizes calibrated prediction. We first prove that this reward function (or any reward function that uses a bounded, proper scoring rule) yields models whose predictions are both accurate and well-calibrated. We next show that across diverse datasets, RLCR substantially improves calibration with no loss in accuracy, on both in-domain and out-of-domain evaluations -- outperforming both ordinary RL training and classifiers trained to assign post-hoc confidence scores. While ordinary RL hurts calibration, RLCR improves it. Finally, we demonstrate that verbalized confidence can be leveraged at test time to improve accuracy and calibration via confidence-weighted scaling methods. Our results show that explicitly optimizing for calibration can produce more generally reliable reasoning models. Code, models, and further info is available at https://rl-calibration.github.io/.

Questions this paper answers

Does adding a confidence-calibration reward to RL training hurt task accuracy?
Can a reasoning model be trained for accuracy and calibrated confidence at the same time?
Is there a tradeoff between correctness and calibration in RLCR?
A reward equal to a scaled correctness indicator minus a proper scoring rule provably incentivizes a calibrated confidence and the answer with the highest success probability. The guarantee requires the scoring rule's S(p,1)-S(p,0) to be bounded by the correctness weight lambda.
Holds for: Assumes the correctness indicator for a fixed prediction is Bernoulli and that confidence can be chosen freely. Brier satisfies the bound at lambda=1; the log score does not, since S(p,1)-S(p,0) diverges as p approaches 0.
On in-distribution HotpotQA, RLCR cuts expected calibration error from 0.37 to 0.03 and Brier score from 0.37 to 0.21 relative to RLVR, at 62.1% versus 63.0% accuracy.
Holds for: Qwen2.5-7B base, GRPO with no KL regularization, trained on 20,000 HotpotQA-Modified examples; evaluated at temperature 0 with exact-match correctness.
Trained on Big-Math, RLCR reduces in-domain expected calibration error from 0.26 to 0.10 versus RLVR while matching accuracy (72.7% vs 72.9%) averaged over MATH-500, GSM8K and Big-Math.
Holds for: Qwen2.5-7B base; Big-Math subset of 15,000 numerical-answer problems with LLaMA-8B solve rate between 0 and 70%; correctness scored by math-verify.
Why use the Brier score instead of log loss as a calibration reward?
What goes wrong if you reward verbalized confidence with the logarithmic scoring rule?
Which proper scoring rules are safe to combine with a correctness reward?
A reward equal to a scaled correctness indicator minus a proper scoring rule provably incentivizes a calibrated confidence and the answer with the highest success probability. The guarantee requires the scoring rule's S(p,1)-S(p,0) to be bounded by the correctness weight lambda.
Holds for: Assumes the correctness indicator for a fixed prediction is Bernoulli and that confidence can be chosen freely. Brier satisfies the bound at lambda=1; the log score does not, since S(p,1)-S(p,0) diverges as p approaches 0.
Replacing the Brier term with the unbounded log score collapses training on a 5-arm next-draw prediction task: RLCR-Log outputs the invalid arm at confidence 0 and reaches 0 accuracy. RLCR-Brier on the same task reaches 34.4% accuracy at ECE 0.02.
Holds for: Qwen2.5-7B trained on 10,000 synthetic toy-arm examples with high aleatoric uncertainty and no uncertainty reasoning. On HotpotQA the same substitution does not collapse: RLCR-Log reaches 59.5% accuracy and ECE 0.07 (Table 3).
Does standard RL with verifiable rewards make language models overconfident?
What happens to calibration when a model is trained with binary correctness rewards?
Does RLVR degrade calibration on tasks it was not trained on?
Averaged over 6 out-of-distribution datasets, RLVR training on HotpotQA worsens Brier score from the base model's 0.41 to 0.46, whereas RLCR improves it to 0.21 and raises AUROC from 0.50 to 0.68.
Holds for: Qwen2.5-7B trained only on HotpotQA-Modified; OOD set is TriviaQA, SimpleQA, MATH-500, GSM8K, CommonsenseQA and GPQA. OOD accuracy is roughly flat for all RL methods.
Prompting an RLVR model to reason about uncertainty at test time lowers HotpotQA ECE only from 0.37 to 0.34, whereas training with the calibration reward reaches 0.03. Calibration-aware reward matters far more than the uncertainty-reasoning prompt.
Holds for: HotpotQA-trained Qwen2.5-7B models; RLVR w/ Analysis uses the identical analysis prompt RLCR was trained with. Both components help: OOD Brier is 0.41 for RLVR w/ Analysis versus 0.46 for plain RLVR.
Is training a separate confidence classifier as good as training a reasoning LM to verbalize its own confidence?
How does RLCR compare to post-hoc confidence heads and linear probes?
Do I need a second model to get calibrated confidence from a reasoning LM?
RLCR reaches out-of-distribution Brier score 0.21 with a single model, against 0.27 for a same-size BCE confidence classifier and 0.32 for a Brier-loss classifier. A linear probe on RLVR embeddings reaches 0.38 and answer-token probability 0.42.
Holds for: Classifiers and probe are trained on RLVR outputs from Qwen2.5-7B on HotpotQA-Modified and share the RLVR generator, so their accuracies equal RLVR's. In-distribution the BCE classifier is close (ECE 0.07 versus RLCR's 0.03).
Optimizing a Brier reward alone over the whole generation collapses a HotpotQA reasoning model to 0.00 accuracy, with empty or trivial answers at confidence 0. Restricting that reward to the analysis and confidence spans preserves 62.0% accuracy but leaves OOD Brier at 0.27 versus RLCR's 0.21.
Holds for: Calibration-only baselines are initialized from the RLVR model, not the base model, and use no KL regularization. An abstention-RL baseline with reward 0.5 for abstaining reaches OOD Brier 0.35.
Can a model's own stated confidence be used for best-of-N or majority voting?
Does confidence-weighted majority voting beat plain majority voting?
Is verbalized confidence useful for test-time scaling without a reward model?
Weighting majority votes by an RLCR model's own verbalized confidence yields higher accuracy than plain majority vote, max-confidence selection, and two generation-likelihood baselines as the number of samples grows.
Holds for: RLCR trained on HotpotQA, accuracy averaged over the 7 evaluation datasets of Table 1; needs no external reward model or extra supervision.
Is it enough to just prompt a model to reason about its uncertainty?
Does the calibration gain come from the reward or from the uncertainty-reasoning prompt?
How much of RLCR's calibration improvement survives if the uncertainty analysis is removed?
Prompting an RLVR model to reason about uncertainty at test time lowers HotpotQA ECE only from 0.37 to 0.34, whereas training with the calibration reward reaches 0.03. Calibration-aware reward matters far more than the uncertainty-reasoning prompt.
Holds for: HotpotQA-trained Qwen2.5-7B models; RLVR w/ Analysis uses the identical analysis prompt RLCR was trained with. Both components help: OOD Brier is 0.41 for RLVR w/ Analysis versus 0.46 for plain RLVR.
RLCR without any uncertainty analysis in the chain-of-thought matches RLVR's accuracy (61.7% vs 63.0%) and token cost (113 vs 92 tokens) on HotpotQA while cutting ECE from 0.37 to 0.09.
Holds for: Qwen2.5-7B trained on HotpotQA-Modified, evaluated with the analysis section removed at inference. Full RLCR with analysis is still better calibrated (ECE 0.03) at 249 tokens.
Does RLCR add many extra tokens to each response?
Is there a cheap version of calibrated-confidence RL with RLVR-like token cost?
RLCR without any uncertainty analysis in the chain-of-thought matches RLVR's accuracy (61.7% vs 63.0%) and token cost (113 vs 92 tokens) on HotpotQA while cutting ECE from 0.37 to 0.09.
Holds for: Qwen2.5-7B trained on HotpotQA-Modified, evaluated with the analysis section removed at inference. Full RLCR with analysis is still better calibrated (ECE 0.03) at 249 tokens.
Are verbalized confidence scores from an RLCR model stable across samples?
Does an RLCR-trained model give consistent confidence when the same answer is re-analysed?
Do confidences over competing answers to one question sum to 1?
Resampling multiple uncertainty-reasoning chains for the same answer from an RLCR model gives confidence scores with mostly low standard deviation, so the model has little 'uncertainty about its uncertainty'.
Holds for: RLCR trained on HotpotQA, standard deviations over analysis chains for a fixed solution across 7 datasets. Variability is higher when full chains, and hence answers, are resampled.
RLCR's mean confidences over distinct sampled answers sum to close to the ideal 1 on in-distribution HotpotQA, but exceed 1 out of distribution, so overconfidence on contradictory answers persists.
Holds for: Three representative datasets, HotpotQA-trained Qwen2.5-7B; questions with mutually exclusive answers only. RLCR's sums are closer to 1 than RLVR's in every case shown.
Does the calibration reward work on model families other than Qwen2.5?
Has RLCR been tested on OlMo-2 or Qwen3?
Do the calibration gains transfer across base models?
The accuracy-neutral calibration gain of RLCR reproduces on OlMo-2-7B-Instruct (HotpotQA ECE 0.38 for RLVR versus 0.09 for RLCR) and on Qwen-3-8B (OOD Brier 0.28 versus 0.17).
Holds for: Both trained on HotpotQA-Modified with the main GRPO setup; accuracy within about 1 point of RLVR in each case.
What should I read about training language models to express calibrated uncertainty?
Which paper introduced proper scoring rules as an RL reward for reasoning models?
Where should I start reading on hallucination and calibration in RL-trained reasoning models?
What work established that calibration can be optimized jointly with correctness in RLVR?
RLCR shows that RL for reasoning can optimize calibration alongside correctness by adding a proper-scoring-rule term to the binary correctness reward. Confidence is learned by the reasoning model itself rather than fitted post hoc by a separate model.
Holds for: As of the 2026 publication; earlier RL-for-calibration work optimized calibration alone and was evaluated on non-reasoning tasks. Demonstrated on 7B-8B open models.
What happens if you optimize only a calibration reward without any correctness term?
Do calibration-only RL methods reward hack?
How does RLCR compare to abstention-style ternary rewards?
Optimizing a Brier reward alone over the whole generation collapses a HotpotQA reasoning model to 0.00 accuracy, with empty or trivial answers at confidence 0. Restricting that reward to the analysis and confidence spans preserves 62.0% accuracy but leaves OOD Brier at 0.27 versus RLCR's 0.21.
Holds for: Calibration-only baselines are initialized from the RLVR model, not the base model, and use no KL regularization. An abstention-RL baseline with reward 0.5 for abstaining reaches OOD Brier 0.35.
Replacing the Brier term with the unbounded log score collapses training on a 5-arm next-draw prediction task: RLCR-Log outputs the invalid arm at confidence 0 and reaches 0 accuracy. RLCR-Brier on the same task reaches 34.4% accuracy at ECE 0.02.
Holds for: Qwen2.5-7B trained on 10,000 synthetic toy-arm examples with high aleatoric uncertainty and no uncertainty reasoning. On HotpotQA the same substitution does not collapse: RLCR-Log reaches 59.5% accuracy and ECE 0.07 (Table 3).
Is out-of-domain calibration actually solved by adding a Brier reward?
How large is the remaining calibration error after RLCR on unseen tasks?
Averaged over 6 out-of-distribution datasets, RLVR training on HotpotQA worsens Brier score from the base model's 0.41 to 0.46, whereas RLCR improves it to 0.21 and raises AUROC from 0.50 to 0.68.
Holds for: Qwen2.5-7B trained only on HotpotQA-Modified; OOD set is TriviaQA, SimpleQA, MATH-500, GSM8K, CommonsenseQA and GPQA. OOD accuracy is roughly flat for all RL methods.
RLCR's mean confidences over distinct sampled answers sum to close to the ideal 1 on in-distribution HotpotQA, but exceed 1 out of distribution, so overconfidence on contradictory answers persists.
Holds for: Three representative datasets, HotpotQA-trained Qwen2.5-7B; questions with mutually exclusive answers only. RLCR's sums are closer to 1 than RLVR's in every case shown.

Claims and scope

Common misreadings

Terminology in this paper

RLCR reward
The sum of a binary correctness indicator and a negative Brier score on the model's verbalized confidence, used as the RL reward for a reasoning model that outputs both an answer and a confidence in [0,1].
RLVR
Reinforcement learning with verifiable rewards: RL on reasoning chains using only a binary indicator of whether the final answer matches the ground truth.
uncertainty analysis
A segment of the chain-of-thought, emitted in <analysis> tags after the answer, in which the model enumerates specific ways its own solution could be wrong before stating a numerical confidence.
confidence-weighted majority vote
Test-time aggregation over N sampled responses in which each vote is weighted by the model's own verbalized confidence rather than by an external reward model.
inter-solution consistency
The property that, when a model samples several answers to a mutually exclusive question, the mean confidences assigned to the distinct answers sum to at most 1.

How to cite

@inproceedings{damani2026beyond,
title={Beyond Binary Rewards: Training {LM}s to Reason About Their Uncertainty},
author={Damani, Mehul and Puri, Isha and Slocum, Stewart and Shenfeld, Idan and Choshen, Leshem and Kim, Yoon and Andreas, Jacob},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=ASQ649zdHm}
}

References

See the full reference list in the paper.