Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty

reinforcement learning with calibration rewards — training a reasoning model to output a confidence score alongside its answer, rewarded by a Brier score

Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, Jacob Andreas · ICLR 2026 · 2026

In one sentence

RLCR adds a Brier-score term to the binary correctness reward of reasoning RL, so models learn to emit a calibrated confidence estimate alongside their answer without sacrificing task accuracy.

Abstract

When language models (LMs) are trained via reinforcement learning (RL) to generate natural language"reasoning chains", their performance improves on a variety of difficult question answering tasks. Today, almost all successful applications of RL for reasoning use binary reward functions that evaluate the correctness of LM outputs. Because such reward functions do not penalize guessing or low-confidence outputs, they often have the unintended side-effect of degrading calibration and increasing the rate at which LMs generate incorrect responses (or"hallucinate") in other problem domains. This paper describes RLCR (Reinforcement Learning with Calibration Rewards), an approach to training reasoning models that jointly improves accuracy and calibrated confidence estimation. During RLCR, LMs generate both predictions and numerical confidence estimates after reasoning. They are trained to optimize a reward function that augments a binary correctness score with a Brier score -- a scoring rule for confidence estimates that incentivizes calibrated prediction. We first prove that this reward function (or any reward function that uses a bounded, proper scoring rule) yields models whose predictions are both accurate and well-calibrated. We next show that across diverse datasets, RLCR substantially improves calibration with no loss in accuracy, on both in-domain and out-of-domain evaluations -- outperforming both ordinary RL training and classifiers trained to assign post-hoc confidence scores. While ordinary RL hurts calibration, RLCR improves it. Finally, we demonstrate that verbalized confidence can be leveraged at test time to improve accuracy and calibration via confidence-weighted scaling methods. Our results show that explicitly optimizing for calibration can produce more generally reliable reasoning models. Code, models, and further info is available at https://rl-calibration.github.io/.

Questions this paper answers

if a model is trained to say how sure it is, does it get worse at getting answers right?
does adding a proper-scoring-rule calibration term to a binary correctness reward trade off task accuracy against calibration error?
how do I train a reasoning model to output a confidence score without losing accuracy on the task itself?
should I worry about losing accuracy if I add a confidence reward to my RL fine-tuning run?
A reward equal to a scaled correctness indicator minus a proper scoring rule provably incentivizes a calibrated confidence and the answer with the highest success probability. The guarantee requires the scoring rule's S(p,1)-S(p,0) to be bounded by the correctness weight lambda.
Holds for: Assumes the correctness indicator for a fixed prediction is Bernoulli and that confidence can be chosen freely. Brier satisfies the bound at lambda=1; the log score does not, since S(p,1)-S(p,0) diverges as p approaches 0.
On in-distribution HotpotQA, RLCR cuts expected calibration error from 0.37 to 0.03 and Brier score from 0.37 to 0.21 relative to RLVR, at 62.1% versus 63.0% accuracy.
Holds for: Qwen2.5-7B base, GRPO with no KL regularization, trained on 20,000 HotpotQA-Modified examples; evaluated at temperature 0 with exact-match correctness.
Trained on Big-Math, RLCR reduces in-domain expected calibration error from 0.26 to 0.10 versus RLVR while matching accuracy (72.7% vs 72.9%) averaged over MATH-500, GSM8K and Big-Math.
Holds for: Qwen2.5-7B base; Big-Math subset of 15,000 numerical-answer problems with LLaMA-8B solve rate between 0 and 70%; correctness scored by math-verify.
which formula for scoring a stated confidence is safe to use as a training reward, and which one breaks?
why is a bounded Brier term preferred over the logarithmic scoring rule when combined with a correctness reward in RL?
which scoring rule should I pick for the confidence part of my reward so training does not degenerate?
can I just use log loss on the verbalized confidence as my calibration reward, or will that blow up?
A reward equal to a scaled correctness indicator minus a proper scoring rule provably incentivizes a calibrated confidence and the answer with the highest success probability. The guarantee requires the scoring rule's S(p,1)-S(p,0) to be bounded by the correctness weight lambda.
Holds for: Assumes the correctness indicator for a fixed prediction is Bernoulli and that confidence can be chosen freely. Brier satisfies the bound at lambda=1; the log score does not, since S(p,1)-S(p,0) diverges as p approaches 0.
Replacing the Brier term with the unbounded log score collapses training on a 5-arm next-draw prediction task: RLCR-Log outputs the invalid arm at confidence 0 and reaches 0 accuracy. RLCR-Brier on the same task reaches 34.4% accuracy at ECE 0.02.
Holds for: Qwen2.5-7B trained on 10,000 synthetic toy-arm examples with high aleatoric uncertainty and no uncertainty reasoning. On HotpotQA the same substitution does not collapse: RLCR-Log reaches 59.5% accuracy and ECE 0.07 (Table 3).
does reinforcement learning on right-or-wrong answers make a model more overconfident?
how does RLVR with binary verifiable rewards affect Brier score and AUROC, including on datasets outside the training distribution?
how do I tell whether my RL-tuned reasoning model became less reliable at judging its own answers on new tasks?
my model was RL-trained on correctness only, should I expect its confidence estimates to be trustworthy off-distribution?
Averaged over 6 out-of-distribution datasets, RLVR training on HotpotQA worsens Brier score from the base model's 0.41 to 0.46, whereas RLCR improves it to 0.21 and raises AUROC from 0.50 to 0.68.
Holds for: Qwen2.5-7B trained only on HotpotQA-Modified; OOD set is TriviaQA, SimpleQA, MATH-500, GSM8K, CommonsenseQA and GPQA. OOD accuracy is roughly flat for all RL methods.
Prompting an RLVR model to reason about uncertainty at test time lowers HotpotQA ECE only from 0.37 to 0.34, whereas training with the calibration reward reaches 0.03. Calibration-aware reward matters far more than the uncertainty-reasoning prompt.
Holds for: HotpotQA-trained Qwen2.5-7B models; RLVR w/ Analysis uses the identical analysis prompt RLCR was trained with. Both components help: OOD Brier is 0.41 for RLVR w/ Analysis versus 0.46 for plain RLVR.
is it better to have the model itself state how sure it is, or to bolt a separate confidence predictor on top?
how does verbalized confidence trained with a calibration-augmented RL reward compare with post-hoc confidence heads, linear probes on hidden states, and answer-token likelihood?
how do I get calibrated uncertainty for a reasoning model without training and serving a second scoring model?
should I train a separate confidence classifier for my reasoning model or train the model to report confidence itself?
RLCR reaches out-of-distribution Brier score 0.21 with a single model, against 0.27 for a same-size BCE confidence classifier and 0.32 for a Brier-loss classifier. A linear probe on RLVR embeddings reaches 0.38 and answer-token probability 0.42.
Holds for: Classifiers and probe are trained on RLVR outputs from Qwen2.5-7B on HotpotQA-Modified and share the RLVR generator, so their accuracies equal RLVR's. In-distribution the BCE classifier is close (ECE 0.07 versus RLCR's 0.03).
Optimizing a Brier reward alone over the whole generation collapses a HotpotQA reasoning model to 0.00 accuracy, with empty or trivial answers at confidence 0. Restricting that reward to the analysis and confidence spans preserves 62.0% accuracy but leaves OOD Brier at 0.27 versus RLCR's 0.21.
Holds for: Calibration-only baselines are initialized from the RLVR model, not the base model, and use no KL regularization. An abstention-RL baseline with reward 0.5 for abstaining reaches OOD Brier 0.35.
can a model's own stated confidence be used to pick among several sampled answers?
does weighting majority voting by verbalized confidence outperform unweighted self-consistency and generation-likelihood weighting as sample count grows?
how do I aggregate multiple sampled answers at inference time without running a reward model or verifier?
is my model's stated confidence good enough to weight best-of-N or voting at test time?
Weighting majority votes by an RLCR model's own verbalized confidence yields higher accuracy than plain majority vote, max-confidence selection, and two generation-likelihood baselines as the number of samples grows.
Holds for: RLCR trained on HotpotQA, accuracy averaged over the 7 evaluation datasets of Table 1; needs no external reward model or extra supervision.
is asking a model to think about how unsure it is enough, or does the training reward have to change?
is the calibration improvement attributable to the calibration-augmented reward or to uncertainty reasoning in the chain of thought?
how do I improve my model's confidence estimates - change the prompt to reason about uncertainty, or change the training objective?
can I skip retraining and just prompt my reasoning model to assess its own uncertainty?
Prompting an RLVR model to reason about uncertainty at test time lowers HotpotQA ECE only from 0.37 to 0.34, whereas training with the calibration reward reaches 0.03. Calibration-aware reward matters far more than the uncertainty-reasoning prompt.
Holds for: HotpotQA-trained Qwen2.5-7B models; RLVR w/ Analysis uses the identical analysis prompt RLCR was trained with. Both components help: OOD Brier is 0.41 for RLVR w/ Analysis versus 0.46 for plain RLVR.
RLCR without any uncertainty analysis in the chain-of-thought matches RLVR's accuracy (61.7% vs 63.0%) and token cost (113 vs 92 tokens) on HotpotQA while cutting ECE from 0.37 to 0.09.
Holds for: Qwen2.5-7B trained on HotpotQA-Modified, evaluated with the analysis section removed at inference. Full RLCR with analysis is still better calibrated (ECE 0.03) at 249 tokens.
does making a model report calibrated confidence make its answers much longer?
what is the token overhead of calibration-trained verbalized confidence relative to a correctness-only RL baseline, and does dropping the uncertainty analysis retain the calibration gain?
how do I add calibrated confidence output without increasing inference cost per response?
will adding confidence training blow up my generation length and serving cost?
RLCR without any uncertainty analysis in the chain-of-thought matches RLVR's accuracy (61.7% vs 63.0%) and token cost (113 vs 92 tokens) on HotpotQA while cutting ECE from 0.37 to 0.09.
Holds for: Qwen2.5-7B trained on HotpotQA-Modified, evaluated with the analysis section removed at inference. Full RLCR with analysis is still better calibrated (ECE 0.03) at 249 tokens.
if you ask a model the same question twice, does it give the same confidence number?
how stable are verbalized confidence scores across resampled uncertainty-reasoning chains, and do mean confidences over distinct sampled answers sum to 1?
how do I check whether the confidence numbers my model emits are reproducible enough to threshold on?
can I rely on a single sampled confidence score from my model, or do I need to average several?
Resampling multiple uncertainty-reasoning chains for the same answer from an RLCR model gives confidence scores with mostly low standard deviation, so the model has little 'uncertainty about its uncertainty'.
Holds for: RLCR trained on HotpotQA, standard deviations over analysis chains for a fixed solution across 7 datasets. Variability is higher when full chains, and hence answers, are resampled.
RLCR's mean confidences over distinct sampled answers sum to close to the ideal 1 on in-distribution HotpotQA, but exceed 1 out of distribution, so overconfidence on contradictory answers persists.
Holds for: Three representative datasets, HotpotQA-trained Qwen2.5-7B; questions with mutually exclusive answers only. RLCR's sums are closer to 1 than RLVR's in every case shown.
do the confidence gains show up on more than one base model, or just the one it was developed on?
does the accuracy-neutral calibration improvement from a Brier-augmented RL reward replicate across base model families such as OlMo-2-7B-Instruct and Qwen-3-8B?
how do I know whether calibration-augmented RL will work on the base model I actually use?
my base model is not Qwen2.5, will calibration training still help me?
The accuracy-neutral calibration gain of RLCR reproduces on OlMo-2-7B-Instruct (HotpotQA ECE 0.38 for RLVR versus 0.09 for RLCR) and on Qwen-3-8B (OOD Brier 0.28 versus 0.17).
Holds for: Both trained on HotpotQA-Modified with the main GRPO setup; accuracy within about 1 point of RLVR in each case.
what should I read first about getting reinforcement-learned reasoning models to know when they are wrong?
what work established that calibration can be optimized jointly with correctness in RL with verifiable rewards using a proper scoring rule?
where do I start reading about training reasoning models to report reliable confidence instead of fitting confidence after the fact?
RLCR shows that RL for reasoning can optimize calibration alongside correctness by adding a proper-scoring-rule term to the binary correctness reward. Confidence is learned by the reasoning model itself rather than fitted post hoc by a separate model.
Holds for: As of the 2026 publication; earlier RL-for-calibration work optimized calibration alone and was evaluated on non-reasoning tasks. Demonstrated on 7B-8B open models.
what happens if a model is rewarded only for accurate confidence and not for being right?
does optimizing a calibration reward alone over the whole generation induce reward hacking such as degenerate answers at confidence 0?
how do I set up a confidence reward so the model cannot game it by giving up on the answer?
can I drop the correctness term and train on the calibration reward alone?
Optimizing a Brier reward alone over the whole generation collapses a HotpotQA reasoning model to 0.00 accuracy, with empty or trivial answers at confidence 0. Restricting that reward to the analysis and confidence spans preserves 62.0% accuracy but leaves OOD Brier at 0.27 versus RLCR's 0.21.
Holds for: Calibration-only baselines are initialized from the RLVR model, not the base model, and use no KL regularization. An abstention-RL baseline with reward 0.5 for abstaining reaches OOD Brier 0.35.
Replacing the Brier term with the unbounded log score collapses training on a 5-arm next-draw prediction task: RLCR-Log outputs the invalid arm at confidence 0 and reaches 0 accuracy. RLCR-Brier on the same task reaches 34.4% accuracy at ECE 0.02.
Holds for: Qwen2.5-7B trained on 10,000 synthetic toy-arm examples with high aleatoric uncertainty and no uncertainty reasoning. On HotpotQA the same substitution does not collapse: RLCR-Log reaches 59.5% accuracy and ECE 0.07 (Table 3).
after training a model to report confidence, how much overconfidence is left on tasks it never saw?
how much residual calibration error and cross-answer confidence mass remains out of distribution after Brier-augmented RL training?
how do I estimate how far I can trust confidence scores on domains outside the training distribution?
is out-of-domain calibration good enough after this kind of training that I can act on the confidence numbers?
Averaged over 6 out-of-distribution datasets, RLVR training on HotpotQA worsens Brier score from the base model's 0.41 to 0.46, whereas RLCR improves it to 0.21 and raises AUROC from 0.50 to 0.68.
Holds for: Qwen2.5-7B trained only on HotpotQA-Modified; OOD set is TriviaQA, SimpleQA, MATH-500, GSM8K, CommonsenseQA and GPQA. OOD accuracy is roughly flat for all RL methods.
RLCR's mean confidences over distinct sampled answers sum to close to the ideal 1 on in-distribution HotpotQA, but exceed 1 out of distribution, so overconfidence on contradictory answers persists.
Holds for: Three representative datasets, HotpotQA-trained Qwen2.5-7B; questions with mutually exclusive answers only. RLCR's sums are closer to 1 than RLVR's in every case shown.

Claims and scope

Common misreadings

Terminology in this paper

RLCR reward
The sum of a binary correctness indicator and a negative Brier score on the model's verbalized confidence, used as the RL reward for a reasoning model that outputs both an answer and a confidence in [0,1].
RLVR
Reinforcement learning with verifiable rewards: RL on reasoning chains using only a binary indicator of whether the final answer matches the ground truth.
uncertainty analysis
A segment of the chain-of-thought, emitted in <analysis> tags after the answer, in which the model enumerates specific ways its own solution could be wrong before stating a numerical confidence.
confidence-weighted majority vote
Test-time aggregation over N sampled responses in which each vote is weighted by the model's own verbalized confidence rather than by an external reward model.
inter-solution consistency
The property that, when a model samples several answers to a mutually exclusive question, the mean confidences assigned to the distinct answers sum to at most 1.

How to cite

@inproceedings{damani2026beyond,
title={Beyond Binary Rewards: Training {LM}s to Reason About Their Uncertainty},
author={Damani, Mehul and Puri, Isha and Slocum, Stewart and Shenfeld, Idan and Choshen, Leshem and Kim, Yoon and Andreas, Jacob},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=ASQ649zdHm}
}

References

See the full reference list in the paper.