← Back to Publications

When Confidence Proxies Confound Reasoning Complexity: Pitfalls of Uncertainty-Weighted Credit Assignment in Language Model Reinforcement Learning

Sham Satish Thakare

Abstract

In language model reinforcement learning (RLVR/GRPO), weighting policy gradients by outcome confidence proxies has been proposed to focus updates on uncertain steps. In this paper, we expose length confounding in token predictive entropy ($r = +0.486$), mean NLL ($r = +0.432$), and logit margin ($r = +0.495$). We demonstrate that sample-level uncertainty weighting (CA-GRPO) yields 0.00% Pass@1 gain over standard outcome-supervised GRPO across 3 matched seeds, misidentifying correct complex derivations as uncertain in 42.1% of cases.

Artifacts & Code

BibTeX

@article{thakare2026confidence,
  author  = {Thakare, Sham Satish},
  title   = {When Confidence Proxies Confound Reasoning Complexity: Pitfalls of Uncertainty-Weighted Credit Assignment in Language Model Reinforcement Learning},
  journal = {IEEE Transactions on Artificial Intelligence},
  year    = {2026},
  note    = {Under Review, Manuscript ID: TAI-2026-Aug-A-01878},
  url     = {https://github.com/shamddd/ear_grpo_reasoning}
}