Accepted · ACDSA 2027
EVALUATION / Accepted · ACDSA 2027
What does a reward model really learn?
Testing the gap between reward-score accuracy, policy agreement, and action correctness in language-conditioned navigation.
01 / MOTIVATION
The question
When an LLM judge learns to agree with a reference policy, has it learned to choose the right action? This project examines that distinction in a controlled 2-D navigation environment.
02 / METHOD
The approach
Four controlled experiments compare geometric rewards, an LLM judge, and a distilled reward network in a shared gridworld. Language instructions and iterative feedback are used to shape the judge’s decisions.
A reward network is trained on 258 LLM judgments, with evaluation on 48 held-out situations. The experiments assess both rubric-score error and agreement on selected actions.
03 / EVIDENCE
Findings & scope
Judge agreement with a rule-based policy increased from 78% to 100% across 64 situations. Feedback could also steer the judge toward an imperfect policy, demonstrating that reference agreement is different from action correctness.
The distilled network achieved 0.058 rubric-score MAE but only 48% action agreement on the 48 held-out situations. These results are specific to the controlled gridworld and do not establish performance on physical robots.
READ THE WORK