Notes
What I took away from the papers on my reading list.
2026
- Goal Misgeneralization: Why Correct Specifications Aren't Enough for Correct Goals
(1) Specification Gaming: The post defines specification gaming as behavior that satisfies the literal objective without achieving the designer's intended outcome, for example the agent that flips the red Lego block instead of stacking it. It argues this comes from misspecified tasks (poor reward shaping, missed criteria, flawed human feedback in RLHF, simulator bugs), not…
- Specification Gaming: The Flip Side of AI ingenuity
Specification Gaming behaviour that satisfies the literal specification of an objective without achieving the intended outcome. Eg: Kind Midas and his golden touch basically while trying to learn the objective and gaming to find the reaward, it will find loopholes and exploit the task specification. RL algorithm agent can find a shortcut to getting lots…
- Key Concepts in RL
RL formalizes learning by trial and error. An agent sees a state or partial observation, acts according to a policy, and collects rewards along a trajectory, and the goal is a policy that maximizes expected return $J(\pi)$. Value functions ($V$, $Q$), which satisfy Bellman equations, and the advantage $A = Q - V$ are the…
- A Reduction of Imitation Learning and Sturcture Prediction to No-Regret Online Learning
The core idea that the paper addresses how to train a policy by imitating an expert so that it still performs well once it acts on its own and reaches situations the expert never demonstrated. The paper proposes DAgger, an iterative imitation learning algorithm that repeatedly runs the current policy, has the expert label the…
- Behavioural Cloning from Observation
BCO imitates from state-only demonstrations by fitting an inverse dynamics model on transitions that the agent generates from a random policy and uses that model to label the demonstrated state and clone the recovered pairs. The paper also talks about environment interactions into what happens before the demonstrations arrives and after , showing that that…