Abstract: Systems and methods described herein relate to dynamics-aware comparison of reward functions. One embodiment generates a reference reward function; computes a dynamics-aware transformation of the reference reward function based on a transition model of an environment of a robot; computes a dynamics-aware transformation of a first candidate reward function based on the transition model; computes a dynamics-aware transformation of a second candidate reward function based on the transition model; selects, as a final reward function, the first or second candidate reward function based on which is closer to the reference reward function as measured by pseudometrics computed between their respective dynamics-aware transformations and the dynamics-aware transformation of the reference reward function; and optimizes the final reward function to control, at least in part, operation of the robot.