Abstract: Generally discussed herein are devices, systems, and methods for determining a minimal controllable latent state and operating a model trained to implement the minimal controllable latent state. A method can include receiving first and second observations of a temporal series of observations produced by a sensor, the second observation separated from the first observation by more than one other observation, the first observation associated with a first action performed by an agent responsive to the first observation, encoding, by an encoder, the first and second observations into respective first and second hidden state representations, combining the first and second hidden state representations resulting in a combined representation, operating, by an action predictor, on the combined representation resulting in a second action, and altering weights of the encoder and the action predictor based on a difference between the first and second actions.