Abstract: Navigation based on internal state inference and interactivity estimation may include training a policy for autonomous navigation by extracting spatio-temporal features from one or more historical observations of one or more agents within a simulation environment including an ego-agent, analyzing the spatio-temporal features to infer one or more internal states of one or more of the agents, predicting one or more future behaviors for one or more of the one or more of the agents in a first scenario including an existence of the ego-agent within the simulation environment and in a second scenario excluding the existence of the ego-agent within the simulation environment, and calculating one or more interactivity scores for one or more of the agents based on a difference between the first scenario and the second scenario. The trained policy may be implemented to control an autonomous vehicle.