Abstract: A method of operating a self-driving vehicle is provided. The method includes predicting a trajectory of an agent in a vicinity of the self-driving vehicle. This is done by: receiving sensor data indicative of a current situation in the vicinity of the self-driving vehicle; generating a current situation feature vector based, at least in part, on the sensor data; searching a pre-built bank of library feature vectors to identify a result feature vector from the pre-built bank that is most relevant to the current situation feature vector, wherein each library feature vector in the pre-built bank is associated with an observed trajectory; and predicting the trajectory of the agent based, at least in part, on the result feature vector. The method also includes planning an action of the vehicle based, at least in part, on the predicted agent trajectory, and operating the vehicle according to the planned action.