Abstract: A framework for safely operating autonomous machinery, such as vehicles and other heavy equipment, in an in-field or off-road environment, includes detecting, identifying, and tracking objects from on-board sensors configured with the autonomous machinery as it performs activities in either an agricultural setting or a transportation environment. The framework generates commands for navigational control of autonomously-operated vehicles in response to detected objects and predicted tracks thereof for safe operation in the performance of those activities. The framework processes image data and range data in multiple fields of view around the autonomously-operated to discern and track objects in a deep learning to accurately interpret this data for determining and effecting such navigational control.