Abstract: Deep learning inference for imaging systems includes an imaging device comprising an image capture component and a vision processing unit configured to process the image through a first trained inference network to determine a first inference result. A host system trains a neural network for image classification and produces and transmits the first trained inference network to the vision processing unit of the imaging device. The first trained inference network provides stand-alone image classification, an object detection and/or a confidence score to the imaging device. The vision processing unit may further comprise a two or more trained inference networks configured to receive an inference result as an input and output a second inference result. A stereo camera system uses features and/or results from one or more inference network to control processing of three-dimensional data for the detected object.