Abstract: A stack is described. The stack includes a rule engine. The rule engine is accessed via application programming interface (API) from any plug-in mission service. The rule engine includes rules and facts. The rules and facts are loaded separately from the API and from a cloudlet monitor. The rules and facts do not require source code updates to change. The stack is executable on processors of a node within a cloudlet. The cloudlet includes multiple of the nodes. The stack divides high-level tasks into low-level tasks which are individually executable by the nodes in the cloudlet.

Abstract: A mixed reality (MR) system is disclosed. The MR system may determine a first predicted head pose corresponding to a time that virtual reality imagery is rendered, determine a second predicted head pose corresponding to a selected point in time during a camera shutter period, and combine the virtual reality imagery with the stereoscopic imagery based on the first predicted head pose and the second predicted head pose. A simulator that employs remote (e.g., cloud) rendering is also disclosed. The simulator/client device may determine a first pose (e.g., vehicle pose and/or head pose), receive video imagery rendered by a remote server based on the first pose, and apply a timewarp correction to the video imagery based on a comparison of the first pose and a second pose.

Abstract: An augmented reality (AR) vision system is disclosed. A display is configured to present a surrounding environment to eyes of a user of the AR vision system. A depth tracker is configured to produce a measurement of a focal depth of a focus point in the surrounding environment. Two or more image sensors receive illumination from the focus point and generate a respective image. A controller receives the measurement of the focal depth, generates an interpolated look-up-table (LUT) function by interpolating between two or more precalculated LUTs, applies the interpolated LUT function to the images to correct a parallax error and a distortion error at the measured focal depth, generates a single image of the surrounding environment, and displays the single image to the user.

Abstract: A mixed reality (MR) system is disclosed. The MR system may determine a first predicted head pose corresponding to a time that virtual reality imagery is rendered, determine a second predicted head pose corresponding to a selected point in time during a camera shutter period, and combine the virtual reality imagery with the stereoscopic imagery based on the first predicted head pose and the second predicted head pose. A simulator that employs remote (e.g., cloud) rendering is also disclosed. The simulator/client device may determine a first pose (e.g., vehicle pose and/or head pose), receive video imagery rendered by a remote server based on the first pose, and apply a timewarp correction to the video imagery based on a comparison of the first pose and a second pose.

Abstract: A video processing device for a mixed reality system is disclosed. A mixed reality system may include a computer system configured to generate a virtual reality video stream and a head mounted device communicatively coupled to the computer system. The head mounted device may include a display, a depth sensor, and a stereoscopic camera system. The video processing device can be communicatively coupled to the computer system and the head mounted device. The video processing device can be configured to employ confidence-based fusion for depth mapping and/or exploit parallelism in high-speed video distortion correction.

Abstract: An image processing system incorporates a distortion correction (DC) sub-system in order to quickly correct skewed images. The DC sub-system includes a buffer, a processor and a sparse matrix table (SMT). The buffer is sized according to an amount of distortion in an input image. Input image pixels from an input frame are buffered in the buffer, and other input image pixels from the same frame overwrite the buffered input image pixels, reducing latency of the DC sub-system. The SMT is dynamically configurable and provides mapping values for mapping output pixels to input pixels. The processor implements combinational logic, including multipliers, lookup tables and adders. The combinational logic interpolates flow control parameters, pixel coordinate values, and pixel intensity values. The distortion correction values are streamed to a display or provided to a subsequent image processing block for further processing.

Abstract: A head wearable device, a method, and a system. The head wearable device may include a display, a camera, a convolutional neural network (CNN) processor, and a processor. The CNN processor may be configured to: receive real scene image data from the camera; identify and classify objects in a real scene image; and generate object classification and position data. The processor may be configured to receive the real scene image data; receive the object classification and position data from the CNN processor; perform an image segmentation operation on the real scene image to fill in the objects; generate filled-in object data indicative of filled-in objects; generate a pixel mask; receive virtual scene image data; create mixed reality scene image data; and output the mixed reality scene image data to the display for presentation to a user.

Abstract: A head wearable device, a method, and a system. The head wearable device may include a display, a camera, a convolutional neural network (CNN) processor, and a processor. The CNN processor may be configured to: receive real scene image data from the camera; identify and classify objects in a real scene image; and generate object classification and position data. The processor may be configured to receive the real scene image data; receive the object classification and position data from the CNN processor; perform an image segmentation operation on the real scene image to fill in the objects; generate filled-in object data indicative of filled-in objects; generate a pixel mask; receive virtual scene image data; create mixed reality scene image data; and output the mixed reality scene image data to the display for presentation to a user.

Abstract: A head wearable device, a method, and a system. The head wearable device may include a display, a camera, a buffer, and a processor. The buffer may be configured to buffer a portion of real scene image data corresponding to a real scene image from the camera. The processor may be configured to: perform a combined distortion correction operation; perform a foreground separation operation; perform a smoothing operation on blending values; perform a chromatic aberration distortion correction operation; receive virtual scene image data corresponding to a virtual scene image; blend processed real scene image data with the virtual scene image data to create a mixed reality scene image as mixed reality scene image data; and output the mixed reality scene image data to the display for presentation to a user.