Abstract: Systems and techniques are provided for fusing raw sensor data captured by a camera sensor and one or more depth sensors to generate a dense depth map. An example process includes receiving raw camera data captured by a camera sensor and descriptive of a scene, receiving raw depth data captured by one or more depth-sensing sensors and descriptive of the scene, and providing the raw camera data and the raw depth data to a neural network, which is configured to fuse the raw camera data and the raw depth data. The example process can further include generating a depth map of the scene based on the fusion of the raw camera data and the raw depth data.

Abstract: The present disclosure generally relates to determining depth information for an image frame and, more specifically, to using partially overlapping image portions to propagate depth information through the entire image frame. In some aspects, a method of the disclosed technology includes steps for identifying an overlapping portion of a first image frame and a second image frame, and a non-overlapping portion of the first image frame; determining depth information for one or more pixels in the overlapping portion of the first image frame, wherein a stereo overlap technique is used to determine the depth information; determining semantic information for the non-overlapping portion of the first image frame; and providing the depth information and the semantic information to a machine learning model to get depth information for the non-overlapping portion of first image frame. Systems and machine-readable media are also provided.

Abstract: Systems and methods of simulating an effect of fog on a Light Detection and Ranging (LiDAR) sensor are disclosed. The method includes the steps of determining whether a target is present within the field-of-view (FOV) of the LiDAR sensor, determining a fog probability density function (PDFfog) vs range, modifying, if a target is present within the FOV, the PDFfog to account for the target, calculating a cumulative density function (CDF) for the PDFfog, randomly sampling the CDF to determine a plurality of ranges and additively plotting a predetermined Gaussian distribution centered on each range, and identifying a peak value of the additive plot and reporting the range associated with the peak value as the strongest return of the LiDAR unit.

Abstract: The systems and methods disclosed herein address simulating the effect of fog on a photon. One method defines a target at a position in a 3D environment and includes the steps of selecting a starting position of the photon in the 3D environment, selecting a propagation vector directed from the starting position toward the target, selecting a propagation distance, determining a new position of the photon based in part on the starting position of the photon and the propagation vector and the propagation distance, determining whether the photon is absorbed before reaching the new position and determining, if the photon has not been absorbed, whether the photon intersects the target before reaching the new position.

Abstract: Systems and techniques are provided for chaining machine learning (ML) models using a confidence level of an output of a provider model. An example method can include receiving a first output generated by a first ML model, determining a confidence level of the first output generated by the first ML model, and providing the first output of the first ML model and the confidence level of the first output to a second ML model as an input of the second ML model. The second ML model can be configured to process the first output of the first ML model and the confidence level of the first output of the first ML model to generate a second output of the second ML model.

Abstract: Systems and techniques are provided for fusing raw sensor data captured by a camera sensor and one or more depth sensors to generate a dense depth map. An example process includes receiving raw camera data captured by a camera sensor and descriptive of a scene, receiving raw depth data captured by one or more depth-sensing sensors and descriptive of the scene, and providing the raw camera data and the raw depth data to a neural network, which is configured to fuse the raw camera data and the raw depth data. The example process can further include generating a depth map of the scene based on the fusion of the raw camera data and the raw depth data.

Abstract: The present disclosure generally relates to determining depth information for an image frame and, more specifically, to using partially overlapping image portions to propagate depth information through the entire image frame. In some aspects, a method of the disclosed technology includes steps for identifying an overlapping portion of a first image frame and a second image frame, and a non-overlapping portion of the first image frame; determining depth information for one or more pixels in the overlapping portion of the first image frame, wherein a stereo overlap technique is used to determine the depth information; determining semantic information for the non-overlapping portion of the first image frame; and providing the depth information and the semantic information to a machine learning model to get depth information for the non-overlapping portion of first image frame. Systems and machine-readable media are also provided.

Abstract: A method includes receiving, from a multiscopic image capture system, a plurality of images depicting a scene. The method includes determining, by application of a neural network based on the plurality of images, a disparity map of the scene. The neural network includes a plurality of layers, and the layers include a rectification layer. The method include determining a matching error of the disparity map based on differences between corresponding pixels of two or more images associated with the disparity map. The method includes back-propagating the matching error to the rectification layer of the neural network. Back-propagating the matching error includes updating one or more weights applied to the rectification layer.

Abstract: A method includes receiving a first depth map that includes a plurality of first pixel depths and a second depth map that includes a plurality of second pixel depths. The first depth map corresponds to a reference depth scale and the second depth map corresponds to a relative depth scale. The method includes aligning the second pixel depths with the first pixel depths. The method includes transforming the aligned region of the second pixel depths such that transformed second edge pixel depths of the aligned region are coextensive with first edge pixel depths surrounding the corresponding region of the first pixel depths. The method includes generating a third depth map. The third depth map includes a first region corresponding to the first pixel depths and a second region corresponding to the transformed and aligned region of the second pixel depths.

Abstract: A method includes receiving sensor data representing a first object in an environment and generating, based on the sensor data, a first state vector that represents physical properties of the first object. The method also includes generating, by a first machine learning model and based on the first state vector and a second state vector that represents physical properties of a second object previously observed in the environment, a metric indicating a likelihood that the first object is the same as the second object. The method further includes determining, based on the metric, to update the second state vector and updating, by a second machine learning model configured to maintain the second state vector over time and based on the first state vector, the second state vector to incorporate into the second state vector information concerning physical properties of the second object as represented in the first state vector.

Abstract: A method includes identifying a target surface in an environment of a robotic device. The method further includes controlling a moveable component of the robotic device to move along a motion path relative to the target surface, wherein the moveable component comprises a light source and a camera. The method additionally includes receiving a plurality of images from the camera when the moveable component is at a plurality of poses along the motion path and when the light source is illuminating the target surface. The method also includes determining bidirectional reflectance distribution function (BRDF) image data, wherein the BRDF image data comprises the plurality of images converted to angular space with respect to the target surface. The method further includes determining, based on the BRDF image data and by applying at least one pre-trained machine learning model, a material property of the target surface.

Abstract: A method includes receiving sensor data from a plurality of robot sensors on a robot. The method includes generating a depth map that includes a plurality of pixel depths. The method includes determining, for each respective pixel depth, based on the at least one robot sensor associated with the respective pixel depth, a pixelwise confidence level indicative of a likelihood that the respective pixel depth accurately represents a distance between the robot and a feature of the environment. The method includes generating a pixelwise filterable depth map for a control system of the robot. The pixelwise filterable depth map is filterable to produce a robot operation specific depth map. The robot operation specific depth map is determined based on a comparison of each respective pixelwise confidence level with a confidence threshold corresponding to at least one operation of the robot controlled by the control system of the robot.

Abstract: A sensing device is described for mounting on a movable component of a robotic device. The sensing device includes a plurality of illumination sources comprising at least one ultraviolet (UV) illumination source. The sensing device further includes at least two cameras arranged in a stereo pair. The sensing device additionally includes a camera with a UV filter, wherein the UV filter is configured to allow wavelengths corresponding to UV light and to block wavelengths corresponding to visible and near infrared light, wherein the UV filter allows transmission of light within an angular range such that the UV filter allows for the transmission of light at one end of the angular range to be equivalent to the transmission of light at an opposite end of the angular range.

Abstract: A system includes a robotic device, a sensor disposed on the robotic device, and circuitry configured to perform operations. The operations include determining a map that represents stationary features of an environment and receiving, from the sensor, sensor data representing the environment. The operations also include determining, based on the sensor data, a representation of an actor within the environment, where the representation includes keypoints representing corresponding body locations of the actor. The operations also include determining that a portion of a particular stationary feature is positioned within a threshold distance of a particular keypoint and, based on thereon, updating the map to indicate that the portion is to be cleaned. The operations further include, based on the map as updated, causing the robotic device to clean the portion of the particular stationary feature.

Abstract: A method includes receiving sensor data from a plurality of robot sensors on a robot. The method includes generating a depth map that includes a plurality of pixel depths. The method includes determining, for each respective pixel depth, based on the at least one robot sensor associated with the respective pixel depth, a pixelwise confidence level indicative of a likelihood that the respective pixel depth accurately represents a distance between the robot and a feature of the environment. The method includes generating a pixelwise filterable depth map for a control system of the robot. The pixelwise filterable depth map is filterable to produce a robot operation specific depth map. The robot operation specific depth map is determined based on a comparison of each respective pixelwise confidence level with a confidence threshold corresponding to at least one operation of the robot controlled by the control system of the robot.

Abstract: A light source may illuminate a scene that is obscured by fog. Light may reflect back to a time-resolved light sensor. For instance, the light sensor may comprise avalanche photodiodes that are not single-photon sensitive. The light sensor may perform a raster scan. The imaging system may determine reflectance and depth of the fog-obscured target. The imaging system may perform a probabilistic algorithm that exploits the fact that times of arrival of photons reflected from fog have a Gamma distribution that is different than the Gaussian distribution of times of arrival of photons reflected from the target. The imaging system may adjust frame rate locally depending on local density of fog, as indicated by a local Gamma distribution determined in a prior step. The imaging system may perform one or more of spatial regularization, temporal regularization, and deblurring.

Abstract: A method includes receiving, from a multiscopic image capture system, a plurality of images depicting a scene. The method includes determining, by application of a neural network based on the plurality of images, a disparity map of the scene. The neural network includes a plurality of layers, and the layers include a rectification layer. The method include determining a matching error of the disparity map based on differences between corresponding pixels of two or more images associated with the disparity map. The method includes back-propagating the matching error to the rectification layer of the neural network. Back-propagating the matching error includes updating one or more weights applied to the rectification layer.

Abstract: A method includes receiving sensor data representing a first object in an environment and generating, based on the sensor data, a first state vector that represents physical properties of the first object. The method also includes generating, by a first machine learning model and based on the first state vector and a second state vector that represents physical properties of a second object previously observed in the environment, a metric indicating a likelihood that the first object is the same as the second object. The method further includes determining, based on the metric, to update the second state vector and updating, by a second machine learning model configured to maintain the second state vector over time and based on the first state vector, the second state vector to incorporate into the second state vector information concerning physical properties of the second object as represented in the first state vector.

Abstract: A method includes receiving a first depth map that includes a plurality of first pixel depths and a second depth map that includes a plurality of second pixel depths. The first depth map corresponds to a reference depth scale and the second depth map corresponds to a relative depth scale. The method includes aligning the second pixel depths with the first pixel depths. The method includes transforming the aligned region of the second pixel depths such that transformed second edge pixel depths of the aligned region are coextensive with first edge pixel depths surrounding the corresponding region of the first pixel depths. The method includes generating a third depth map. The third depth map includes a first region corresponding to the first pixel depths and a second region corresponding to the transformed and aligned region of the second pixel depths.

Abstract: A method includes receiving, from a multiscopic image capture system, a plurality of images depicting a scene. The method includes determining, by application of a neural network based on the plurality of images, a disparity map of the scene. The neural network includes a plurality of layers, and the layers include a rectification layer. The method include determining a matching error of the disparity map based on differences between corresponding pixels of two or more images associated with the disparity map. The method includes back-propagating the matching error to the rectification layer of the neural network. Back-propagating the matching error includes updating one or more weights applied to the rectification layer.