Abstract: A machine-learning (ML) architecture for determining three or more outputs, such as a two and/or three-dimensional region of interest, semantic segmentation, direction logits, depth data, and/or instance segmentation associated with an object in an image. The ML architecture may output these outputs at a rate of 30 or more frames per second on consumer grade hardware.

Abstract: A vehicle can use an image sensor to both detect objects and determine depth data associated with the environment the vehicle is traversing. The vehicle can capture image data and lidar data using the various sensors. The image data can be provided to a machine-learned model trained to output depth data of an environment. Such models may be trained, for example, by using lidar data and/or three-dimensional map data associated with a region in which training images and/or lidar data were captured as ground truth data. The autonomous vehicle can further process the depth data and generate additional data including localization data, three-dimensional bounding boxes, and relative depth data and use the depth data and/or the additional data to autonomously traverse the environment, provide calibration/validation for vehicle sensors, and the like.

Abstract: Techniques for training a machine learned (ML) model to determine depth data based on image data are discussed herein. Training can use stereo image data and depth data (e.g., lidar data). A first (e.g., left) image can be input to a ML model, which can output predicted disparity and/or depth data. The predicted disparity data can be used with second image data (e.g., a right image) to reconstruct the first image. Differences between the first and reconstructed images can be used to determine a loss. Losses may include pixel, smoothing, structural similarity, and/or consistency losses. Further, differences between the depth data and the predicted depth data and/or differences between the predicted disparity data and the predicted depth data can be determined, and the ML model can be trained based on the various losses. Thus, the techniques can use self-supervised training and supervised training to train a ML model.

Abstract: Depth estimates for an object made by one or more sensors of a vehicle may be refined using locations of environmental attributes that are proximate the object. An image captured of the object proximate an environmental attribute may be analyzed to determine where the object is positioned relative to the environmental attribute. A machine-learned model may be used to detect the environmental attribute, and a location of the environmental attribute may be determined from map data. A probability of a location of the object may be determined based on the known location of the environmental attribute. The location probability of the object may be used to refine depth estimates generated by other means, such as a monocular depth estimation from an image using computer vision.

Abstract: Depth estimates for an object made by one or more sensors of a vehicle may be refined using locations of environmental attributes that are proximate the object. An image captured of the object proximate an environmental attribute may be analyzed to determine where the object is positioned relative to the environmental attribute. A machine-learned model may be used to detect the environmental attribute, and a location of the environmental attribute may be determined from map data. A probability of a location of the object may be determined based on the known location of the environmental attribute. The location probability of the object may be used to refine depth estimates generated by other means, such as a monocular depth estimation from an image using computer vision.

Abstract: Depth estimates for an object made by one or more sensors of a vehicle may be refined using locations of environmental attributes that are proximate the object. An image captured of the object proximate an environmental attribute may be analyzed to determine where the object is positioned relative to the environmental attribute. A machine-learned model may be used to detect the environmental attribute, and a location of the environmental attribute may be determined from map data. A probability of a location of the object may be determined based on the known location of the environmental attribute. The location probability of the object may be used to refine depth estimates generated by other means, such as a monocular depth estimation from an image using computer vision.

Abstract: Techniques for training a machine learned (ML) model to determine depth data based on image data are discussed herein. Training can use stereo image data and depth data (e.g., lidar data). A first (e.g., left) image can be input to a ML model, which can output predicted disparity and/or depth data. The predicted disparity data can be used with second image data (e.g., a right image) to reconstruct the first image. Differences between the first and reconstructed images can be used to determine a loss. Losses may include pixel, smoothing, structural similarity, and/or consistency losses. Further, differences between the depth data and the predicted depth data and/or differences between the predicted disparity data and the predicted depth data can be determined, and the ML model can be trained based on the various losses. Thus, the techniques can use self-supervised training and supervised training to train a ML model.

Abstract: A vehicle can use an image sensor to both detect objects and determine depth data associated with the environment the vehicle is traversing. The vehicle can capture image data and lidar data using the various sensors. The image data can be provided to a machine-learned model trained to output depth data of an environment. Such models may be trained, for example, by using lidar data and/or three-dimensional map data associated with a region in which training images and/or lidar data were captured as ground truth data. The autonomous vehicle can further process the depth data and generate additional data including localization data, three-dimensional bounding boxes, and relative depth data and use the depth data and/or the additional data to autonomously traverse the environment, provide calibration/validation for vehicle sensors, and the like.

Abstract: Techniques for training a machine learned (ML) model to determine depth data based on image data are discussed herein. Training can use stereo image data and depth data (e.g., lidar data). A first (e.g., left) image can be input to a ML model, which can output predicted disparity and/or depth data. The predicted disparity data can be used with second image data (e.g., a right image) to reconstruct the first image. Differences between the first and reconstructed images can be used to determine a loss. Losses may include pixel, smoothing, structural similarity, and/or consistency losses. Further, differences between the depth data and the predicted depth data and/or differences between the predicted disparity data and the predicted depth data can be determined, and the ML model can be trained based on the various losses. Thus, the techniques can use self-supervised training and supervised training to train a ML model.

Abstract: A vehicle can use an image sensor to both detect objects and determine depth data associated with the environment the vehicle is traversing. The vehicle can capture image data and lidar data using the various sensors. The image data can be provided to a machine-learned model trained to output depth data of an environment. Such models may be trained, for example, by using lidar data and/or three-dimensional map data associated with a region in which training images and/or lidar data were captured as ground truth data. The autonomous vehicle can further process the depth data and generate additional data including localization data, three-dimensional bounding boxes, and relative depth data and use the depth data and/or the additional data to autonomously traverse the environment, provide calibration/validation for vehicle sensors, and the like.

Abstract: A machine-learning (ML) architecture for determining three or more outputs, such as a two and/or three-dimensional region of interest, semantic segmentation, direction logits, depth data, and/or instance segmentation associated with an object in an image. The ML architecture may output these outputs at a rate of 30 or more frames per second on consumer grade hardware.

Abstract: Techniques for training a machine learned (ML) model to determine depth data based on image data are discussed herein. Training can use stereo image data and depth data (e.g., lidar data). A first (e.g., left) image can be input to a ML model, which can output predicted disparity and/or depth data. The predicted disparity data can be used with second image data (e.g., a right image) to reconstruct the first image. Differences between the first and reconstructed images can be used to determine a loss. Losses may include pixel, smoothing, structural similarity, and/or consistency losses. Further, differences between the depth data and the predicted depth data and/or differences between the predicted disparity data and the predicted depth data can be determined, and the ML model can be trained based on the various losses. Thus, the techniques can use self-supervised training and supervised training to train a ML model.

Abstract: Techniques for training a machine learned (ML) model to determine depth data based on image data are discussed herein. Training can use stereo image data and depth data (e.g., lidar data). A first (e.g., left) image can be input to a ML model, which can output predicted disparity and/or depth data. The predicted disparity data can be used with second image data (e.g., a right image) to reconstruct the first image. Differences between the first and reconstructed images can be used to determine a loss. Losses may include pixel, smoothing, structural similarity, and/or consistency losses. Further, differences between the depth data and the predicted depth data and/or differences between the predicted disparity data and the predicted depth data can be determined, and the ML model can be trained based on the various losses. Thus, the techniques can use self-supervised training and supervised training to train a ML model.

Abstract: A vehicle can use an image sensor to both detect objects and determine depth data associated with the environment the vehicle is traversing. The vehicle can capture image data and lidar data using the various sensors. The image data can be provided to a machine-learned model trained to output depth data of an environment. Such models may be trained, for example, by using lidar data and/or three-dimensional map data associated with a region in which training images and/or lidar data were captured as ground truth data. The autonomous vehicle can further process the depth data and generate additional data including localization data, three-dimensional bounding boxes, and relative depth data and use the depth data and/or the additional data to autonomously traverse the environment, provide calibration/validation for vehicle sensors, and the like.

Abstract: Training a machine-learning (ML) architecture to determine three or more outputs at a rate of 30 or more frames per second on consumer grade hardware may comprise jointly training components of the ML using loss(es) determined across the components and/or consistency losses determined between outputs of two or more components. The ML architecture discussed herein may comprise one or more sets of neural network layers and/or respective components for determining a two and/or three-dimensional region of interest, semantic segmentation, direction logits, depth data, and/or instance segmentation associated with an object in an image.

Abstract: A vehicle can use an image sensor to both detect objects and determine depth data associated with the environment the vehicle is traversing. The vehicle can capture image data and lidar data using the various sensors. The image data can be provided to a machine-learned model trained to output depth data of an environment. Such models may be trained, for example, by using lidar data and/or three-dimensional map data associated with a region in which training images and/or lidar data were captured as ground truth data. The autonomous vehicle can further process the depth data and generate additional data including localization data, three-dimensional bounding boxes, and relative depth data and use the depth data and/or the additional data to autonomously traverse the environment, provide calibration/validation for vehicle sensors, and the like.

Abstract: Depth estimates for an object made by one or more sensors of a vehicle may be refined using locations of environmental attributes that are proximate the object. An image captured of the object proximate an environmental attribute may be analyzed to determine where the object is positioned relative to the environmental attribute. A machine-learned model may be used to detect the environmental attribute, and a location of the environmental attribute may be determined from map data. A probability of a location of the object may be determined based on the known location of the environmental attribute. The location probability of the object may be used to refine depth estimates generated by other means, such as a monocular depth estimation from an image using computer vision.

Abstract: A video visual and audio query system for quickly identifying video within a large known corpus of videos being played on any screen or display. In one embodiment, the system can record via a mobile phone camera and microphone a live video clip from the TV and transcode it into a sequence of frame-signatures. The signatures representative of the clips can then be matched against the signatures of the TV content in a corpus across a network to identify the correct TV show or movie.

Abstract: Motion capture animation, shape completion and markerless motion capture methods are provided. A pose deformation space model encoding variability in pose is learnt from a three-dimensional (3D) dataset. Body shape deformation space model encoding variability in pose and shape is learnt from another 3D dataset. The learnt pose model is combined with the learnt body shape model. For motion capture animation, given parameter set, the combined model generates a 3D shape surface of a body in a pose and shape. For shape completion, given partial surface of a body defined as 3D points, the combined model generates a 3D surface model in the combined spaces that fits the 3D points. For markerless motion capture, given 3D information of a body, the combined model traces the movement of the body using the combined spaces that fits the 3D information or reconstructing the body's shape or deformations that fits the 3D information.

Abstract: A video visual and audio query system for quickly identifying video within a large known corpus of videos being played on any screen or display. In one embodiment, the system can record via a mobile phone camera and microphone a live video clip from the TV and transcode it into a sequence of frame-signatures. The signatures representative of the clips can then be matched against the signatures of the TV content in a corpus across a network to identify the correct TV show or movie.