Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using a surfel map to generate long range localization. One of the methods includes obtaining, for a particular location of a vehicle having a camera and a detection sensor, surfel data including a plurality of surfels. Each surfel in the surfel data has a respective location and corresponds to a different respective detected surface in an environment. Image data captured by the camera is obtained. It is determined that a region of interest for detecting objects for a vehicle planning process is outside a detectable region for the detection sensor. In response, it is determined that the image data for the region of interest matches surfel color data for the surfels corresponding to the region of interest. In response, the vehicle planning process is performed with the region of interest designated as having no unexpected objects.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for sensor calibration with environment map. In some implementations, a three-dimensional surfel representation of a real-world environment is obtained. One or more surfels of the surfel representation having a particular classification of the different classifications are selected. Input sensor data from one or more sensors installed on an autonomous or semi-autonomous vehicle are received. The input sensor data is compared to the surfel representation to identify one or more differences between the observation and the surfel representation. At least one sensor of the one or more sensors is calibrated using the one or more differences between the observation and the surfel representation.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using a surfel map to generate long range localization. One of the methods includes obtaining, for a particular location of a vehicle having a camera and a detection sensor, surfel data including a plurality of surfels. Each surfel in the surfel data has a respective location and corresponds to a different respective detected surface in an environment. Image data captured by the camera is obtained. It is determined that a region of interest for detecting objects for a vehicle planning process is outside a detectable region for the detection sensor. In response, it is determined that the image data for the region of interest matches surfel color data for the surfels corresponding to the region of interest. In response, the vehicle planning process is performed with the region of interest designated as having no unexpected objects.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using a surfel map to generate a prediction for a state of an environment. One of the methods includes obtaining surfel data comprising a plurality of surfels, wherein each surfel corresponds to a respective different location in an environment, and each surfel has associated data that comprises an uncertainty measure; obtaining sensor data for one or more locations in the environment, the sensor data having been captured by one or more sensors of a first vehicle; determining one or more particular surfels corresponding to respective locations of the obtained sensor data; and combining the surfel data and the sensor data to generate a respective object prediction for each of the one or more locations of the obtained sensor data.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for sensor calibration with environment map. In some implementations, a three-dimensional surfel representation of a real-world environment is obtained. One or more surfels of the surfel representation having a particular classification of the different classifications are selected. Input sensor data from one or more sensors installed on an autonomous or semi-autonomous vehicle are received. The input sensor data is compared to the surfel representation to identify one or more differences between the observation and the surfel representation. At least one sensor of the one or more sensors is calibrated using the one or more differences between the observation and the surfel representation.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using a surfel map to generate long range localization. One of the methods includes obtaining, for a particular location of a vehicle having a camera and a detection sensor, surfel data including a plurality of surfels. Each surfel in the surfel data has a respective location and corresponds to a different respective detected surface in an environment. Image data captured by the camera is obtained. It is determined that a region of interest for detecting objects for a vehicle planning process is outside a detectable region for the detection sensor. In response, it is determined that the image data for the region of interest matches surfel color data for the surfels corresponding to the region of interest. In response, the vehicle planning process is performed with the region of interest designated as having no unexpected objects.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing data characterizing a first region in an environment to generate a prediction characterizing a second region in the environment. One of the methods includes obtaining surfel data comprising a plurality of surfels; obtaining sensor data for a plurality of locations in a first region of the environment; determining, from the surfel data, a plurality of first surfels corresponding to respective locations in the first region of the environment; determining, using the first surfels and the sensor data, a difference between i) a first representation of the first region of the environment corresponding to the first surfels and ii) a second representation of the first region of the environment corresponding to the sensor data; and generating a respective object prediction for one or more locations in a second region of the environment.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using a surfel map to generate a prediction for a state of an environment. One of the methods includes obtaining surfel data comprising a plurality of surfels, wherein each surfel corresponds to a respective different location in an environment, and each surfel has associated data that comprises an uncertainty measure; obtaining sensor data for one or more locations in the environment, the sensor data having been captured by one or more sensors of a first vehicle; determining one or more particular surfels corresponding to respective locations of the obtained sensor data; and combining the surfel data and the sensor data to generate a respective object prediction for each of the one or more locations of the obtained sensor data.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for sensor calibration with environment map. In some implementations, a three-dimensional surfel representation of a real-world environment is obtained. One or more surfels of the surfel representation having a particular classification of the different classifications are selected. Input sensor data from one or more sensors installed on an autonomous or semi-autonomous vehicle are received. The input sensor data is compared to the surfel representation to identify one or more differences between the observation and the surfel representation. At least one sensor of the one or more sensors is calibrated using the one or more differences between the observation and the surfel representation.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using a surfel map to generate long range localization. One of the methods includes obtaining, for a particular location of a vehicle having a camera and a detection sensor, surfel data including a plurality of surfels. Each surfel in the surfel data has a respective location and corresponds to a different respective detected surface in an environment. Image data captured by the camera is obtained. It is determined that a region of interest for detecting objects for a vehicle planning process is outside a detectable region for the detection sensor. In response, it is determined that the image data for the region of interest matches surfel color data for the surfels corresponding to the region of interest. In response, the vehicle planning process is performed with the region of interest designated as having no unexpected objects.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing data characterizing a first region in an environment to generate a prediction characterizing a second region in the environment. One of the methods includes obtaining surfel data comprising a plurality of surfels; obtaining sensor data for a plurality of locations in a first region of the environment; determining, from the surfel data, a plurality of first surfels corresponding to respective locations in the first region of the environment; determining, using the first surfels and the sensor data, a difference between i) a first representation of the first region of the environment corresponding to the first surfels and ii) a second representation of the first region of the environment corresponding to the sensor data; and generating a respective object prediction for one or more locations in a second region of the environment.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for autonomous driving with surfel maps. In some implementations, a three-dimensional representation of a real-world environment is obtained. Each of the surfels can correspond to a respective point of plurality of points in a three-dimensional space of the real-world environment. Input sensor data is received from multiple sensors installed on the autonomous vehicle. A pedestrian is detected from the input sensor data. A determination is made that the pedestrian is located behind a barrier. A driving plan is updated based on determining that the pedestrian is located behind a barrier.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using a surfel map to generate a prediction for a state of an environment. One of the methods includes obtaining surfel data comprising a plurality of surfels, wherein each surfel corresponds to a respective different location in an environment, and each surfel has associated data that comprises an uncertainty measure; obtaining sensor data for one or more locations in the environment, the sensor data having been captured by one or more sensors of a first vehicle; determining one or more particular surfels corresponding to respective locations of the obtained sensor data; and combining the surfel data and the sensor data to generate a respective object prediction for each of the one or more locations of the obtained sensor data.

Abstract: The technology relates to transitioning between panoramic imagery and the imagery from which the panoramic imagery was generated. Individual images of a portion of a scene may be combined to create a panoramic image of the entire scene. The individual images may be associated with the portion of the panoramic image which they form. The full panoramic image may then be displayed. Based on user input, a portion of the panoramic image may be zoomed-in on, and the individual image associated with that portion of the panoramic image may be displayed.

Abstract: Aspects of the disclosure relate to capturing panoramic images using a computing device. For example, the computing device may record a set of video frames and tracking features each including one or more features that appear in two or more video frames of the set of video frames within the set of video frames may be determined. A set of frame-based features based on the displacement of the tracking features between two or more video frames of the set of video frames may be determined by the computing device. A set of historical feature values based on the set of frame-based features may also be determined by the computing device. The computing device may determine then whether a user is attempting to capture a panoramic image based on the set of historical feature values. In response, the computing device may capture a panoramic image.

Abstract: Systems and methods are described include defining, at a computing device, a set of images based on captured images, projecting, at the computing device, a portion of the set of images from a planar perspective image plane onto a spherical image plane by recasting a plurality of viewing rays associated with the portion of the set of images from a plurality of viewpoints arranged around a curved path to a viewpoint, determining, at the computing device, a periphery boundary corresponding to the viewpoint and generating updated images by removing pixels that are outside of the periphery boundary, and providing, for display, the updated images within the bounds of the periphery boundary.

Abstract: Systems and methods for capturing omnistereo content for a mobile device may include receiving an indication to capture a plurality of images of a scene, capturing the plurality of images using a camera associated with a mobile device and displaying on a screen of the mobile device and during capture, a representation of the plurality of images and presenting a composite image that includes a target capture path and an indicator that provides alignment information corresponding to a source capture path associated with the mobile device during capture of the plurality of images. The system may detect that a portion of the source capture path does not match a target capture path. The system can provide an updated indicator in the screen that may include a prompt to a user of the mobile device to adjust the mobile device to align the source capture path with the target capture path.

Abstract: Systems and methods are related to a camera rig and generating stereoscopic panoramas from captured images for display in a virtual reality (VR) environment.

Abstract: System and method for producing an intermediate image in three passes is provided. An initial image and a final image are identified, each image comprising an associated depthmap and corresponding to a respective camera pose from a respective angular heading. An intermediate image corresponding to an intermediate camera pose with an intermediate angular heading is rendered. A first intermediate image is rendered at a first opacity based on the final image and the associated final depthmap, a second intermediate image is rendered at a second opacity based on the initial image and the associated initial depthmap, and a third intermediate image is rendered at a third opacity based on the final image and the associated final depthmap. The rendered first, second, and third intermediate images, each of which correspond to the intermediate camera pose with the intermediate angular heading, are overlaid to produce a combined image for display.

Abstract: Systems and methods are described for defining a set of images based on captured images, receiving a viewing direction associated with a user of a virtual reality (VR) head mounted display, receiving an indication of a change in the viewing direction. The methods further include configuring, a re-projection of a portion of the set of images, the re-projection based at least in part on the changed viewing direction and a field of view associated with the captured images, and converting the portion from a spherical perspective projection into a planar perspective projection, rendering by the computing device and for display in the VR head mounted display, an updated view based on the re-projection, the updated view configured to correct distortion and provide stereo parallax in the portion, and providing, to the head mounted display, the updated view including a stereo panoramic scene corresponding to the changed viewing direction.