Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using a surfel map to predict reflections in an environment. One of the methods includes receiving a surfel map comprising a plurality of surfels, wherein each surfel corresponds to a respective different location in an environment. Sensor data for one or more locations in the environment is obtained. The sensor data has been captured by one or more sensors of a vehicle. A range map that represents a projection of the surfel map is generated. Range data in the sensor data is compared to the range map to identify one or more locations in the range map that do not match the range data in the sensor data. The one or more locations in the range map that do not match the range data in the sensor data is classified as reflections.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using a surfel map to predict reflections in an environment. One of the methods includes receiving a surfel map comprising a plurality of surfels, wherein each surfel corresponds to a respective different location in an environment. Sensor data for one or more locations in the environment is obtained. The sensor data has been captured by one or more sensors of a vehicle. A range map that represents a projection of the surfel map is generated. Range data in the sensor data is compared to the range map to identify one or more locations in the range map that do not match the range data in the sensor data. The one or more locations in the range map that do not match the range data in the sensor data is classified as reflections.

Abstract: Aspects of the disclosure relate generally to generating depth data from a video. As an example, one or more computing devices may receive an initialization request for a still image capture mode. After receiving the request to initialize the still image capture mode, the one or more computing devices may automatically begin to capture a video including a plurality of image frames. The one or more computing devices track features between a first image frame of the video and each of the other image frames of the video. Points corresponding to the tracked features may be generated by the one or more computing devices using a set of assumptions. The assumptions may include a first assumption that there is no rotation and a second assumption that there is no translation. The one or more computing devices then generate a depth map based at least in part on the points.

Abstract: Systems and methods are related to a camera rig and generating stereoscopic panoramas from captured images for display in a virtual reality (VR) environment.

Abstract: Systems and methods are described for defining a set of images based on captured images, receiving a viewing direction associated with a user of a virtual reality (VR) head mounted display, receiving an indication of a change in the viewing direction. The methods further include configuring, a re-projection of a portion of the set of images, the re-projection based at least in part on the changed viewing direction and a field of view associated with the captured images, and converting the portion from a spherical perspective projection into a planar perspective projection, rendering by the computing device and for display in the VR head mounted display, an updated view based on the re-projection, the updated view configured to correct distortion and provide stereo parallax in the portion, and providing, to the head mounted display, the updated view including a stereo panoramic scene corresponding to the changed viewing direction.

Abstract: A method of multi-view scene flow stitching includes capture of imagery from a three-dimensional (3D) scene by a plurality of cameras and stitching together captured imagery to generate virtual reality video that is both 360-degree panoramic and stereoscopic. The plurality of cameras capture sequences of video frames, with each camera providing a different viewpoint of the 3D scene. Each image pixel of the sequences of video frames is projected into 3D space to generate a plurality of 3D points. By optimizing for a set of synchronization parameters, stereoscopic image pairs may be generated for synthesizing views from any viewpoint. In some embodiments, the set of synchronization parameters includes a depth map for each of the plurality of video frames, a plurality of motion vectors representing movement of each one of the plurality of 3D points in 3D space over a period of time, and a set of time calibration parameters.

Abstract: In on the general aspect, a camera rig can include a first tier of images sensors including a first plurality of image sensors where the first plurality of image sensors are arranged in a circular shape and oriented such that a field of view of each of the first plurality of image sensors has an axis perpendicular to a tangent of the circular shape. The camera rig can include a second tier of image sensors including a second plurality of image sensors where the second plurality of image sensors are oriented such that a field of view of each of the second plurality of image sensors has an axis non-parallel to the field of view of each of the first plurality of image sensors.

Abstract: Systems and methods are described for defining a set of images based on captured images, receiving a viewing direction associated with a user of a virtual reality (VR) head mounted display, receiving an indication of a change in the viewing direction. The methods further include configuring, a re-projection of a portion of the set of images, the re-projection based at least in part on the changed viewing direction and a field of view associated with the captured images, and converting the portion from a spherical perspective projection into a planar perspective projection, rendering by the computing device and for display in the VR head mounted display, an updated view based on the re-projection, the updated view configured to correct distortion and provide stereo parallax in the portion, and providing, to the head mounted display, the updated view including a stereo panoramic scene corresponding to the changed viewing direction.

Abstract: Systems and methods are related to a camera rig and generating stereoscopic panoramas from captured images for display in a virtual reality (VR) environment.

Abstract: Methods, systems, and articles of manufacture for generating a panoramic image of a long scene are disclosed. These include, fitting planes to 3D points associated with input images of portions of the long scene, where respective planes are fitted to a ground surface, a dominant surface, and at least one of foreground objects and background objects, and where distances from the 3D points to the fitted planes are minimized. These also include, selecting, for respective pixels in the panoramic image, an input image and a fitted plane such that a distance is minimized from the selected the fitted plane to a surface corresponding to the pixels and occlusion of the pixels is reduced in the selected input image, and stitching by projecting the selected input image using the selected fitted plane into the virtual camera.

Abstract: Methods, systems, and articles of manufacture for generating a panoramic image of a long scene are disclosed. These include, fitting planes to 3D points associated with input images of portions of the long scene, where respective planes are fitted to a ground surface, a dominant surface, and at least one of foreground objects and background objects, and where distances from the 3D points to the fitted planes are minimized. These also include, selecting, for respective pixels in the panoramic image, an input image and a fitted plane such that a distance is minimized from the selected the fitted plane to a surface corresponding to the pixels and occlusion of the pixels is reduced in the selected input image, and stitching by projecting the selected input image using the selected fitted plane into the virtual camera.

Abstract: Aspects of the disclosure relate generally to providing a user with an image navigation experience. In order to do so, a reference image may be identified. A set of potential target images for the reference image may also be identified. A drag vector for user input relative to the reference image is determined. For particular image of the set of target images an associated cost is determined based at least in part on a cost function and the drag vector. A target image is selected based on the determined associated costs.

Abstract: Aspects of the disclosure relate generally to providing a user with an image navigation experience. In order to do so, a reference image may be identified. A set of potential target images for the reference image may also be identified. A drag vector for user input relative to the reference image is determined. For particular image of the set of target images an associated cost is determined based at least in part on a cost function and the drag vector. A target image is selected based on the determined associated costs.

Abstract: Aspects of the disclosure relate generally to generating depth data from a video. As an example, one or more computing devices may receive an initialization request for a still image capture mode. After receiving the request to initialize the still image capture mode, the one or more computing devices may automatically begin to capture a video including a plurality of image frames. The one or more computing devices track features between a first image frame of the video and each of the other image frames of the video. Points corresponding to the tracked features may be generated by the one or more computing devices using a set of assumptions. The assumptions may include a first assumption that there is no rotation and a second assumption that there is no translation. The one or more computing devices then generate a depth map based at least in part on the points.

Abstract: Aspects of the disclosure relate generally to providing a user with an image navigation experience. In order to do so, a reference image may be identified. A set of potential target images for the reference image may also be identified. An area of the reference image is identified. For each particular image of the set of potential target images an associated cost for the identified area is determined based at least in part on a cost function for transitioning between the reference image and the particular target image. A target image is selected for association with the identified area based on the determined associated cost functions.

Abstract: Systems, methods, and machine-readable media for generating a classifier configured to label segments of an image, are discussed. According to one aspect, the system may include a training module, a labeling module, and an update module. The training module may be configured to train a first sub-classifier based on photographic data for a set of pre-labeled image segments and a second sub-classifier based on 3-dimensional point data for the set of pre-labeled image segments. The labeling module may be configured to generate a labeling solution comprising a plurality of associations between an image segment from the set of unlabeled image segments and a label. The update module may be configured to update the set of pre-labeled image segments based on the labeling solution. The training module may also be configured to train the first sub-classifier and the second sub-classifier based on the updated set of pre-labeled image segments.

Abstract: Aspects of the disclosure relate generally to generating depth data from a video. As an example, one or more computing devices may receive an initialization request for a still image capture mode. After receiving the request to initialize the still image capture mode, the one or more computing devices may automatically begin to capture a video including a plurality of image frames. The one or more computing devices track features between a first image frame of the video and each of the other image frames of the video. Points corresponding to the tracked features may be generated by the one or more computing devices using a set of assumptions. The assumptions may include a first assumption that there is no rotation and a second assumption that there is no translation. The one or more computing devices then generate a depth map based at least in part on the points.

Abstract: In an embodiment, a method determines a three-dimensional model from a plurality of images taken of a geographic region by one or more cameras from different perspectives. The method includes determining, using a first stereo reconstruction technique: (i) a plurality of three-dimensional candidate surface points from the plurality of images, and (ii) which of the images in the plurality of images view each of the plurality of candidate surface points. The method also includes identifying an empty space between each of the plurality of candidate surface points and each camera model for the respective images determined to view the candidate surface point. The method further includes for each of a plurality of pairs of images from the plurality of images, determining, using a second stereo reconstruction technique, a surface estimate for the pair of images. The method also includes merging the surface estimates to identify a final surface.

Abstract: A control system alters one or more characteristics of an ablating element to ablate tissue. In one aspect, the control system delivers energy nearer to the surface of the tissue by changing the frequency or power. In another aspect, the ablating element delivers focused ultrasound which is focused in at least one dimension. The ablating device may also have a number of ablating elements with different characteristics such as focal length.

Abstract: A cardiac ablation method including the following steps: inserting a treatment catheter into an atrium of a heart, the treatment catheter including an ultrasound emitter; positioning the ultrasound emitter to face heart tissue within the left atrium outside of a pulmonary vein; emitting ultrasound energy from the ultrasound emitter while rotating the ultrasound emitter about a rotation axis; and ablating heart tissue with the ultrasound energy to form a lesion outside of a pulmonary vein.