Abstract: The subject disclosure is directed towards controlling the intensity of illumination of a scene or part of a scene, including to conserve illumination power. Quality of depth data in stereo images may be measured with different illumination states; environmental conditions, such as ambient light, natural texture may affect the quality. The illumination intensity may be controllably varied to obtain sufficient quality while conserving power. The control may be directed to one or more regions of interest corresponding to an entire scene or part of a scene.

Abstract: A processor-implemented method for encoding a block-based volumetric video having a plurality of video frames of a 3D object into a 2D video format is provided. The method including (i) splitting each video frame of the plurality of video frames into a first region that includes RGB data, a second region that includes depth data, and at least a third region containing render metadata of the 3D object; and (ii) storing the render metadata of the 3D object in at least one of the first region that includes the RGB data, the second region that includes the depth data and the at least the third region in at least one channel that is selected from a U chroma channel, a V chroma channel, and a luma channel of the block-based volumetric video.

Abstract: A processor-implemented method of generating a three-dimensional (3D) volumetric video with an overlay representing visibility counts per pixel of a texture atlas, associated with a viewer telemetry data is provided. The method includes (i) capturing the viewer telemetry data, (ii) determining a visibility of each pixel in the texture atlas associated with a 3D content based on the viewer telemetry data, (iii) generating at least one visibility counts per pixel of the texture atlas based on the visibility of each pixel in the texture atlas, and (iv) generating one of: the 3D volumetric video with the overlay of at least one heat map associated with the viewer telemetry data, using the at least one visibility counts per pixel and a curated selection of the 3D volumetric content based on the viewer telemetry data, using the visibility counts per pixel.

Abstract: Systems and methods for stereo matching based upon active illumination using a patch in a non-actively illuminated image to obtain weights that are used in patch similarity determinations in actively illuminated stereo images is provided. To correlate pixels in actively illuminated stereo images, adaptive support weights computations are used to determine similarity of patches corresponding to the pixels. In order to obtain adaptive support weights for the adaptive support weights computations, weights are obtained by processing a non-actively illuminated (“clean”) image.

Abstract: A processor-implemented method of generating a three-dimensional (3D) volumetric video with an overlay representing visibility counts per pixel of a texture atlas, associated with a viewer telemetry data is provided. The method includes (i) capturing the viewer telemetry data, (ii) determining a visibility of each pixel in the texture atlas associated with a 3D content based on the viewer telemetry data, (iii) generating at least one visibility counts per pixel of the texture atlas based on the visibility of each pixel in the texture atlas, and (iv) generating one of: the 3D volumetric video with the overlay of at least one heat map associated with the viewer telemetry data, using the at least one visibility counts per pixel and a curated selection of the 3D volumetric content based on the viewer telemetry data, using the visibility counts per pixel.

Abstract: Systems and methods for stereo matching based upon active illumination using a patch in a non-actively illuminated image to obtain weights that are used in patch similarity determinations in actively illuminated stereo images is provided. To correlate pixels in actively illuminated stereo images, adaptive support weights computations are used to determine similarity of patches corresponding to the pixels. In order to obtain adaptive support weights for the adaptive support weights computations, weights are obtained by processing a non-actively illuminated (“clean”) image.

Abstract: A processor implemented method for compressing surface data of a 3 dimensional object in a global digital space, using an image encoder that supports an image data compression algorithm, the image encoder being coupled to a transmitter. The method includes the steps of (i) decomposing the surface data into at least one surface representation that is encoded in an oriented bounding box, (ii) transforming the oriented bounding box into a canonical camera representation to obtain canonical coordinates for the at least one surface representation, (iii) converting each of the at least one surface representation into at least one bounding box image pair that includes a grayscale image representing depth, and a color image and (iv) tiling the at least one bounding box image pair to produce a tiled bounding box image.

Abstract: A processor implemented method for compressing surface data of a 3 dimensional object in a global digital space, using an image encoder that supports an image data compression algorithm, the image encoder being coupled to a transmitter. The method includes the steps of (i) decomposing the surface data into at least one surface representation that is encoded in an oriented bounding box, (ii) transforming the oriented bounding box into a canonical camera representation to obtain canonical coordinates for the at least one surface representation, (iii) converting each of the at least one surface representation into at least one bounding box image pair that includes a grayscale image representing depth, and a color image and (iv) tiling the at least one bounding box image pair to produce a tiled bounding box image.

Abstract: A processor-implemented method for streaming visible blocks of volumetric video to a client device during a predefined time period is provided. The method includes (i) receiving at least one block description file from a content server, (ii) processing each block description in the at least one block description file, to determine the visible blocks that are selected from a set of blocks, that are capable of being visible to a viewer of the client device during the predefined time period, based on a 3D position, size, and an orientation of each block in the set of blocks and at least one view parameter of a user of the client device, (iii) transmitting a request for the visible blocks, to the content server, and (iv) receiving the visible blocks as a visible blocks video at the client device.

Abstract: A processor-implemented method for streaming visible blocks of volumetric video to a client device during a predefined time period is provided. The method includes (i) receiving at least one block description file from a content server, (ii) processing each block description in the at least one block description file, to determine the visible blocks that are selected from a set of blocks, that are capable of being visible to a viewer of the client device during the predefined time period, based on a 3D position, size, and an orientation of each block in the set of blocks and at least one view parameter of a user of the client device, (iii) transmitting a request for the visible blocks, to the content server, and (iv) receiving the visible blocks as a visible blocks video at the client device.

Abstract: A processor implemented method for compressing time-varying surface data of a 3 dimensional object in a global digital space having frames, using a video encoder that supports a video data compression algorithm, the video encoder being coupled to a transmitter. The method includes the steps of (i) decomposing the time-varying surface data into at least one surface representation that is encoded in an oriented bounding box, (ii) transforming the oriented bounding box into a canonical camera representation for each frame to obtain canonical coordinates for the at least one surface representation, (iii) converting each of the at least one surface representation into at least one bounding box video pair that includes a grayscale video representing depth, and a color video and (iv) tiling the at least one bounding box video pair for each frame to produce a tiled bounding box video.

Abstract: A processor implemented method for compressing time-varying surface data of a 3 dimensional object in a global digital space having frames, using a video encoder that supports a video data compression algorithm, the video encoder being coupled to a transmitter. The method includes the steps of (i) decomposing the time-varying surface data into at least one surface representation that is encoded in an oriented bounding box, (ii) transforming the oriented bounding box into a canonical camera representation for each frame to obtain canonical coordinates for the at least one surface representation, (iii) converting each of the at least one surface representation into at least one bounding box video pair that includes a grayscale video representing depth, and a color video and (iv) tiling the at least one bounding box video pair for each frame to produce a tiled bounding box video.

Abstract: The subject disclosure is directed towards a high resolution, high frame rate, robust stereo depth system. The system provides depth data in varying conditions based upon stereo matching of images, including actively illuminated IR images in some implementations. A clean IR or RGB image may be captured and used with any other captured images in some implementations. Clean IR images may be obtained by using a notch filter to filter out the active illumination pattern. IR stereo cameras, a projector, broad spectrum IR LEDs and one or more other cameras may be incorporated into a single device, which may also include image processing components to internally compute depth data in the device for subsequent output.

Abstract: The subject disclosure is directed towards communicating image-related data between a base station and/or one or more satellite computing devices, e.g., tablet computers and/or smartphones. A satellite device captures image data and communicates image-related data (such as the images or depth data processed therefrom) to another device, such as a base station. The receiving device uses the image-related data to enhance depth data (e.g., a depth map) based upon the image data captured from the satellite device, which may be physically closer to something in the scene than the base station, for example. To more accurately capture depth data in various conditions, an active illumination pattern may be projected from the base station or another external projector, whereby satellite units may use the other source's active illumination and thereby need not consume internal power to benefit from active illumination.

Abstract: An active rangefinder system disclosed herein parameterizes a set of transformations predicting different possible appearances of a projection feature projected into a three-dimensional scene. A matching module matches an image of the projected projection feature with one of the transformations, and a depth estimation module estimates a distance to an object reflecting the projection feature based on the transformation identified by the matching module.

Abstract: The subject disclosure is directed towards controlling the intensity of illumination of a scene or part of a scene, including to conserve illumination power. Quality of depth data in stereo images may be measured with different illumination states; environmental conditions, such as ambient light, natural texture may affect the quality. The illumination intensity may be controllably varied to obtain sufficient quality while conserving power. The control may be directed to one or more regions of interest corresponding to an entire scene or part of a scene.

Abstract: The subject disclosure is directed towards stereo matching based upon active illumination, including using a patch in a non-actively illuminated image to obtain weights that are used in patch similarity determinations in actively illuminated stereo images. To correlate pixels in actively illuminated stereo images, adaptive support weights computations may be used to determine similarity of patches corresponding to the pixels. In order to obtain meaningful adaptive support weights for the adaptive support weights computations, weights are obtained by processing a non-actively illuminated (“clean”) image.

Abstract: The subject disclosure is directed towards a high resolution, high frame rate, robust stereo depth system. The system provides depth data in varying conditions based upon stereo matching of images, including actively illuminated IR images in some implementations. A clean IR or RGB image may be captured and used with any other captured images in some implementations. Clean IR images may be obtained by using a notch filter to filter out the active illumination pattern. IR stereo cameras, a projector, broad spectrum IR LEDs and one or more other cameras may be incorporated into a single device, which may also include image processing components to internally compute depth data in the device for subsequent output.

Abstract: The subject disclosure is directed towards communicating image-related data between a base station and/or one or more satellite computing devices, e.g., tablet computers and/or smartphones. A satellite device captures image data and communicates image-related data (such as the images or depth data processed therefrom) to another device, such as a base station. The receiving device uses the image-related data to enhance depth data (e.g., a depth map) based upon the image data captured from the satellite device, which may be physically closer to something in the scene than the base station, for example. To more accurately capture depth data in various conditions, an active illumination pattern may be projected from the base station or another external projector, whereby satellite units may use the other source's active illumination and thereby need not consume internal power to benefit from active illumination.

Abstract: Real-time stereo matching is described, for example, to find depths of objects in an environment from an image capture device capturing a stream of stereo images of the objects. For example, the depths may be used to control augmented reality, robotics, natural user interface technology, gaming and other applications. Streams of stereo images, or single stereo images, obtained with or without patterns of illumination projected onto the environment are processed using a parallel-processing unit to obtain depth maps. In various embodiments a parallel-processing unit propagates values related to depth in rows or columns of a disparity map in parallel. In examples, the values may be propagated according to a measure of similarity between two images of a stereo pair; propagation may be temporal between disparity maps of frames of a stream of stereo images and may be spatial within a left or right disparity map.