Abstract: The subject disclosure is directed towards a framework that is configured to allow different background-foreground segmentation modalities to contribute towards segmentation. In one aspect, pixels are processed based upon RGB background separation, chroma keying, IR background separation, current depth versus background depth and current depth versus threshold background depth modalities. Each modality may contribute as a factor that the framework combines to determine a probability as to whether a pixel is foreground or background. The probabilities are fed into a global segmentation framework to obtain a segmented image.

Abstract: The subject disclosure is directed towards controlling the intensity of illumination of a scene or part of a scene, including to conserve illumination power. Quality of depth data in stereo images may be measured with different illumination states; environmental conditions, such as ambient light, natural texture may affect the quality. The illumination intensity may be controllably varied to obtain sufficient quality while conserving power. The control may be directed to one or more regions of interest corresponding to an entire scene or part of a scene.

Abstract: A processor-implemented method for encoding a block-based volumetric video having a plurality of video frames of a 3D object into a 2D video format is provided. The method including (i) splitting each video frame of the plurality of video frames into a first region that includes RGB data, a second region that includes depth data, and at least a third region containing render metadata of the 3D object; and (ii) storing the render metadata of the 3D object in at least one of the first region that includes the RGB data, the second region that includes the depth data and the at least the third region in at least one channel that is selected from a U chroma channel, a V chroma channel, and a luma channel of the block-based volumetric video.

Abstract: A processor-implemented method of generating a three-dimensional (3D) volumetric video with an overlay representing visibility counts per pixel of a texture atlas, associated with a viewer telemetry data is provided. The method includes (i) capturing the viewer telemetry data, (ii) determining a visibility of each pixel in the texture atlas associated with a 3D content based on the viewer telemetry data, (iii) generating at least one visibility counts per pixel of the texture atlas based on the visibility of each pixel in the texture atlas, and (iv) generating one of: the 3D volumetric video with the overlay of at least one heat map associated with the viewer telemetry data, using the at least one visibility counts per pixel and a curated selection of the 3D volumetric content based on the viewer telemetry data, using the visibility counts per pixel.

Abstract: Systems and methods for stereo matching based upon active illumination using a patch in a non-actively illuminated image to obtain weights that are used in patch similarity determinations in actively illuminated stereo images is provided. To correlate pixels in actively illuminated stereo images, adaptive support weights computations are used to determine similarity of patches corresponding to the pixels. In order to obtain adaptive support weights for the adaptive support weights computations, weights are obtained by processing a non-actively illuminated (“clean”) image.

Abstract: A processor-implemented method of generating a three-dimensional (3D) volumetric video with an overlay representing visibility counts per pixel of a texture atlas, associated with a viewer telemetry data is provided. The method includes (i) capturing the viewer telemetry data, (ii) determining a visibility of each pixel in the texture atlas associated with a 3D content based on the viewer telemetry data, (iii) generating at least one visibility counts per pixel of the texture atlas based on the visibility of each pixel in the texture atlas, and (iv) generating one of: the 3D volumetric video with the overlay of at least one heat map associated with the viewer telemetry data, using the at least one visibility counts per pixel and a curated selection of the 3D volumetric content based on the viewer telemetry data, using the visibility counts per pixel.

Abstract: Systems and methods for stereo matching based upon active illumination using a patch in a non-actively illuminated image to obtain weights that are used in patch similarity determinations in actively illuminated stereo images is provided. To correlate pixels in actively illuminated stereo images, adaptive support weights computations are used to determine similarity of patches corresponding to the pixels. In order to obtain adaptive support weights for the adaptive support weights computations, weights are obtained by processing a non-actively illuminated (“clean”) image.

Abstract: A processor implemented method for compressing surface data of a 3 dimensional object in a global digital space, using an image encoder that supports an image data compression algorithm, the image encoder being coupled to a transmitter. The method includes the steps of (i) decomposing the surface data into at least one surface representation that is encoded in an oriented bounding box, (ii) transforming the oriented bounding box into a canonical camera representation to obtain canonical coordinates for the at least one surface representation, (iii) converting each of the at least one surface representation into at least one bounding box image pair that includes a grayscale image representing depth, and a color image and (iv) tiling the at least one bounding box image pair to produce a tiled bounding box image.

Abstract: The subject disclosure is directed towards a framework that is configured to allow different background-foreground segmentation modalities to contribute towards segmentation. In one aspect, pixels are processed based upon RGB background separation, chroma keying, IR background separation, current depth versus background depth and current depth versus threshold background depth modalities. Each modality may contribute as a factor that the framework combines to determine a probability as to whether a pixel is foreground or background. The probabilities are fed into a global segmentation framework to obtain a segmented image.

Abstract: A processor implemented method for compressing surface data of a 3 dimensional object in a global digital space, using an image encoder that supports an image data compression algorithm, the image encoder being coupled to a transmitter. The method includes the steps of (i) decomposing the surface data into at least one surface representation that is encoded in an oriented bounding box, (ii) transforming the oriented bounding box into a canonical camera representation to obtain canonical coordinates for the at least one surface representation, (iii) converting each of the at least one surface representation into at least one bounding box image pair that includes a grayscale image representing depth, and a color image and (iv) tiling the at least one bounding box image pair to produce a tiled bounding box image.

Abstract: A processor-implemented method for streaming visible blocks of volumetric video to a client device during a predefined time period is provided. The method includes (i) receiving at least one block description file from a content server, (ii) processing each block description in the at least one block description file, to determine the visible blocks that are selected from a set of blocks, that are capable of being visible to a viewer of the client device during the predefined time period, based on a 3D position, size, and an orientation of each block in the set of blocks and at least one view parameter of a user of the client device, (iii) transmitting a request for the visible blocks, to the content server, and (iv) receiving the visible blocks as a visible blocks video at the client device.

Abstract: A processor-implemented method for streaming visible blocks of volumetric video to a client device during a predefined time period is provided. The method includes (i) receiving at least one block description file from a content server, (ii) processing each block description in the at least one block description file, to determine the visible blocks that are selected from a set of blocks, that are capable of being visible to a viewer of the client device during the predefined time period, based on a 3D position, size, and an orientation of each block in the set of blocks and at least one view parameter of a user of the client device, (iii) transmitting a request for the visible blocks, to the content server, and (iv) receiving the visible blocks as a visible blocks video at the client device.

Abstract: The subject disclosure is directed towards color correcting for infrared (IR) components that are detected in the R, G, B parts of a sensor photosite. A calibration process determines true R, G, B based upon obtaining or estimating IR components in each photosite, such as by filtering techniques and/or using different IR lighting conditions. A set of tables or curves obtained via offline calibration model the correction data needed for online correction of an image.

Abstract: A processor implemented method for compressing time-varying surface data of a 3 dimensional object in a global digital space having frames, using a video encoder that supports a video data compression algorithm, the video encoder being coupled to a transmitter. The method includes the steps of (i) decomposing the time-varying surface data into at least one surface representation that is encoded in an oriented bounding box, (ii) transforming the oriented bounding box into a canonical camera representation for each frame to obtain canonical coordinates for the at least one surface representation, (iii) converting each of the at least one surface representation into at least one bounding box video pair that includes a grayscale video representing depth, and a color video and (iv) tiling the at least one bounding box video pair for each frame to produce a tiled bounding box video.

Abstract: A processor implemented method for compressing time-varying surface data of a 3 dimensional object in a global digital space having frames, using a video encoder that supports a video data compression algorithm, the video encoder being coupled to a transmitter. The method includes the steps of (i) decomposing the time-varying surface data into at least one surface representation that is encoded in an oriented bounding box, (ii) transforming the oriented bounding box into a canonical camera representation for each frame to obtain canonical coordinates for the at least one surface representation, (iii) converting each of the at least one surface representation into at least one bounding box video pair that includes a grayscale video representing depth, and a color video and (iv) tiling the at least one bounding box video pair for each frame to produce a tiled bounding box video.

Abstract: The subject disclosure is directed towards a high resolution, high frame rate, robust stereo depth system. The system provides depth data in varying conditions based upon stereo matching of images, including actively illuminated IR images in some implementations. A clean IR or RGB image may be captured and used with any other captured images in some implementations. Clean IR images may be obtained by using a notch filter to filter out the active illumination pattern. IR stereo cameras, a projector, broad spectrum IR LEDs and one or more other cameras may be incorporated into a single device, which may also include image processing components to internally compute depth data in the device for subsequent output.

Abstract: The automated camera array calibration technique described herein pertains to a technique for automating camera array calibration. The technique can leverage corresponding depth and single or multi-spectral intensity data (e.g., RGB (Red Green Blue) data) captured by hybrid capture devices to automatically determine camera geometry. In one embodiment it does this by finding common features in the depth maps between two hybrid capture devices and derives a rough extrinsic calibration based on shared depth map features. It then uses the intensity (e.g., RGB) data corresponding to the depth maps and uses the features of the intensity (e.g., RGB) data to refine the rough extrinsic calibration.

Abstract: The subject disclosure is directed towards communicating image-related data between a base station and/or one or more satellite computing devices, e.g., tablet computers and/or smartphones. A satellite device captures image data and communicates image-related data (such as the images or depth data processed therefrom) to another device, such as a base station. The receiving device uses the image-related data to enhance depth data (e.g., a depth map) based upon the image data captured from the satellite device, which may be physically closer to something in the scene than the base station, for example. To more accurately capture depth data in various conditions, an active illumination pattern may be projected from the base station or another external projector, whereby satellite units may use the other source's active illumination and thereby need not consume internal power to benefit from active illumination.

Abstract: The glancing angle exclusion technique described herein selectively limits projective texturing near depth map discontinuities. A depth discontinuity is defined by a jump between a near-depth surface and a far-depth surface. The claimed technique can limit projective texturing on near and far surfaces to a different degree—for example, the technique can limit far-depth projective texturing within a certain distance to a depth discontinuity but not near-depth projective texturing.

Abstract: The subject disclosure is directed towards controlling the intensity of illumination of a scene or part of a scene, including to conserve illumination power. Quality of depth data in stereo images may be measured with different illumination states; environmental conditions, such as ambient light, natural texture may affect the quality. The illumination intensity may be controllably varied to obtain sufficient quality while conserving power. The control may be directed to one or more regions of interest corresponding to an entire scene or part of a scene.