Abstract: A computing system and method that can be used for safe and privacy preserving video representations of participants in a videoconference. In particular, the present disclosure provides a general pipeline for generating reconstructions of videoconference participants based on semantic statuses and/or activity statuses of the participants. The systems and methods of the present disclosure allow for videoconferences that convey necessary or meaningful information of participants through presentation of generalized representations of participants while filtering unnecessary or unwanted information from the representations by leveraging machine-learning models.

Abstract: Systems, apparatus, and methods are presented for improving the composition of images. One method includes receiving a first preview image of a scene and processing the first preview image using a first machine learning model to determine at least one direction to move a user device based on a composition score of the first preview image and at least one composition score of a plurality of candidate images. The method also includes detecting movement of the user device and receiving a second preview image of the scene after movement of the user device.

Abstract: Systems and methods for multi-attendee video conferencing are described. A system can convert from huddle video conference mode to spatial video conference mode. In particular, by assigning user roles, specific users can have greater control of the video conference as compared to other users. For instance, moderators may have a greater level of control of the video conferencing system. Thus, in example implementations of the present disclosure, specific users can affect transition between two or more video conferencing modes, such as between a huddle video conference mode and a spatial video conference mode.

Abstract: Described are methods, systems, and computer-readable media to automatically crop videos using personalized parameters. Some implementations include a computer-implemented method that comprises obtaining an input video, determining a per-frame crop score for one or more candidate crop regions in each frame of the input video, generating a face signal for the one or more candidate crop regions, adjusting each per-frame crop score based on the face signal, determining a minimal cost path that represents crop region locations based on motion cost and the adjusted per-frame crop score, generating crop keyframing corresponding to the crop region locations along the minimal cost path, wherein the crop keyframing includes a start frame, an end frame, and crop region location, and outputting a modified video that has one or more of an output aspect ratio or an output orientation that is different than a corresponding aspect ratio or an orientation of the input video.

Abstract: Systems and methods for multi-attendee video conferencing are described. A system can convert from huddle video conference mode to spatial video conference mode. In particular, by assigning user roles, specific users can have greater control of the video conference as compared to other users. For instance, moderators may have a greater level of control of the video conferencing system. Thus, in example implementations of the present disclosure, specific users can affect transition between two or more video conferencing modes, such as between a huddle video conference mode and a spatial video conference mode.

Abstract: A computing system and method that can be used for safe and privacy preserving video representations of participants in a videoconference. In particular, the present disclosure provides a general pipeline for generating reconstructions of videoconference participants based on semantic statuses and/or activity statuses of the participants. The systems and methods of the present disclosure allow for videoconferences that convey necessary or meaningful information of participants through presentation of generalized representations of participants while filtering unnecessary or unwanted information from the representations by leveraging machine-learning models.

Abstract: Systems and methods for multi-attendee video conferencing are described. A system can convert from huddle video conference mode to spatial video conference mode. In particular, by assigning user roles, specific users can have greater control of the video conference as compared to other users. For instance, moderators may have a greater level of control of the video conferencing system. Thus, in example implementations of the present disclosure, specific users can affect transition between two or more video conferencing modes, such as between a huddle video conference mode and a spatial video conference mode.

Abstract: Systems and methods for multi-attendee video conferencing are described. A system can convert from huddle video conference mode to spatial video conference mode. In particular, by assigning user roles, specific users can have greater control of the video conference as compared to other users. For instance, moderators may have a greater level of control of the video conferencing system. Thus, in example implementations of the present disclosure, specific users can affect transition between two or more video conferencing modes, such as between a huddle video conference mode and a spatial video conference mode.

Abstract: Devices and methods related to capturing 360 degree spherical images are provided. A computing device can capture, substantially simultaneously, a first image with the first image sensor and a second image with a second image sensor. The first image sensor can be positioned on a first side of the computing device and oriented at a first orientation with respect to an axis of rotation. The second image sensor can be positioned on a second side of the device substantially opposite the first side of the device and oriented at a second orientation that is axially rotated from the first orientation with respect to the axis of rotation. The computing device can stitch together the first image and the second image to create an output image that captures a 360 degree field of view with respect to the computing device.

Abstract: Described are methods, systems, and computer-readable media to automatically crop videos using personalized parameters. Some implementations include a computer-implemented method that comprises obtaining an input video, determining a per-frame crop score for one or more candidate crop regions in each frame of the input video, generating a face signal for the one or more candidate crop regions, adjusting each per-frame crop score based on the face signal, determining a minimal cost path that represents crop region locations based on motion cost and the adjusted per-frame crop score, generating crop keyframing corresponding to the crop region locations along the minimal cost path, wherein the crop keyframing includes a start frame, an end frame, and crop region location, and outputting a modified video that has one or more of an output aspect ratio or an output orientation that is different than a corresponding aspect ratio or an orientation of the input video.

Abstract: Described are methods, systems, and computer-readable media to automatically crop videos using personalized parameters. Some implementations include a computer-implemented method that comprises obtaining an input video, determining a per-frame crop score for one or more candidate crop regions in each frame of the input video, generating a face signal for the one or more candidate crop regions, adjusting each per-frame crop score based on the face signal, determining a minimal cost path that represents crop region locations based on motion cost and the adjusted per-frame crop score, generating crop keyframing corresponding to the crop region locations along the minimal cost path, wherein the crop keyframing includes a start frame, an end frame, and crop region location, and outputting a modified video that has one or more of an output aspect ratio or an output orientation that is different than a corresponding aspect ratio or an orientation of the input video.

Abstract: Described are methods, systems, and computer-readable media to automatically crop videos using personalized parameters. Some implementations include a computer-implemented method that comprises obtaining an input video, determining a per-frame crop score for one or more candidate crop regions in each frame of the input video, generating a face signal for the one or more candidate crop regions, adjusting each per-frame crop score based on the face signal, determining a minimal cost path that represents crop region locations based on motion cost and the adjusted per-frame crop score, generating crop keyframing corresponding to the crop region locations along the minimal cost path, wherein the crop keyframing includes a start frame, an end frame, and crop region location, and outputting a modified video that has one or more of an output aspect ratio or an output orientation that is different than a corresponding aspect ratio or an orientation of the input video.

Abstract: A light-field camera system such as a tiled camera array may be used to capture a light-field of an environment. The tiled camera array may be a tiered camera array with a first plurality of cameras and a second plurality of cameras that are arranged more densely, but have lower resolution, than those of the first plurality of cameras. The first plurality of cameras may be interspersed among the second plurality of cameras. The first and second pluralities may cooperate to capture the light-field. According to one method, a subview may be captured by each camera of the first and second pluralities. Estimated world properties of the environment may be computed for each subview. A confidence map may be generated to indicate a level of confidence in the estimated world properties for each subview. The confidence maps and subviews may be used to generate a virtual view of the environment.

Abstract: A light-field camera may generate four-dimensional light-field data indicative of incoming light. The light-field camera may have an aperture configured to receive the incoming light, an image sensor, and a microlens array configured to redirect the incoming light at the image sensor. The image sensor may receive the incoming light and, based on the incoming light, generate the four-dimensional light-field data, which may have first and second spatial dimensions and first and second angular dimensions. The first angular dimension may have a first resolution higher than a second resolution of the second angular dimension.

Abstract: A video stream of a scene for a virtual reality or augmented reality experience may be captured by one or more image capture devices. Data from the video stream may be retrieved, including base vantage data with base vantage color data depicting the scene from a base vantage location, and target vantage data with target vantage color data depicting the scene from a target vantage location. The base vantage data may be reprojected to the target vantage location to obtain reprojected target vantage data. The reprojected target vantage data may be compared with the target vantage data to obtain residual data. The residual data may be compressed by removing a subset of the residual data that is likely to be less viewer-discernable than a remainder of the residual data. A compressed video stream may be stored, including the base vantage data and the compressed residual data.

Abstract: A virtual reality or augmented reality experience of a scene may be presented to a viewer using layered data retrieval and/or processing. A first layer of a video stream may be retrieved, and a first viewer position and/or orientation may be received. The first layer may be processed to generate first viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation. The first viewpoint video may be displayed for the viewer. Then, a second layer of the video stream may be retrieved, and a second viewer position and/or orientation may be received. The second layer may be processed to generate second viewpoint video of the scene from a second virtual viewpoint corresponding to the second viewer position and/or orientation, with higher quality than the first viewpoint video. The second viewpoint video may be displayed for the viewer.

Abstract: A light-field camera system such as a tiled camera array may be used to capture a light-field of an environment. The tiled camera array may be a tiered camera array with a first plurality of cameras and a second plurality of cameras that are arranged more densely, but have lower resolution, than those of the first plurality of cameras. The first plurality of cameras may be interspersed among the second plurality of cameras. The first and second pluralities may cooperate to capture the light-field. According to one method, a subview may be captured by each camera of the first and second pluralities. Estimated world properties of the environment may be computed for each subview. A confidence map may be generated to indicate a level of confidence in the estimated world properties for each subview. The confidence maps and subviews may be used to generate a virtual view of the environment.

Abstract: A virtual reality or augmented reality experience of a scene may be decoded for playback for a viewer through a combination of CPU and GPU processing. A video stream may be retrieved from a data store. A first viewer position and/or orientation may be received from an input device, such as the sensor package on a head-mounted display (HMD). At a processor, the video stream may be partially decoded to generate a partially-decoded bitstream. At a graphics processor, the partially-decoded bitstream may be further decoded to generate viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation. The viewpoint video may be displayed on a display device, such as screen of the HMD.

Abstract: A video stream for a scene for a virtual reality or augmented reality experience may be stored and delivered to a viewer. The video stream may be divided into a plurality of units based on time segmentation, viewpoint segmentation, and/or view orientation segmentation. Each of the units may be divided into a plurality of sub-units based on a different segmentation from the units, via time segmentation, viewpoint segmentation, and/or view orientation segmentation. At least a portion of the video stream may be stored in a file that includes a plurality of the units. Each unit may be a group of pictures that is a sequence of successive frames in time. Each sub-unit may be a vantage defining a viewpoint from which the scene is viewable. Each vantage may be further divided into tiles, each of which is part of the vantage, limited to one or more particular view orientations.

Abstract: A light-field camera system such as a tiled camera array may be used to capture a light-field of an environment. The tiled camera array may be a tiered camera array with a first plurality of cameras and a second plurality of cameras that are arranged more densely, but have lower resolution, than those of the first plurality of cameras. The first plurality of cameras may be interspersed among the second plurality of cameras. The first and second pluralities may cooperate to capture the light-field. According to one method, a subview may be captured by each camera of the first and second pluralities. Estimated world properties of the environment may be computed for each subview. A confidence map may be generated to indicate a level of confidence in the estimated world properties for each subview. The confidence maps and subviews may be used to generate a virtual view of the environment.