Abstract: In one embodiment, a method includes receiving a user's eye gaze and using the eye gaze to determine a point of interest. The method further includes determining a current region of interest based on the point of interest, determining a temporal stability of the eye gaze by comparing the current region of interest to a past region of interest, and assigning a weight to the region of interest. The method further includes using the weight and region of interest to compute an exposure control parameter, and instructing a camera to capture an image with the exposure control parameter.

Abstract: In one embodiment, a computing system may determine a gaze of a user of the computing system. The computing system may generate a foveated map based on the gaze to determine a sensor readout for an image sensor of the computing system. The foveated map may include several foveal regions. The computing system may determine the sensor readout including several zones corresponding to the image sensor based on the several foveal regions. Each of the several zones may indicate a readout resolution for an area of the image sensor for the respective zone. The computing system may capture a first image using the image sensor. The computing system may generate a modified first image based on the captured first image and the sensor readout.

Abstract: A method implemented by a computing device includes rendering on displays of a computing device an extended reality (XR) environment, and determining a context of the XR environment with respect to a user. Determining the context includes determining characteristics associated with an eye of the user with respect to content displayed. The method includes generating, based on the characteristics associated with the eye, a foveated map including a plurality of foveal regions. The plurality of foveal regions includes a plurality of zones each corresponding to a low-resolution area of the content for the respective zone. The method includes inputting one or more of the plurality of zones into a machine-learning model trained to generate a super-resolution reconstruction of the foveated map based on regions of interest identified within the one or more of the plurality of zones, and outputting, by the machine-learning model, the super-resolution reconstruction of the foveated map.

Abstract: Methods of video streaming are generally described. In some examples, a camera device periodically captures an image, communicates encoded frame data representing that image to a server, and decodes and stores the previously encoded frame data as a background picture. The server receives the encoded frame data, decodes it, and stores the decoded frame in a buffer for future use. Subsequently, upon initiation of a streaming session, the camera device captures another image and encodes a predicted key frame based on differences between the captured image and the background picture. The camera device sends the predicted key frame to the server, which receives it and reconstructs a facsimile of the captured image utilizing the previously decoded frame stored in the buffer. Methods of acknowledging successfully decoded frames for use in selecting background pictures is also described.

Abstract: Methods and systems for improved quality of a streaming session using multiple simultaneous streams. For the multiple simultaneous streams an audio/video device (A/V device) records and generates a high-resolution stream and a low-resolution stream for simultaneous transmission to a server. The server selects one of the two streams for retransmission to a destination client device. The server also monitors the streaming session and estimates a total available bandwidth between the server and the A/V device and assigns a confidence value to the bandwidth estimation. The server periodically transmits the bandwidth estimate and confidence value to the A/V device to improve the efficiency of the streams being generated by the A/V device. The A/V device can use the received bandwidth estimate and confidence value to adapt the resolution of each of the streams to efficiently use the total available bandwidth between the A/V device and the server.

Abstract: In particular embodiments, a computing system may receive a color image captured by a color camera and a monochrome image captured by a monochrome camera. The color camera and the monochrome camera are associated with an artificial reality system. The computing system may compute, for each of the color and monochrome images, histogram statistics and perform, based on the histogram statistics, tone map matching to normalize the monochrome image with respect to the color image. The computing system may perform local motion estimation to calculate motion vectors indicating pixel correspondence between the normalized monochrome image and the color image. The computing system may generate a mono-color merged image for display on the artificial reality system by adding, for each pixel in the normalized monochrome image, color information extracted from corresponding pixel in the color image using the motion vectors.

Abstract: Methods and systems for improved quality of a streaming session using multiple simultaneous streams. For the multiple simultaneous streams an audio/video device (A/V device) records and generates a high-resolution stream and a low-resolution stream for simultaneous transmission to a server. The server selects one of the two streams for retransmission to a destination client device. The server also monitors the streaming session and estimates a total available bandwidth between the server and the A/V device and assigns a confidence value to the bandwidth estimation. The server periodically transmits the bandwidth estimate and confidence value to the A/V device to improve the efficiency of the streams being generated by the A/V device. The A/V device can use the received bandwidth estimate and confidence value to adapt the resolution of each of the streams to efficiently use the total available bandwidth between the A/V device and the server.

Abstract: An embodiment of a camera device may include an optical filter. The camera device may attempt to actuate the optical filter from an inactive position to an active position, record video, count a quantity of pixels, and/or a quantity of lines of pixels, in one or more frames of the video that are a color that is characteristic of the absence of the optical filter, determine that the quantity is greater than a threshold, and again attempt to move the optical filter to its active position.

Abstract: Described herein are techniques related to multi-pass quantization and bit-packing (PAK) parameters algorithm in video encoding.

Abstract: Approaches to enable a computing device, such as a phone or tablet computer, to utilize a number of cues to detect the presence of a face (or head) in an image captured by a camera of the computing device. The cues may include the elliptical shape of the face, the stereo disparity signature of the face, color or image brightness, among others. The facial detection may be performed by using a boosted classifier ensemble that has been trained using a plurality of images that are known (i.e., have been previously identified) to contain faces. The classifier ensemble can combine a number of different cues, such as the elliptical shape and/or stereo disparity signature, to be used for detecting faces within an image.

Abstract: Described herein are techniques related to multi-pass quantization and bit-packing (PAK) parameters algorithm in video encoding.

Abstract: Methods and systems to determine a quantization parameter (QP) that may then be used in a bit rate control process during video compression and decompression. The QP for a frame may be derived by determining an initial QP, adapting this value, and applying a final control calculation. The QP for a frame may be calculated from a variety of variables, that may include the frame type (intracoded (I), predictive (P), or bipredictive (B)), the fullness of a hypothetical reference decoder (HRD) buffer, the instant and target bit rates, a total encode size, a total target size, the number of P frames between I frames in a group of pictures (GOP), the number of B frames between I frames in the GOP, and the size of previous frames.

Abstract: Methods and systems to determine a quantization parameter (QP) that may then be used in a bit rate control process during video compression and decompression. The QP for a frame may be derived by determining an initial QP, adapting this value, and applying a final control calculation. The QP for a frame may be calculated from a variety of variables, that may include the frame type (intracoded (I), predictive (P), or bipredictive (B)), the fullness of a hypothetical reference decoder (HRD) buffer, the instant and target bit rates, a total encode size, a total target size, the number of P frames between I frames in a group of pictures (GOP), the number of B frames between I frames in the GOP, and the size of previous frames.

Abstract: Methods, apparatus, and systems are provided for order-adaptive compression. A sting is iterated to produce a compression sequence relative to an alphabet having symbols associated with the string. During the iteration, the order of the alphabet is altered in response to a given frequency condition.