Abstract: A mixed reality system that includes a device and a base station that communicate via a wireless connection The device may include sensors that collect information about the user's environment and about the user. The information collected by the sensors may be transmitted to the base station via the wireless connection. The base station renders frames or slices based at least in part on the sensor information received from the device, encodes the frames or slices, and transmits the compressed frames or slices to the device for decoding and display. The base station may provide more computing power than conventional stand-alone systems, and the wireless connection does not tether the device to the base station as in conventional tethered systems. The system may implement methods and apparatus to maintain a target frame rate through the wireless link and to minimize latency in frame rendering, transmittal, and display.

Abstract: A video encoding system in which pixel data is decomposed into frequency bands prior to encoding. The frequency bands for a slice of a frame may be buffered so that complexity statistics may be calculated across the frequency bands prior to encoding. The statistics may then be used by a rate control component in determining quantization parameters for the frequency bands for modulating the rate in the encoder for the current slice. The quantization parameters for the frequency bands may be calculated jointly to optimize the quality of the displayed frames after decoder reconstruction and wavelet synthesis on a receiving device. Information about one or more previously processed frames may be used in combination with the statistics for a current slice in determining the quantization parameters for the current slice.

Abstract: Methods and systems include neural network-based image processing and blending circuitry to blend an output of the neural network to compensate for potential artifacts from the neural network-based image processing. The neural network(s) apply image processing to image data using one or more neural networks as processed data. Enhance circuitry enhances the image data in a scaling circuitry to generate enhanced data. Blending circuitry receives the processed image data and the enhanced data along with an image plane of the processed data. The blending circuitry also determines whether the image processing using the one or more neural networks has applied a change to the image data greater than a threshold amount. The blending circuitry then, based at least in part in response to the change being greater than the threshold amount and/or edge information of the image data, blends the processed data with the enhanced data.

Abstract: A device may include an electronic display to display an image frame based on blended image data and image processing circuitry to generate the blended image data by combining first image data and second image data via a blend operation. The blend operation may include receiving graphics alpha data indicative of a transparency factor to be applied to the first image data to generate a first layer of the blend operation. The blend operation may also include overlaying the first layer onto a second layer that is based on the second image data. Overlaying the first layer onto the second layer may include adding first pixels values of the first image data that include negative pixel values and are augmented by the transparency factor to second pixel values of the second image data to generate blended pixel values of the blended image data.

Abstract: An electronic device may include an electronic display to display an image based on processed image data. The electronic device may also include image processing circuitry to generate the processed image data. The image processing circuitry may receive input image data corresponding to an image in a first perspective and warp the input image data from the first perspective to a second perspective, generating warped image data. Additionally, the image processing circuitry may determine one or more occluded regions in the second perspective and determine fill-data corresponding to the occluded regions. The processed image data may be generated by combining the warped image data and the fill-data.

Abstract: A device may include a display that display an image frame that is divided into adjustable regions having respective resolutions based on compensated image data. The device may also include image processing circuitry to generate the compensated image data by applying gains that compensate for burn-in related aging of pixels of the display. The gains are based on an aggregation of history updates indicative of estimated amounts of aging associated with pixel utilization. The circuitry may generate a history update by obtaining boundary data indicative of the boundaries between the adjustable regions, determining an estimated amount of aging, and dynamically resampling the estimated amount of aging by resampling a portion of the estimated amount of aging corresponding to an adjustable region by a factor and resampling of a different portion of the estimated amount of aging corresponding to another adjustable region by a different factor based on the boundary data.

Abstract: A video encoding system in which pixel data is decomposed into frequency bands prior to encoding. The frequency bands are organized into blocks that are provided to a block-based encoder. The encoded frequency data is packetized and transmitted to a receiving device. On the receiving device, the encoded data is decoded to recover the frequency bands. Wavelet synthesis is then performed on the frequency bands to reconstruct the pixel data for display. The system may encode parts of frames (tiles or slices) using one or more encoders and transmit the encoded parts as they are ready. A pre-filter component may perform a lens warp on the pixel data prior to the wavelet transform.

Abstract: An electronic device may include scaling circuitry to scale input pixel data to a greater resolution. The directional scaling circuitry may include first interpolation circuitry to receive best mode data, including one or more angles corresponding to content of the image and interpolate first pixel values at first pixel positions diagonally offset from input pixel positions of the input pixel data based on the best mode data and input pixel values corresponding to the input pixel positions. The directional scaling circuitry may also include second interpolation circuitry to receive the best mode data and the input pixel values and interpolate second pixel values at second pixel positions horizontally or vertically offset from the input pixel positions based at least in part on the best mode data and the input pixel values.

Abstract: An electronic device may include an electronic display to display an image based on compensated image data in a panel space. The electronic device may also include image processing circuitry to generate the compensated image data. Further, generating the compensated image data may include determining a first inverse mapping of a pixel grid from the panel space to a rendering space and determining a forward mapping of the pixel grid from the rendering space to the panel space based on the first inverse mapping. The forward mapping may include corrections for multiple different warp operations stacked in a single warp operation. Additionally, the image processing circuitry may apply the forward mapping to input image data to generate the compensated image data.

Abstract: An electronic device may include an electronic display to display an image based on processed image data. The electronic device may also include image processing circuitry to generate the processed image data. The image processing circuitry may receive input image data corresponding to an image in a first perspective and warp the input image data from the first perspective to a second perspective, generating warped image data. Additionally, the image processing circuitry may determine one or more occluded regions in the second perspective and determine fill-data corresponding to the occluded regions. The processed image data may be generated by combining the warped image data and the fill-data.

Abstract: A device may include a display that display an image frame that is divided into adjustable regions having respective resolutions based on compensated image data. The device may also include image processing circuitry to generate the compensated image data by applying gains that compensate for burn-in related aging of pixels of the display. The gains are based on an aggregation of history updates indicative of estimated amounts of aging associated with pixel utilization. The circuitry may generate a history update by obtaining boundary data indicative of the boundaries between the adjustable regions, determining an estimated amount of aging, and dynamically resampling the estimated amount of aging by resampling a portion of the estimated amount of aging corresponding to an adjustable region by a factor and resampling of a different portion of the estimated amount of aging corresponding to another adjustable region by a different factor based on the boundary data.

Abstract: An electronic device may include scaling circuitry to scale input pixel data to a greater resolution. The directional scaling circuitry may include first interpolation circuitry to receive best mode data, including one or more angles corresponding to content of the image and interpolate first pixel values at first pixel positions diagonally offset from input pixel positions of the input pixel data based on the best mode data and input pixel values corresponding to the input pixel positions. The directional scaling circuitry may also include second interpolation circuitry to receive the best mode data and the input pixel values and interpolate second pixel values at second pixel positions horizontally or vertically offset from the input pixel positions based at least in part on the best mode data and the input pixel values.

Abstract: A device may include a display that display an image frame that is divided into adjustable regions having respective resolutions based on compensated image data. The device may also include image processing circuitry to generate the compensated image data by applying gains that compensate for burn-in related aging of pixels of the display. The gains are based on an aggregation of history updates indicative of estimated amounts of aging associated with pixel utilization. The circuitry may generate a history update by obtaining boundary data indicative of the boundaries between the adjustable regions, determining an estimated amount of aging, and dynamically resampling the estimated amount of aging by resampling a portion of the estimated amount of aging corresponding to an adjustable region by a factor and resampling of a different portion of the estimated amount of aging corresponding to another adjustable region by a different factor based on the boundary data.

Abstract: Systems and methods are provided for adding digital film grain to images and videos displayed by a display of an electronic device. An image processing circuitry of the electronic device may include hardware, such as display pipeline hardware, memory-to-memory scaler and rotator (MSR) hardware, and/or other possible hardware, that enables generation of film grain templates based on characteristics of a target film grain, pseudo-randomly samples the programmable template to fetch film grain values, scales the film grain values, and combines the scaled film grain values with values of pixels in an image frame.

Abstract: A device may include a display for displaying an image frame based on warped image data and image processing circuitry to generate the warped image data by warping input image data to account for one or more distortions associated with displaying the image. The image processing circuitry may include a two-stage cache architecture having an first cache and an second cache and warp the input image data by generating mapping data indicative of a warp between the input image space and the output image space and fetching the input image data to populate the first cache. Warping may also include populating the second cache with a grouping of pixel values from the first cache that are selected according to a sliding window that traverses the first cache based on the mapping data and interpolating between pixel values of the grouping to generate pixel values of the warped image data.

Abstract: A device may include an electronic display to display an image frame based on blended image data and image processing circuitry to generate the blended image data by combining first image data and second image data via a blend operation. The blend operation may include receiving graphics alpha data indicative of a transparency factor to be applied to the first image data to generate a first layer of the blend operation. The blend operation may also include overlaying the first layer onto a second layer that is based on the second image data. Overlaying the first layer onto the second layer may include adding first pixels values of the first image data that include negative pixel values and are augmented by the transparency factor to second pixel values of the second image data to generate blended pixel values of the blended image data.

Abstract: In one implementation, a method includes receiving a warped image representing simulated reality (SR) content (e.g., to be displayed in a display space), the warped image having a plurality of pixels at respective locations uniformly spaced in a grid pattern in a warped space, wherein the plurality of pixels are respectively associated with a plurality of respective pixel values and a plurality of respective scaling factors indicating a plurality of respective resolutions at a plurality of respective locations of the SR content (e.g., in the display space). The method includes processing the warped image in the warped space based on the plurality of respective scaling factors to generate a processed warped image and transmitting the processed warped image.

Abstract: Systems and methods are provided for using an optical crosstalk compensation (OXTC) block to compensate for optical crosstalk resulted from a combination of viewing angle change across field of view (FoV), color filter (CF) crosstalk, and the OLED various angle color shift (VACS) of a foveated electronic display. One or more two-dimensional (2D) OXTC factor maps are used to determine OXTC factors for input image data of the OXTC block, and the OXTC factors are updated on a per frame basis. Offset values are determined using a parallel architecture and used to determine the OXTC factors. Compensation weights are used to determine weighted OXTC factors to improve processing efficiency. Output image data are obtained by applying the weighted OXTC factors to the input image data.

Abstract: The present disclosure is directed towards image processing circuitry that applies temporal filtering to video image data along motion trajectories in the video image data. The temporal filtering may be applied along motion trajectories in the image data, by filtering source pixels by reference pixel values and the refined motion vectors. The temporal filtering circuitry may fetch source and reference pixel values based on received motion vectors from an encoding pipeline. Additionally, the temporal filtering circuitry may include a motion vector refinement block along with a temporal filtering block, such that the video image data may be filtered based on refined motion vectors and source and reference pixel values.

Abstract: Systems and methods for improving determination of encoded image data using a video encoding pipeline, which includes a first transcode engine that entropy encodes a first portion of a bin stream to determine a first bit stream including first encoded image data that indicates a first coding group row and that determines first characteristic data corresponding to the first bit stream to facilitate communicating a combined bit stream; and a second transcode engine that entropy encodes a second portion of the bin stream to determine a second bit stream including second encoded image data that indicates a second coding group row while the first transcode engine entropy encodes the first portion of the bin stream and that determines second characteristic data corresponding to the second bit stream to facilitate communicating the combined bit stream, which includes the first bit stream and the second bit stream, to a decoding device.