Abstract: A machine learning model is used to receive a recorded video game for presentation on a spectator computer and to derive from the video identifications of controller operations during play of the game that resulted in the recorded video game. Indications of identified controller operations may be presented with the recorded video game to assist a viewer in learning how to play the game.

Abstract: Application context may be interpolated between application state updates from structured and unstructured application state data. Irrelevant unimodal modules may be deactivated based on the structured application state data while relevant unimodal modules remain active. Unimodal features are generated from the unstructured application using the relevant modules. A neural module selection network module may be trained with a machine learning algorithm. Each unimodal modules may generate unimodal feature vectors from unstructured application data. A context state update module may determine which unimodal modules are irrelevant from structured application state data and deactivate the irrelevant modules but not the relevant ones. A multimodal neural network may take the active unimodal feature vectors and predict structured context data and send it to a uniform data system.

Abstract: A system for generating gameplay context information for a game may include a game screen classification module trained to classify contextually relevant data from gameplay data, one or more game object recognition modules trained to detect game icons from gameplay data, and a multimodal context generation neural network module trained to generate structured gameplay context information from the contextually relevant data and icons within the gameplay data. The multimodal context generation neural network module at least partially generates structured gameplay context information. The modules may include neural networks trained by suitable machine learning algorithms using suitable masked data and labeled data.

Abstract: An image of a computer game player wearing a headset that occludes part of the face is input to a trained machine learning (ML) model, which outputs in response a full-face image that is not occluded for use in, e.g., social network settings related to the game.

Abstract: A machine learning (ML) model is trained using pairs of images. Each pair includes an image of a human face and a duplicate of the image with a computer game headset overlaid on the face using computer graphics. The ML model subsequently can be used to receive an image of a gamer wearing a headset and output a full-face image of the gamer for use in, e.g., social network settings related to the game.

Abstract: Responsive to a zoom command when presenting a first video, a second video is combined with the first video and presented. The first and second videos are generated from substantially the same camera location as each other at substantially the same time with substantially the same resolution. However, the second video is generated by a physical or virtual lens having a field of view (FOV) smaller than the FOV of a physical or virtual lens used in generating the first video. The technique gives the appearance of zooming without loss of resolution.

Abstract: Responsive to a zoom command when presenting a first video, a second video is combined with the first video and presented. The first and second videos are generated from substantially the same camera location as each other at substantially the same time with substantially the same resolution. However, the second video is generated by a physical or virtual lens having a field of view (FOV) smaller than the FOV of a physical or virtual lens used in generating the first video. Modules are described for using alignment metrics to correctly place the second video over the inner video and make it appear seamless.

Abstract: Responsive to a zoom command when presenting a first video, a second video is combined with the first video and presented. The first and second videos are generated from substantially the same camera location as each other at substantially the same time with substantially the same resolution. However, the second video is generated by a physical or virtual lens having a field of view (FOV) smaller than the FOV of a physical or virtual lens used in generating the first video. The technique gives the appearance of zooming without loss of resolution.

Abstract: Video stream data is selectively scaled so that sections within regions of interest (ROI) maintain high resolution while areas not within the region of interest are down-scaled to reduce bandwidth cost of transmission. A low compression encoder compresses sections of a video frame corresponding to one or more ROI without motion search or prediction mode decision to generate low-compression section data. The video frame is downscaled and a high compression encoder compresses the resulting downscaled video frame with prediction mode decision to generate high-compression frame data.

Abstract: A machine learning model is used to receive a recorded video game for presentation on a spectator computer and to derive from the video identifications of controller operations during play of the game that resulted in the recorded video game. Indications of identified controller operations may be presented with the recorded video game to assist a viewer in learning how to play the game.

Abstract: A method, system, and computer readable medium for encoding one or more input digital frames with obscured negative regions of interest. The method comprising, comparing frame data to mask data for one or more sections of an input digital frame to determine whether or not a given section of the input digital frame is to be obscured, setting one or more encoding parameters to values that result in corresponding obscured sections when decoded after encoding for each section of the frame that is to be obscured, encoding the one or more sections of the input digital frame to generate corresponding encoded frame data and storing or transmitting the encoded frame data.

Abstract: A method, system and computer readable instructions for video encoding comprising, determining one or more region of interest (ROI) parameters for pictures in a picture stream and a temporal down sampling interval. One or more areas outside the ROI in a picture in the picture stream are temporally down sampled according to the interval. The resulting temporally down sampled picture is then encoded and the encoded temporally down-sampled picture is transmitted. Additionally, a picture encoded in this way in an encoded picture stream may be decoded and areas outside an ROI of the picture may be temporally up sampled. The temporally up sampled areas outside the ROI are inserted into the decoded encoded picture stream.

Abstract: Some embodiments provide methods of playing back content, comprising: accessing video content comprising a series of frames that if fully decoded would extend beyond a viewer's field of view, and wherein each encoded frame comprises multiple encoded sections; determining a field of view of the viewer; identifying one or more sections of the first frame that are at least partially within the field of view; decoding the one or more sections of the first frame while not decoding one or more of the sections of the first frame that are not within the field of view; and displaying the one or more decoded sections of the first frame such that the portion of the first frame is displayed, and wherein less than all of the first frame is decoded and less than all of the first frame is displayed during playback.

Abstract: Video stream data is selectively scaled so that sections within regions of interest (ROI) maintain high resolution while areas not within the region of interest are down-scaled to reduce bandwidth cost of transmission. A low compression encoder compresses sections of a video frame corresponding to one or more ROI without motion search or prediction mode decision to generate low-compression section data. The video frame is downscaled and a high compression encoder compresses the resulting downscaled video frame with prediction mode decision to generate high-compression frame data.

Abstract: Gaze tracking data is analyzed to determine one or more regions of interest within an image of a video stream. The video stream data is selectively scaled so that sections within the regions of interest maintain high resolution while areas not within the region of interest are down-scaled to reduce bandwidth cost of transmission. A scheme for reduction of motion sickness by reducing the size of the high resolution area is also claimed.

Abstract: A method, system and computer readable instructions for video encoding comprising, determining one or more region of interest (ROI) parameters for pictures in a picture stream and a temporal down sampling interval. One or more areas outside the ROI in a picture in the picture stream are temporally down sampled according to the interval. The resulting temporally down sampled picture is then encoded and the encoded temporally down-sampled picture is transmitted. Additionally, a picture encoded in this way in an encoded picture stream may be decoded and areas outside an ROI of the picture may be temporally up sampled. The temporally up sampled areas outside the ROI are inserted into the decoded encoded picture stream.

Abstract: In a method for video encoding, one or more parameters are determined for a region of interest (ROI) within a digital image. The parameters relate to a size, location, and shape of the ROI. Multi-segment downsampling is performed on the input image to generate a downsampled image having fewer pixels than the digital image. The downsampling uses a different sample density for the ROI than for a portion of the digital image outside the ROI. The downsampled image is encoded to generate encoded image data and the encoded image data is combined with the one or more parameters to generate combined data, which may then be transmitted or stored. A method for video decoding comprising decoding the encoded video and performing multi-segment upsampling on the decoded video is also disclosed.

Abstract: Input digital frames may be down-sampled to create one or more base frames characterized by a lower resolution than the input digital frames. Enhancement information corresponding to a difference between pixel values for the one or more input digital frames and corresponding pixel values of up-sampled versions of the one or more base frames is then created. The one base frames are encoded to form a set of base data and the enhancement information is encoded to form a set of enhancement data. The base data and enhancement data may then be transmitted over a network or stored in a memory.

Abstract: In a method for video encoding, one or more parameters are determined for a region of interest (ROI) within a digital image. The parameters relate to a size, location, and shape of the ROI. Multi-segment downsampling is performed on the input image to generate a downsampled image having fewer pixels than the digital image. The downsampling uses a different sample density for the ROI than for a portion of the digital image outside the ROI. The downsampled image is encoded to generate encoded image data and the encoded image data is combined with the one or more parameters to generate combined data, which may then be transmitted or stored. A method for video decoding comprising decoding the encoded video and performing multi-segment upsampling on the decoded video is also disclosed.

Abstract: Some embodiments provide methods of playing back content, comprising: accessing video content comprising a series of frames that if fully decoded would extend beyond a viewer's field of view, and wherein each encoded frame comprises multiple encoded sections; determining a field of view of the viewer; identifying one or more sections of the first frame that are at least partially within the field of view; decoding the one or more sections of the first frame while not decoding one or more of the sections of the first frame that are not within the field of view; and displaying the one or more decoded sections of the first frame such that the portion of the first frame is displayed, and wherein less than all of the first frame is decoded and less than all of the first frame is displayed during playback.