Abstract: Embodiments are disclosed for predicting, using neural networks, editing operations for application to a video sequence based on processing conversational messages by a video editing system. In particular, in one or more embodiments, the disclosed systems and methods comprise receiving an input including a video sequence and text sentences, the text sentences describing a modification to the video sequence, mapping, by a first neural network content of the text sentences describing the modification to the video sequence to a candidate editing operation, processing, by a second neural network, the video sequence to predict parameter values for the candidate editing operation, and generating a modified video sequence by applying the candidate editing operation with the predicted parameter values to the video sequence.

Abstract: The present disclosure describes multi-stage image editing techniques to improve detail and accuracy in edited images. An input image including a target region to be edited and an edit parameter specifying a modification to the target region are received. A parsing map of the input image is generated. A latent representation of the parsing map is generated. An edit is applied to the latent representation of the parsing map based on the edit parameter. The edited latent representation is input to a neural network to generate a modified parsing map including the target region with a shape change according to the edit parameter. Based on the input image and the modified parsing map, a masked image corresponding to the shape change is generated. Based on the masked image, a neural network is used to generate an edited image with the modification to the target region.

Abstract: A method includes accessing a first object in a virtual environment, the first object representing a first asset at a first level of detail (LoD). A second object is generated to represent the first asset at a second LoD having decreased complexity. The method further includes determining a first importance value for the first asset and, based on the first importance value, selecting the first object to represent the first asset. Additionally, the method includes accessing a third object representing the second asset at the first LoD and generating a fourth object representing the second asset at the second LoD. The method further includes determining a second importance value, lower than the first importance value, for the second asset and selecting the fourth object to represent the second asset. The method further includes causing a client device to update a display of the virtual environment by transmitting the selected objects.

Abstract: The context-aware optimization method includes training a context model by determining whether to split each node in the context by identifying a first subset of virtual context to evaluate by identifying a second subset of virtual contexts to evaluate and obtaining an encoding cost of splitting of the context model for each virtual context in the second subset and identifying the first subset of virtual contexts to evaluate by selecting a predetermined number of virtual contexts from the second subset based on the encoding cost such that the predetermined number of virtual contexts with lowest encoding cost are selected. The modified tree-traversal method includes encoding a mask or performing a speculative-based method. The modified entropy coding method includes representing data into an array of bits, using multiple coders to process each bit in the array and combining the output from the multiple coders into a data range.

Abstract: Embodiments described herein provide methods and systems for facilitating actively-learned context modeling. In one embodiment, a subset of data is selected from a training dataset corresponding with an image to be compressed, the subset of data corresponding with a subset of data of pixels of the image. A context model is generated using the selected subset of data. The context model is generally in the form of a decision tree having a set of leaf nodes. Entropy values corresponding with each leaf node of the set of leaf nodes are determined. Each entropy value indicates an extent of diversity of context associated with the corresponding leaf node. Additional data from the training dataset is selected based on the entropy values corresponding with the leaf nodes. The updated subset of data is used to generate an updated context model for use in performing compression of the image.

Abstract: Systems and methods for video processing are configured. Embodiments of the present disclosure receive a procedural document comprising a plurality of instructions; extract a plurality of key concepts for an instruction of the plurality of instructions; compute an information coverage distribution for each of a plurality of candidate multi-media assets, wherein the information coverage distribution indicates whether a corresponding multi-media asset relates to each of the plurality of key concepts; select a set of multi-media assets for the instruction based on the information coverage distribution; and generate a multi-media presentation describing the procedural document by combining the set of multi-media assets based on a presentation template.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer-readable media to enhance texture image delivery and processing at a client device. For example, the disclosed systems can utilize a server-side compression combination that includes, in sequential order, a first compression pass, a decompression pass, and a second compression pass. By applying this compression combination to a texture image at the server-side, the disclosed systems can leverage both GPU-friendly and network-friendly image formats. For example, at a client device, the disclosed system can instruct the client device to execute a combination of decompression-compression passes on a GPU-network-friendly image delivered over a network connection to the client device.

Abstract: A first device determines relative position data representative of a position of one or more other user devices relative to the first device. To determine relative position data between the first device and a second device, the first device determines a distance between the first device and the second device at a plurality of timestamps. Additionally, the first device determines movement data at each timestamp from one or more device sensors. The movement data at each corresponding timestamp may reflect movement of the first device and/or the second device between a prior timestamp and the corresponding timestamp. The first device computes relative position data for the second device by combining the distance measurements and movement data over the plurality of timestamps, for instance, through a process of sensor fusion.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer-readable media to enhance texture image delivery and processing at a client device. For example, the disclosed systems can utilize a server-side compression combination that includes, in sequential order, a first compression pass, a decompression pass, and a second compression pass. By applying this compression combination to a texture image at the server-side, the disclosed systems can leverage both GPU-friendly and network-friendly image formats. For example, at a client device, the disclosed system can instruct the client device to execute a combination of decompression-compression passes on a GPU-network-friendly image delivered over a network connection to the client device.

Abstract: A method includes accessing a first object in a virtual environment, the first object representing a first asset at a first level of detail (LoD). A second object is generated to represent the first asset at a second LoD having decreased complexity. The method further includes determining a first importance value for the first asset and, based on the first importance value, selecting the first object to represent the first asset. Additionally, the method includes accessing a third object representing the second asset at the first LoD and generating a fourth object representing the second asset at the second LoD. The method further includes determining a second importance value, lower than the first importance value, for the second asset and selecting the fourth object to represent the second asset. The method further includes causing a client device to update a display of the virtual environment by transmitting the selected objects.

Abstract: Techniques and systems are provided for generating a video from texture images, and for reconstructing the texture images from the video. For example, a texture image can be divided into a number of tiles, and the number of tiles can be sorted into a sequence of ordered tiles. The sequence of ordered tiles can be provided to a video coder for generating a coded video. The number of tiles can be encoded based on the sequence of ordered tiles. The encoded video including the encoded sequence of ordered tiles can be decoded. At least a portion of the decoded video can include the number of tiles sorted into a sequence of ordered tiles. A data file associated with at least the portion of the decoded video can be used to reconstruct the texture image using the tiles.

Abstract: In implementations for highlight video generated with adaptable multimodal customization, a multimodal detection system tracks activities based on poses and faces of persons depicted in video clips of video content. The system determines a pose highlight score and a face highlight score for each of the video clips that depict at least one person, the highlight scores representing a relative level of the interest in an activity depicted in a video clip. The system also determines pose-based emotion features for each of the video clips. The system can detect actions based on the activities of the persons depicted in the video clips, and detect emotions exhibited by the persons depicted in the video clips. The system can receive input selections of actions and emotions, and filter the video clips based on the selected actions and emotions. The system can then generate a highlight video of ranked and filtered video clips.

Abstract: Embodiments are disclosed for lossless image compression using block-based prediction and context adaptive entropy coding. A method of lossless image compression using block-based prediction and context adaptive entropy coding comprises dividing an input image into a plurality of blocks, determining a pixel predictor for each block based on a block strategy, determining a plurality of residual values using the pixel predictor for each block, selecting a subset of features associated with the plurality of residual values, performing context modeling on the plurality of residual values based on the subset of features to identify a plurality of residual clusters, and entropy coding the plurality of residual clusters.

Abstract: In implementations for highlight video generated with adaptable multimodal customization, a multimodal detection system tracks activities based on poses and faces of persons depicted in video clips of video content. The system determines a pose highlight score and a face highlight score for each of the video clips that depict at least one person, the highlight scores representing a relative level of the interest in an activity depicted in a video clip. The system also determines pose-based emotion features for each of the video clips. The system can detect actions based on the activities of the persons depicted in the video clips, and detect emotions exhibited by the persons depicted in the video clips. The system can receive input selections of actions and emotions, and filter the video clips based on the selected actions and emotions. The system can then generate a highlight video of ranked and filtered video clips.

Abstract: Systems and methods predict a performance metric for a video and identify key portions of the video that contribute to the performance metric, which can be used to edit the video to improve the ultimate viewer response to the video. An initial performance metric is computed for an initial video (e.g., using a neural network). A perturbed video is generated by perturbing a video portion of the initial video. A modified performance metric is computed for the perturbed video. Based on a difference between the initial and modified performance metrics, the system determines that the video portion contributed to a predicted user viewer response to the initial video. An indication of the video portion that contributed to the predicted user viewer response is provided as output, which can be used to edit the video to improve the predicted viewer response.

Abstract: Embodiments are disclosed for lossless image compression using block-based prediction and context adaptive entropy coding. A method of lossless image compression using block-based prediction and context adaptive entropy coding comprises dividing an input image into a plurality of blocks, determining a pixel predictor for each block based on a block strategy, determining a plurality of residual values using the pixel predictor for each block, selecting a subset of features associated with the plurality of residual values, performing context modeling on the plurality of residual values based on the subset of features to identify a plurality of residual clusters, and entropy coding the plurality of residual clusters.

Abstract: A first device determines relative position data representative of a position of one or more other user devices relative to the first device. To determine relative position data between the first device and a second device, the first device determines a distance between the first device and the second device at a plurality of timestamps. Additionally, the first device determines movement data at each timestamp from one or more device sensors. The movement data at each corresponding timestamp may reflect movement of the first device and/or the second device between a prior timestamp and the corresponding timestamp. The first device computes relative position data for the second device by combining the distance measurements and movement data over the plurality of timestamps, for instance, through a process of sensor fusion.

Abstract: Embodiments are disclosed for lossless image compression using block-based prediction and context adaptive entropy coding. A method of lossless image compression using block-based prediction and context adaptive entropy coding comprises dividing an input image into a plurality of blocks, determining a pixel predictor for each block based on a block strategy, determining a plurality of residual values using the pixel predictor for each block, selecting a subset of features associated with the plurality of residual values, performing context modeling on the plurality of residual values based on the subset of features to identify a plurality of residual clusters, and entropy coding the plurality of residual clusters.

Abstract: In implementations of systems for digital image compression using context-based pixel predictor selection, a computing device implements a compression system to receive digital image data describing pixels of a digital image. The compression system groups first differences between values of the pixels and first prediction values of the pixels into context groups. A pixel predictor is determined for each of the context groups based on a compression criterion. The compression system generates second prediction values of the pixels using the determined pixel predictor for pixels corresponding to the first differences included in each of the context groups. Second differences between the values of the pixels and the second prediction values of the pixels are grouped into different context groups. The compression system compresses the digital image using entropy coding based on the different context groups.

Abstract: Methods, system, and computer storage media are provided for novel view synthesis. An input image depicting an object is received and utilized to generate, via a neural network, a target view image. In exemplary aspects, additional view images are also generated within the same pass of the neural network. A loss is determined based on the target view image and additional view images and is used to modify the neural network to reduce errors. In some aspects, a rotated view image is generated by warping a ground truth image from an initial angle to a rotated view angle that matches a view angle of an image synthesized via the neural network, such as a target view image. The rotated view image and the synthesized image matching the rotated view angle (e.g., a target view image) are utilized to compute a rotational loss.