Abstract: Control data for a motion-constrained tile set (“MCTS”) indicates that inter-picture prediction processes within a specified set of tiles are constrained to reference only regions within the same set of tiles in previous pictures in decoding (or encoding) order. For example, a video encoder encodes multiple pictures partitioned into tiles to produce encoded data. The encoder outputs the encoded data along with control data (e.g., in a supplemental enhancement information message) that indicates that inter-picture prediction dependencies across tile set boundaries are constrained for a given tile set of one or more of the tiles. A video decoder or other tool receives the encoded data and MCTS control data, and processes the encoded data. Signaling and use of MCTS control data can facilitate region-of-interest decoding and display, transcoding to limit encoded data to a selected set of tiles, loss robustness, parallelism in encoding and/or decoding, and other video processing.

Abstract: Innovations in encoding and decoding of video pictures in a high-resolution chroma sampling format (such as YUV 4:4:4) using a video encoder and decoder operating on coded pictures in a low-resolution chroma sampling format (such as YUV 4:2:0) are presented. For example, high chroma resolution details are selectively encoded on a region-by-region basis. Or, as another example, coded pictures that contain sample values for low chroma resolution versions of input pictures and coded pictures that contain sample values for high chroma resolution details of the input pictures are encoded as separate sub-sequences of a single sequence of coded pictures, which can facilitate effective motion compensation. In this way, available encoders and decoders operating on coded pictures in the low-resolution chroma sampling format can be effectively used to provide high chroma resolution details.

Abstract: In one embodiment, a video processing system 300 may filter a video data set to correct skew and wobble using a central processing unit 220 and a graphical processing unit 230. The video processing system 300 may apply a rolling shutter effect correction filter to an initial version of a video data set. The video processing system 300 may simultaneously apply a video stabilization filter to the initial version to produce a final version video data set.

Abstract: Techniques are described for optimizing memory used by a video decoder. A residual coefficient matrix including non-zero value residual coefficients of a larger parent matrix with both non-zero and zero value residual coefficients can be provided to the decoder. Residual coefficient matrix metadata can also be provided so that a modified and reduced inverse transform matrix can be selected and applied to the residual coefficient matrix.

Abstract: An encoded bitstream is processed without re-encoding so as to recombine multiple packets of each image into contiguous data of one packet for the image. Each packet is assigned a presentation time stamp, corresponding to the display order of its image in the sequence of images. In one embodiment, each intra-frame compressed image also is marked as a recovery point indicating that a decompression processor empties its buffers of data for prior groups of pictures before processing the image. A video editing or other playback application uses the converted bitstream for scrubbing and similar playback operations.

Abstract: Techniques and systems for reducing memory usage by a decoder during a format change are disclosed. In a first example technique, discretized memory allocations for new output buffers are sequenced with discretized release operations of previously-allocated memory for previous output buffers in a manner that reduces the amount of in-use memory of a computing device during a format change. In a second example technique, the allocation of new memory for new decoder buffers associated with a new format is conditioned upon the release of previously-allocated memory for decoder buffers associated with a previous format to reduce memory usage during a format change. The first and second techniques, when combined, result in optimized reduction in memory usage by a decoder during a format change.

Abstract: Efficient operations in image or video decoding. For example, a tool such as an image or video decoder receives and decodes encoded data for a picture in a bitstream. As part of the decoding, the tool adapts a multi-symbol lookup table to use in decoding of symbols then decodes the symbols using the multi-symbol lookup table, producing exactly correct results. The tool can also perform selectively truncated inverse frequency transforms. For a given block, the tool identifies upper horizontal and vertical frequencies among non-zero coefficients for the block and, based on the upper frequency values, selectively applies a simplified inverse frequency transform to transform coefficients for the block without hurting decoding quality. Using restart markers in the bitstream, the tool can organize multiple blocks of the picture as partitions. The tool decodes at least some of the partitions in parallel on a partition-by-partition basis using multiple processing cores.

Abstract: To decode encoded video using a computer with a central processing unit and a graphics processing unit as a coprocessor, parameters applied to blocks of intermediate image data are transferred from the central processing unit to the graphics processing unit. When the operation being performed applies to a small portion of the blocks of intermediate image data, then the central processing unit can transfer to the graphics processing unit the parameters for only those blocks to which the operation applies. In particular, the central processing unit can transfer a set of parameters for a limited number of blocks of intermediate image data, with an indication of the block to which each set of parameters applies, which both can improve speed of operation and can reduce power consumption.

Abstract: Video decoding innovations for multithreading implementations and graphics processor unit (“GPU”) implementations are described. For example, for multithreaded decoding, a decoder uses innovations in the areas of layered data structures, picture extent discovery, a picture command queue, and/or task scheduling for multithreading. Or, for a GPU implementation, a decoder uses innovations in the areas of inverse transforms, inverse quantization, fractional interpolation, intra prediction using waves, loop filtering using waves, memory usage and/or performance-adaptive loop filtering. Innovations are also described in the areas of error handling and recovery, determination of neighbor availability for operations such as context modeling and intra prediction, CABAC decoding, computation of collocated information for direct mode macroblocks in B slices, reduction of memory consumption, implementation of trick play modes, and picture dropping for quality adjustment.

Abstract: When packing a video frame of a higher-resolution chroma sampling format such as YUV 4:4:4 into frames of a lower-resolution chroma sampling format such as YUV 4:2:0, a computing device performs wavelet decomposition (or other band separation filtering) on sample values of chroma components of the higher-resolution frame, producing sample values of multiple bands. The device assigns the sample values of the bands to parts of the lower-resolution frames. During corresponding unpacking operations, a computing device assigns parts of the frames of the lower-resolution chroma sampling format to sample values of multiple bands. The device performs wavelet reconstruction (or other inverse band separation filtering) on the sample values of the bands, producing sample values of chroma components of the frame of the higher-resolution chroma sampling format.

Abstract: Video decoding innovations for multithreading implementations and graphics processor unit (“GPU”) implementations are described. For example, for multithreaded decoding, a decoder uses innovations in the areas of layered data structures, picture extent discovery, a picture command queue, and/or task scheduling for multithreading. Or, for a GPU implementation, a decoder uses innovations in the areas of inverse transforms, inverse quantization, fractional interpolation, intra prediction using waves, loop filtering using waves, memory usage and/or performance-adaptive loop filtering. Innovations are also described in the areas of error handling and recovery, determination of neighbor availability for operations such as context modeling and intra prediction, CABAC decoding, computation of collocated information for direct mode macroblocks in B slices, reduction of memory consumption, implementation of trick play modes, and picture dropping for quality adjustment.

Abstract: Aspects extend to methods, systems, and computer program products for video bit stream decoding. Aspects include flexible definition and detection of surface alignment requirements for decoding hardware. Surface alignment requirements can be handled by render cropping (e.g., cropping at a video output device), through adjustment and modification of original syntax values in a video bit stream and relaxed media type negotiation in a software (host) decoder. Resolution changes can be hidden with the aligned surface allocation when applicable. Performance can be improved and power consumption reduced by using hidden resolution changes.

Abstract: Aspects extend to methods, systems, and computer program products for transforming video bit streams for parallel decoding. Aspects of the invention can be used to break segment coding structure limitations in video bit streams. Aspects can be used to maximize parallelization of video decoding tasks, including motion compensation processing, to more efficiently utilize multi-core and multi-processor computer systems. Multiple portions of intra-segment data can be processed in parallel to speed up single frame processing. Video communication latency and memory requirements are also reduced.

Abstract: A host has a graphics pipeline that process frames by portions (e.g., pixels or rows) or slices. A remote device transmits a video stream container via a network to the host. A frame of the video stream in the container has encoded portions. The graphics pipeline includes a demultiplexer that extracts the portions of the video frame. When a portion has been extracted it is passed to a decoder, which is next in the pipeline. The decoder may begin decoding the portion before receiving a next portion of the frame, possibly while the demultiplexer is demultiplexing the next portion of the frame. A decoded portion of the frame is passed to a renderer which accumulates the portions of the frame and renders the frame. At any time portions of a frame might concurrently be being received, demultiplexed, decoded, and rendered. The decoder may be single-threaded, multi-threaded, or hardware accelerated.

Abstract: Techniques are described for communicating encoded data using start code emulation prevention. The described techniques include obtaining at least one partially encrypted packet, identifying at least one portion of the packet that is unencrypted, and determining that the identified unencrypted portion(s) emulates a start code. Start code emulation prevention data or emulation prevention bytes (EPBs) may be inserted into only the encrypted portion of the packet. The modified packet may be communicated to another device/storage, along with an indication of which portion(s) of the packet are unencrypted. Upon receiving the packet and indication, the receiving device may identify and remove the EPBs in the identified unencrypted portion(s) of the packet, and decrypt the packet to recover the data. In some aspects, upon identifying the indication, the receiving device may only search for EPBs in the unencrypted portion(s) of the packet, thus yielding a more efficient start code emulation prevention process.

Abstract: Video image stabilization provides better performance on a generic platform for computing devices by evaluating available multimedia digital signal processing components, and selecting the available components to utilize according to a hierarchy structure for video stabilization performance for processing parts of the video stabilization. The video stabilization has improved motion vector estimation that employs refinement motion vector searching according to a pyramid block structure relationship starting from a downsampled resolution version of the video frames. The video stabilization also improves global motion transform estimation by performing a random sample consensus approach for processing the local motion vectors, and selection criteria for motion vector reliability. The video stabilization achieves the removal of hand shakiness smoothly by real-time one-pass or off-line two-pass temporal smoothing with error detection and correction.

Abstract: Techniques are described for verifying long-term reference (LTR) usage by a video encoder and/or a video decoder. For example, verifying that a video encoder and/or a video decoder is applying LTR correctly can done by encoding and decoding a video sequence in two different ways and comparing the results. In some implementations, verifying LTR usage is accomplished by decoding an encoded video sequence that has been encoded according to an LTR usage pattern, decoding a modified encoded video sequence that has been encoded according to the LTR usage pattern and modified according to a lossy channel model, and comparing decoded video content from both the encoded video sequence and the modified encoded video sequence. For example, the comparison can comprise determining whether both decoded video content match bit-exactly beginning from an LTR recovery point location.

Abstract: Object detection techniques are described. In one or more implementations, a plurality of images are received by a computing device. The plurality of images are analyzed by the computing device to detect whether the images include, respectively, a depiction of an object. If an object is found in a first image, the locations, angles and scales for object detection can be further restricted in a second one. If an object is not found in a first one of the image, different portions of a second one of the images are analyzed for object detection.

Abstract: Lossy data stream decoder techniques are described herein. In response to a request for decoded content from a consuming application, a decoder may validate headers and identify portions of the data that are considered pertinent to the request. The decoder then performs lossy extraction to form incomplete data that is provided to the consuming application in response to the request. The full data for the data stream is not exposed to the consuming application or other downstream components. In this way, the consuming application is provided data sufficient to perform requested graphics processing and resource management operations, while at the same time the risk of piracy is mitigated since the consuming application is unable to get a full version of the data in the clear and the data have been validated by the decoder.

Abstract: Embodiments of the present invention provide a method and apparatus for removing cached data. The method comprises determining activeness of a plurality of divided lists; ranking the plurality of divided lists according to the determined activeness of the plurality of divided lists. The method comprises removing a predetermined amount of cached data from the plurality of divided lists according to the ranking result when the used capacity in the cache area reaches a predetermined threshold. Through embodiments of the present invention, the activeness of each divided list may be used to wholly measure the heat of access to the cached data included by each divided list, and upon removal, the cached data with lower heat of access in the whole system can be removed and the cached data with higher heat of access in the whole system can be retained so as to improve the read/write rate of the system.