Systems and methods for the reuse of encoding information in encoding alternative streams of video data

Systems and methods for reusing encoding information in the encoding of alternative streams of video data in accordance with embodiments of the invention are disclosed. In one embodiment of the invention, encoding multimedia content for use in adaptive streaming systems, includes selecting a first encoding level from a plurality of encoding levels using a media server, determining encoding information for a first stream of video data using the first encoding level and the media server, encoding the first stream of video data using the media server, where the first stream of video data includes a first resolution and a first bitrate, selecting a second encoding level from the plurality of encoding levels using the media server, and encoding a second stream of video data using the encoding information and the media server, where the second stream of video data includes a second resolution and a second bitrate.

Skip to: Description  ·  Claims  ·  References Cited  · Patent History  ·  Patent History
Description
CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation is a continuation of U.S. patent application Ser. No. 13/485,609 filed May 31, 2012, entitled Systems and Methods for the Reuse of Encoding Information in Encoding Alternative Streams of Video Data, the disclosure of which is incorporated herein by reference in its entirety.

FIELD OF THE INVENTION

The present invention is directed, in general, to systems and methods for adaptive streaming systems and more specifically to systems and methods for reusing encoding information in the encoding of alternative streams of video data.

BACKGROUND

The term streaming media describes the playback of media on a playback device, where the media is stored on a server and continuously sent to the playback device over a network during playback. Typically, the playback device stores a sufficient quantity of media in a buffer at any given time during playback to prevent disruption of playback due to the playback device completing playback of all the buffered media prior to receipt of the next portion of media. Adaptive bitrate streaming, or adaptive streaming, involves detecting the present streaming conditions (e.g. the user's network bandwidth and CPU capacity) in real time and adjusting the quality of the streamed media accordingly. Streaming video over the Internet has become a phenomenon in modern times. Many popular websites, such as YouTube, a service of Google, Inc. of Mountain View, Calif., and WatchESPN, a service of ESPN of Bristol, Conn., utilize streaming video in order to provide video and television programming to consumers via the Internet.

Scalable Video Coding (SVC) is an extension of the H.264/MPEG-4 AVC video compression standard, which is specified by the ITU-T H.264 standard by the International Telecommunication Union Telecommunication Standardization Sector of Geneva, Switzerland. SVC enables the encoding of a video bitstream that additionally contains one or more sub-bitstreams. The sub-bitstreams are derived from the video bitstream by dropping packets of data from the video bitstream, resulting in a sub-bitstream of lower quality and lower bandwidth than the original video bitstream. SVC supports three forms of scaling a video bitstream into sub-bitstreams: temporal scaling, spatial scaling, and quality scaling. Each of these scaling techniques can be used individually or combined depending on the specific video system.

In adaptive streaming systems, the source media is typically stored on a media server as a top level index file pointing to a number of alternate streams that contain the actual video and audio data. Each stream is typically stored in one or more container files. Different adaptive streaming solutions typically utilize different index and media containers. The Matroska container is a media container developed as an open standard project by the Matroska non-profit organization of Aussonne, France. The Matroska container is based upon Extensible Binary Meta Language (EBML), which is a binary derivative of the Extensible Markup Language (XML). Decoding of the Matroska container is supported by many consumer electronics (CE) devices. The DivX Plus file format developed by DivX, LLC of San Diego, Calif. utilizes an extension of the Matroska container format, including elements that are not specified within the Matroska format.

SUMMARY OF THE INVENTION

Systems and methods for reusing encoding information in the encoding of alternative streams of video data in accordance with embodiments of the invention are disclosed. In one embodiment of the invention, encoding multimedia content for use in adaptive streaming systems, includes selecting a first encoding level from a plurality of encoding levels using a media server, determining encoding information for a first stream of video data using the first encoding level and the media server, encoding the first stream of video data using the media server, where the first stream of video data includes a first resolution and a first bitrate, selecting a second encoding level from the plurality of encoding levels using the media server, and encoding a second stream of video data using the encoding information and the media server, where the second stream of video data includes a second resolution and a second bitrate.

In another embodiment of the invention, the first resolution and the second resolution are the same.

In an additional embodiment of the invention, the first bitrate is lower than the second bitrate.

In yet another additional embodiment of the invention, the first bitrate is greater than the second bitrate.

In still another additional embodiment of the invention, the first resolution is lower than the second resolution.

In yet still another additional embodiment of the invention, the first resolution is higher than the second resolution.

In yet another embodiment of the invention, determining encoding information includes determining encoding information selected from the group consisting of estimating motion estimation results, selecting weighted prediction weights, and determining scene changes.

In yet still another embodiment of the invention, encoding multimedia content further includes determining motion estimations for the first stream of video data using the first encoding level and the media server and scaling the determined motion estimation results for the second stream of video data using the second encoding level and the media server.

In yet another additional embodiment of the invention, determining motion estimation includes hierarchical motion estimations using the media server.

In still another additional embodiment of the invention, encoding multimedia content further includes storing the first stream of video data and the second stream of video data in a container file using the media server.

In yet still another additional embodiment of the invention, the container file is a Matroska container file.

In yet another embodiment of the invention, encoding multimedia content further includes storing the first stream of video data and the second stream of video data in separate container files using the media server.

In still another embodiment of the invention, encoding the first stream of video data and encoding the second stream of video data utilizes scalable video coding.

Still another embodiment of the invention includes a media server, including memory configured to store multimedia content, where the multimedia content includes a source video, and a processor, wherein the processor is configured by a media encoder application to select a first encoding level from a plurality of encoding levels, determine encoding information for a first stream of video data using the first encoding level, encode the first stream of video data, where the first stream of video data includes a first resolution and a first bitrate, select a second encoding level from the plurality of encoding levels, and encode a second stream of video data using the encoding information, where the second stream of video data includes a second resolution and a second bitrate.

In yet another additional embodiment of the invention, the first resolution and the second resolution are the same.

In still another additional embodiment of the invention, the first bitrate is lower than the second bitrate.

In yet still another additional embodiment of the invention, the first bitrate is greater than the second bitrate.

In yet another embodiment of the invention, the first resolution is lower than the second resolution.

In still another embodiment of the invention, the first resolution is higher than the second resolution.

In yet still another embodiment of the invention, the determined encoding information is selected from the group consisting of emotion estimation results, weighted prediction weights, and scene changes.

In yet another additional embodiment of the invention, the processor is further configured to determine motion estimations for the first stream of video data using the first encoding level and scale the determined motion estimation results for the second stream of video data using the second encoding level.

In still another additional embodiment of the invention, the determined motion estimation includes hierarchical motion estimations.

In yet still another additional embodiment of the invention, the processor is further configured to store the first stream of video data and the second stream of video data in a container file.

In yet another embodiment of the invention, the container file is a Matroska container file.

In still another embodiment of the invention, the processor is further configured to store the first stream of video data and the second stream of video data in separate container files.

In yet still another embodiment of the invention, the first stream of video data and the second stream of video data are encoded using scalable video coding.

Still another embodiment of the invention includes a machine readable medium containing processor instructions, where execution of the instructions by a processor causes the processor to perform a process including selecting a first encoding level from a plurality of encoding levels, determining encoding information for a first stream of video data using the first encoding level, encoding the first stream of video data, where the first stream of video data includes a first resolution and a first bitrate, selecting a second encoding level from the plurality of encoding levels, and encoding a second stream of video data using the encoding information, where second stream of video data includes a second resolution and a second bitrate.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a system diagram of an adaptive streaming system in accordance with an embodiment of the invention.

FIG. 2 is a block diagram of a media server configured to encode streams of video data for use in adaptive streaming systems in accordance with an embodiment of the invention.

FIG. 3 conceptually illustrates a media server configured to encode streams of video data for use in adaptive streaming systems in accordance with an embodiment of the invention.

FIG. 4 is a flow chart illustrating a process for reusing encoding information in the encoding of alternative streams of video data in accordance with an embodiment of the invention.

FIG. 5 is a flow chart illustrating a process for reusing motion estimations between encodings of alternative streams of video data having matching macroblocks in accordance with an embodiment of the invention.

FIG. 6 is a flow chart illustrating a process for performing a hierarchical motion estimation of macroblocks in a frame of video data in accordance with an embodiment of the invention.

 DETAILED DESCRIPTION

Turning now to the drawings, systems and methods for reusing encoding information in the encoding of alternative streams of video data in accordance with embodiments of the invention are disclosed. Although the present invention is described below with respect to adaptive streaming systems and block-based video encoding techniques, the systems and methods described are equally applicable in conventional streaming systems where different streams of video data are selected based upon a network client's connection quality and video encoding techniques that are not block-based.

In adaptive streaming systems, multimedia content is encoded as a set of alternative streams of video data. Because each alternative stream of video data is encoded using the same source multimedia content, similar encoding information is determined in the encoding of each alternative stream of video data. Encoding information includes, but is not limited to, motion estimation results, the selection of weighted prediction weight(s), and determining scene changes. Systems and methods in accordance with many embodiments of the invention reuse encoding information determined in the encoding of one alternative stream of video data in the encoding of at least one other alternative stream of video data. By reusing encoding information in the encoding of several alternative streams of video data, significant improvements in the encoding of the alternative streams of video data can be achieved, particularly significant time savings may be realized in accordance with embodiments of the invention.

Adaptive streaming systems are configured to stream multimedia content encoded at different maximum bitrates and resolutions over a network, such as the Internet. Adaptive streaming systems stream the highest quality multimedia content, which can be supported based upon current streaming conditions. Multimedia content typically includes video and audio data, subtitles, and other related metadata. In order to provide the highest quality video experience independent of the network data rate, adaptive streaming systems are configured to switch between the available sources of video data throughout the delivery of the video data according to a variety of factors, including, but not limited to, the available network data rate and video decoder performance. When streaming conditions deteriorate, an adaptive streaming system typically attempts to switch to multimedia streams encoded at lower maximum bitrates. In the event that the available network data rate cannot support streaming of the stream encoded at the lowest maximum bitrate, then playback is often disrupted until a sufficient amount of content can be buffered to restart playback. Systems and methods for switching between video streams during playback that may be utilized in adaptive streaming system in accordance with embodiments of the invention are described in U.S. patent application Ser. No. 13/221,682 entitled “Systems and Methods for Adaptive Bitrate Streaming of Media Stored in Matroska Container Files Using Hypertext Transfer Protocol” to Braness et al., filed Aug. 30, 2011, the entirety of which is incorporated by reference.

To create the multiple sources of video data utilized in adaptive streaming systems, a source encoder can be configured to encode a plurality of alternative streams of video data from a source video contained in a piece of multimedia content. Systems and method for encoding a source video for use in adaptive streaming systems are disclosed in U.S. patent application Ser. No. 13/221,794, entitled “Systems and Methods for Encoding Source Media in Matroska Container Files for Adaptive Bitrate Streaming Using Hypertext Transfer Protocol” to Braness et al., filed Aug. 30, 2011, the entirety of which is incorporated by reference. In accordance with embodiments of the invention, a source encoder may be implemented using a media source and/or a media server.

As stated above, alternative streams of video data based on the same source video contain similar content; therefore, encoding information determined in the encoding of one alternative stream of video data for the source video may be reused in the encoding of the one or more of the other alternative streams of video data. In accordance with embodiments of the invention, a set of alternative streams of video data based on the same source video may contain video data at the same resolution but differing bitrates. In many embodiments of the invention, the motion estimation results calculated for the encoding of a particular alternative stream of video data can be reused amongst all the alternative streams of video data. As is discussed below, a variety of encoding information determined in the encoding of alternative streams of video data may be reused amongst the alternative streams of video data. Systems and methods for reusing encoding information in the encoding of alternative streams of video data in accordance with embodiments of the invention are discussed further below.

System Overview

Adaptive streaming systems in accordance with embodiments of the invention are configured to adapt the video transmitted to network clients based on streaming conditions. An adaptive streaming system in accordance with an embodiment of the invention is illustrated in FIG. 1. The illustrated adaptive streaming system 10 includes a media source 100 containing multimedia content. In a number of embodiments of the invention, the multimedia content stored in media source 100 contains a plurality of pre-encoded streams of video data encoded at multiple bitrates and resolutions for adaptive streaming, where the encoding of at least one of the streams of video data in the plurality of streams of video data reuses encoding information determined in the encoding of a second stream of video data. In several embodiments of the invention, the media source 100 encodes a plurality of streams of video data in real time, where the encoding of at least one of the streams of video data in the plurality of streams of video data reuses encoding information determined in the encoding of a second stream of video data. In many embodiments of the invention, the multimedia content stored in media source 100 contains video data encoded utilizing Scalable Video Coding (SVC). In a number of embodiments, the media source 100 contains multiple streams of video data with equal timelines as alternative streams of video data. The media source 100 is connected to a network renderer 102. In many embodiments, the media source 100 and the network renderer 102 are implemented using a media server. The network renderer 102 is connected to a plurality of network clients 104 via a network 110. The network renderer 102 is configured to stream video data to one or more network clients 104. The network clients 104 each contain a media decoder 106 configured to decode media streamed from the media source 100 using the network renderer 102.

In many embodiments of the invention, the media source 100 and/or the network renderer 102 are implemented using a single machine. In several embodiments of the invention, the media source 100 and/or the network renderer 102 are implemented using a plurality of machines. In many embodiments of the invention, the media source 100 and the network renderer 102 are implemented using a media server. In many embodiments, the network 110 is the Internet. In several embodiments, the network 110 is any IP network. In a number of embodiments, the network 110 is a cellular data network.

Although a specific architecture of an adaptive streaming system is shown in FIG. 1, other implementations appropriate to a specific application can be utilized in accordance with embodiments of the invention. Systems and methods for the reuse of encoding information in the encoding of alternative streams of video data in accordance with embodiments of the invention are discussed further below.

Media Servers Capable of Encoding Media for Adaptive Streaming

In accordance with embodiments of the invention, media servers are configured to encode a plurality of alternative streams of video data for a source video contained in multimedia content. FIG. 2 is a block diagram of a media server in accordance with an embodiment of the invention. The media server 200 includes a processor 210 and a video encoder 212. The processor is configured to encode a plurality of streams of video data given a source video. In a number of embodiments, the video encoder 212 is implemented in hardware. In several embodiments, the video encoder 212 is implemented utilizing a graphics processing unit. In many embodiments, the video encoder 212 is a software application, which configures the processor 210 to encode alternative streams of video data.

In a number of embodiments, a media server 200 includes a storage device 220 connected to the processor 210 configured to store multimedia content and/or alternative streams of video data. In many embodiments of the invention, the process 400 encodes the multimedia content using scalable video coding. In several embodiments, the alternative streams of video data are stored in a container utilizing the Matroska container format. In accordance with embodiments of the invention, the alternative streams of video data have an adaptive group of pictures structure.

In a number of embodiments, the media server 200 includes a display device 224 connected to the processor 210, where the display device may be used to display alternative streams of video data and/or information related to the encoding of alternative streams of video data. In accordance with embodiments of the invention, the media server 200 is implemented using one or more servers configured with software applications appropriate to encode and/or stream multimedia content. A media server 200 optionally includes a network device 222 connected to the processor 210. In many embodiments, the network device 222 is configured to establish a network connection and the processor 210 is configured to stream multimedia content using the network connection for use in adaptive streaming systems.

The basic architecture of a media server in accordance with an embodiment of the invention is illustrated in FIG. 3. The media server 300 includes a processor 310 in communication with non-volatile memory 330 and volatile memory 320. In the illustrated embodiment, the non-volatile memory includes a video encoder 332 that configures the processor to encode a plurality of alternative streams of video data for a source video, where the encoding of at least one of the alternative streams of video data reuses encoding information determined in the encoding of a second alternative stream of video data. In several embodiments, the video encoder 332 is implemented using the processor 310. The network client 300 also includes a network interface 340 configured to send and receive data over a network connection. In a number of embodiments, the network interface 340 is in communication with the processor 310, the non-volatile memory 330, and the volatile memory 320.

In many embodiments of the invention, the media server 300 includes multiple processors. In a number of embodiments, the video encoder 332 is implemented using dedicated hardware. Although a specific architecture for a media server is illustrated in FIG. 3, any of a variety of architectures including architectures where the video encoder 332 is located on disk or some other form of storage and is loaded into volatile memory 320 at runtime may be utilized to implement media servers for use in adaptive streaming systems in accordance with embodiments of the invention.

Although a specific implementation of a media server is shown in FIG. 2, other implementations appropriate to a specific application can be utilized in accordance with embodiments of the invention. Systems and methods for the reuse of encoding information in the encoding of alternative streams of video data in accordance with embodiments of the invention are discussed further below.

Reusing Encoding Information in the Encoding of Alternative Streams of Video Data

In order to provide streams of video data utilized in the adaptive streaming of multimedia content, a media server can encode the multimedia content as alternative streams of video data. The media server can improve the speed and performance of encoding the alternative streams of video data by reusing encoding information determined in the encoding of one stream of video data in the encoding of one or more of the remaining streams of video data. A process for reusing encoding information in the encoding of alternative streams of video data for a source video is illustrated in FIG. 4. The process 400 includes receiving (410) multimedia content containing a source video. In several embodiments, encoding levels are determined (412) for the source video. An encoding level is selected (414). Encoding information is determined (416) for the selected encoding level. The encoding information is reused (418) across one or more of the encoding levels not selected.

In several embodiments, encoding levels are determined (412) based upon factors including, but not limited to, anticipated network data rates, anticipated screen resolutions of network clients, and/or the resolution and/or bitrate of the source video. In a number of embodiments, encoding levels are determined (412) by retrieving stored values. In accordance with embodiments of the invention, an alternative stream of video data having a resolution and bitrate is encoded for each encoding level. In many embodiments, the alternative streams of video data have the same resolution and varying bitrates. In a number of embodiments, the selected (414) encoding level corresponds to the alternative stream of video data with the lowest bitrate. In several embodiments, the selected (414) encoding level corresponds to the alternative stream of video data with the highest bitrate. In a number of embodiments, a variety of determined (416) encoding information may be reused (418) across encoding levels having video data at the same resolution and varying bitrates, including, but not limited to, motion estimation results, the selection of weighted prediction weight(s), and determining scene changes. In accordance with embodiments of the invention, the encoding information can be reused (418) across encoding levels, resulting in a significant savings in computational complexity.

In several embodiments, the alternative streams of video data encoded based on the determined (412) encoding levels have varying resolutions and bitrates. In many embodiments, the selected (414) encoding level corresponds to an alternative stream of video data having the lowest resolution. In a number of embodiments, the selected (414) encoding level corresponds to an alternative stream of video data having the highest resolution. In a number of embodiments, the determined (416) encoding information is reused (418) across streams of video data having different resolutions. In accordance with many embodiments of the invention, the encoding information reused (418) may vary across encoding levels. For example, if the reused (418) encoding information is a motion vector, the motion vector can be scaled relative to the different resolutions of the alternative streams of video data. Other encoding information, such as the information listed above, may likewise be scaled between encoding levels corresponding to alternative streams of video data having different resolutions.

Although a specific process for the reuse of encoding information in the encoding of alternative streams of video data in accordance with an embodiment of the invention is discussed above, a variety of processes, including reusing encoding information not specifically listed and reusing encoding information in alternative streams of video data created at a later time following the initial determination of the encoding information, may be utilized in accordance with embodiments of the invention. A process for reusing motion estimations used in the encoding of alternative streams of video data in accordance with an embodiment of the invention is discussed below.

Reusing Motion Estimations in the Encoding of Alternative Streams of Video Data

A variety of video compression standards, such as H.264, may be utilized to encode alternative streams of video data for use in adaptive streaming systems. Many of the video compression standards utilize frames of video containing macroblocks, and the encoding of the stream of video data involves calculating motion estimations of the macroblocks between frames of video. In accordance with embodiments of the invention, the bitrate of a stream of video data can be varied by allocating the same number of bits to the motion estimations and increasing or decreasing the number of bits assigned to the description of the macroblocks.

A process for reusing motion estimations of the motion of macroblocks between alternative streams of video data derived from the same source video contained in a piece of multimedia content in accordance with an embodiment of the invention is illustrated in FIG. 5. The process 500 includes performing (510) downscaling on the source video to determine the alternative streams of video data. An alternative stream of video data is selected (512). Macroblocks are determined (514) for the selected stream of video data. A search (516) for matching macroblocks in a second alternative stream of video data is performed. If the second alternative stream of video data contains one or more matching macroblocks, the motion estimation results related to the matching macroblocks are reused (518) in the encoding of the second alternative stream of video data.

Several methods exist for computing estimations of the motion of macroblocks in frames of video data, including, but not limited to, full searches, heuristic searches, and hierarchical estimations. Hierarchical estimations are commonly used because they provide an accurate result in a fast and efficient manner. A process for performing a hierarchical motion estimation of macroblocks in a frame of video data in accordance with an embodiment of the invention is illustrated in FIG. 6. The process 600 includes selecting (610) a frame of video data, which is known as the zero resolution used in the search. Resolutions are created (612) for the selected frame. In accordance with embodiments of the invention, two or more resolutions are created (612), where the created resolutions are smaller than the previous resolution. Resolutions may be created (612) using a variety of image sampling techniques, including, but not limited to, mean intensity and subsampling. The highest resolution is searched (614) for macroblocks corresponding to macroblocks in the zero resolution. The motion estimation is computed (616) for the corresponding macroblock in the highest resolution. If the zero resolution frame has been reached (618), the computed (616) motion estimation is used as the motion estimation for the frame of video data and the process completes. If the zero resolution frame has not been reached (618), the next resolution is searched (614) and the motion estimation is computed (616) utilizing the previously computed (616) motion estimation and the distance the corresponding macroblock has moved between the previous resolution and the current resolution. In a number of embodiments, macroblock motion is computed by a variety of methods, including the mean absolute difference, mean square difference, and pixel difference classification.

Returning to FIG. 5, in several embodiments, searching (516) for matching macroblocks utilizes hierarchical motion estimation. In a number of embodiments, hierarchical motion estimation results, including intermediate results, may be reused (518) between alternative streams of video data having different resolutions. In many embodiments, the selected (512) stream of video data is the alternative stream of video data with the lowest resolution. In a number of embodiments, the selected (512) stream of video data has the highest resolution. In several embodiments, the reused (518) motion estimation results are scaled between alternative streams of video data. A specific process for reusing motion estimations in the encoding of alternative streams of video data is described above; however, a variety of processes for reusing motion estimations may be utilized in accordance with embodiments of the invention.

Although the present invention has been described in certain specific aspects, many additional modifications and variations would be apparent to those skilled in the art. It is therefore to be understood that the present invention may be practiced otherwise than specifically described without departing from the scope and spirit of the present invention. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.

Claims

1. A method for encoding multimedia content for use in adaptive streaming systems, comprising:

selecting a first encoding level from a plurality of encoding levels using a media server, where the media server comprises a processor and a memory;
determining encoding information at a video encoder for a first stream of video data using the first encoding level and the media server, where determining encoding information comprises determining motion estimations for the first stream of video data;
encoding the first stream of video data using the determined motion estimation results and the media server, where the first stream of video data includes a first resolution and a first bitrate;
selecting a second encoding level from the plurality of encoding levels using the media server; and encoding a second stream of video data reusing the encoding information determined for the first stream of video data and using the media server, the second stream of video data being independent from the first stream of video data, wherein: reusing the encoding information comprises scaling the determined motion estimation results for the second stream of video data using the second encoding level; and the second stream of video data comprises a second resolution and a second bitrate.

2. The method of claim 1, wherein the first resolution and the second resolution are the same.

3. The method of claim 2, wherein the first bitrate is lower than the second bitrate.

4. The method of claim 2, wherein the first bitrate is greater than the second bitrate.

5. The method of claim 1, wherein the first resolution is lower than the second resolution.

6. The method of claim 1, wherein the first resolution is higher than the second resolution.

7. The method of claim 1, wherein determining motion estimation includes hierarchical motion estimations using the media server.

8. The method of claim 1, further comprising storing the first stream of video data and the second stream of video data in a container file using the media server.

9. The method of claim 8, wherein the container file is a Matroska container file.

10. The method of claim 1, further comprising storing the first stream of video data and the second stream of video data in separate container files using the media server.

11. The method of claim 1, wherein encoding the first stream of video data and encoding the second stream of video data utilizes scalable video coding.

12. A media server, comprising:

memory storing multimedia content comprising a source video; and
a processor;
wherein the processor is directed by a media encoder application to: select a first encoding level from a plurality of encoding levels; determine encoding information at a video encoder for a first stream of video data using the first encoding level, where determining encoding information comprises determining motion estimations for the first stream of video data; encode the first stream of video data using the determined motion estimation results, where the first stream of video data includes a first resolution and a first bitrate; select a second encoding level from the plurality of encoding levels; and encode a second stream of video data reusing the encoding information determined for the first stream of video data, the second stream of video data being independent from the first stream of video data;
wherein reusing the encoding information comprises scaling the determined motion estimation results for the second stream of video data using the second encoding level; and
wherein the second stream of video data includes a second resolution and a second bitrate.

13. The media server of claim 12, wherein the first resolution and the second resolution are the same.

14. The media server of claim 13, wherein the first bitrate is lower than the second bitrate.

15. The media server of claim 13, wherein the first bitrate is greater than the second bitrate.

16. The media server of claim 12, wherein the first resolution is lower than the second resolution.

17. The media server of claim 12, wherein the first resolution is higher than the second resolution.

Referenced Cited
U.S. Patent Documents
5477272 December 19, 1995 Zhang et al.
5592226 January 7, 1997 Lee et al.
5734822 March 31, 1998 Houha et al.
5794061 August 11, 1998 Moussouris
5822524 October 13, 1998 Chen et al.
5930473 July 27, 1999 Teng et al.
6075884 June 13, 2000 Lubin et al.
6118817 September 12, 2000 Wang
6192154 February 20, 2001 Rajagopalan et al.
6195388 February 27, 2001 Choi et al.
6229850 May 8, 2001 Linzer et al.
6233359 May 15, 2001 Ratnakar et al.
6330369 December 11, 2001 Hoag et al.
6343098 January 29, 2002 Boyce
6356668 March 12, 2002 Honsinger et al.
6563517 May 13, 2003 Rubas
6785704 August 31, 2004 McCanne
6859496 February 22, 2005 Boroczky et al.
6956901 October 18, 2005 Boroczky et al.
6970602 November 29, 2005 Li et al.
6976151 December 13, 2005 Overkamp et al.
6995769 February 7, 2006 Ordentlich
7224890 May 29, 2007 Kato
7330509 February 12, 2008 Liu et al.
7424160 September 9, 2008 Basse
7496283 February 24, 2009 Evans et al.
8023562 September 20, 2011 Zheludkov et al.
8054880 November 8, 2011 Yu et al.
8073275 December 6, 2011 Raudnitz
8180920 May 15, 2012 Mills et al.
8249168 August 21, 2012 Graves
8270473 September 18, 2012 Chen et al.
8311115 November 13, 2012 Gu et al.
8331439 December 11, 2012 Sedeffow
8351498 January 8, 2013 Sedeffow
8363715 January 29, 2013 Sedeffow
8588296 November 19, 2013 Ding et al.
8724691 May 13, 2014 Sedeffow
9350990 May 24, 2016 Orton-Jay et al.
9357210 May 31, 2016 Orton-Jay et al.
9532080 December 27, 2016 Zurpal et al.
10178399 January 8, 2019 Orton-Jay et al.
20020054636 May 9, 2002 Yoneyama
20020085638 July 4, 2002 Morad et al.
20020097750 July 25, 2002 Gunaseelan et al.
20020136296 September 26, 2002 Stone et al.
20020169823 November 14, 2002 Coulombe
20030002743 January 2, 2003 Ohta
20030007558 January 9, 2003 Vetro et al.
20030012275 January 16, 2003 Boice et al.
20030031371 February 13, 2003 Kato et al.
20030043905 March 6, 2003 Nakayama et al.
20030067981 April 10, 2003 Zhao
20030086597 May 8, 2003 Ohta et al.
20030228063 December 11, 2003 Nakayama et al.
20040165783 August 26, 2004 Reynolds et al.
20040255139 December 16, 2004 Giobbi
20050009519 January 13, 2005 Murai et al.
20050166245 July 28, 2005 Shin
20050251835 November 10, 2005 Scott et al.
20050278794 December 15, 2005 Leinonen
20060045020 March 2, 2006 Picco et al.
20060056508 March 16, 2006 Lafon et al.
20060078050 April 13, 2006 Chen
20060095942 May 4, 2006 van
20060095944 May 4, 2006 Demircin et al.
20060126728 June 15, 2006 Yu et al.
20060126948 June 15, 2006 Fukuhara et al.
20060154648 July 13, 2006 Oh et al.
20060165381 July 27, 2006 Eckleder
20060190410 August 24, 2006 Harper
20060198392 September 7, 2006 Park
20060268933 November 30, 2006 Kellerer et al.
20070002946 January 4, 2007 Bouton et al.
20070053444 March 8, 2007 Shibata et al.
20070074266 March 29, 2007 Raveendran et al.
20070086528 April 19, 2007 Mauchly et al.
20070086680 April 19, 2007 Lim
20070094691 April 26, 2007 Gazdzinski
20070098162 May 3, 2007 Shin
20070103484 May 10, 2007 Hougui et al.
20070153914 July 5, 2007 Hannuksela et al.
20070230586 October 4, 2007 Shen et al.
20070271106 November 22, 2007 Lee et al.
20070283269 December 6, 2007 Obrador et al.
20080037880 February 14, 2008 Lai
20080043831 February 21, 2008 Sethuraman
20080063051 March 13, 2008 Kwon et al.
20080084925 April 10, 2008 Rozen et al.
20080112483 May 15, 2008 Lu et al.
20080120676 May 22, 2008 Morad et al.
20080137736 June 12, 2008 Richardson et al.
20080192820 August 14, 2008 Brooks et al.
20080196076 August 14, 2008 Shatz et al.
20080273591 November 6, 2008 Brooks et al.
20080273593 November 6, 2008 Sedeffow
20080273594 November 6, 2008 Sedeffow
20080313680 December 18, 2008 Suzanne
20090003432 January 1, 2009 Liu et al.
20090016433 January 15, 2009 Henocq et al.
20090028428 January 29, 2009 Dovstam et al.
20090060033 March 5, 2009 Kimmich et al.
20090067489 March 12, 2009 Jacobs
20090067491 March 12, 2009 Sun et al.
20090086814 April 2, 2009 Leontaris et al.
20090086816 April 2, 2009 Leontaris
20090110070 April 30, 2009 Takahashi et al.
20090132721 May 21, 2009 Soroushian et al.
20090168880 July 2, 2009 Jeon et al.
20100020878 January 28, 2010 Liang et al.
20100034257 February 11, 2010 Sedeffow
20100061448 March 11, 2010 Zhou
20100070773 March 18, 2010 Tanaka et al.
20100111192 May 6, 2010 Graves
20100124274 May 20, 2010 Cheok et al.
20100189179 July 29, 2010 Gu et al.
20100189183 July 29, 2010 Gu et al.
20100195713 August 5, 2010 Coulombe et al.
20100266009 October 21, 2010 Sedeffow
20100316126 December 16, 2010 Chen et al.
20100322302 December 23, 2010 Rodriguez
20110090950 April 21, 2011 Bush et al.
20110090960 April 21, 2011 Leontaris et al.
20110129202 June 2, 2011 Soroushian
20110170790 July 14, 2011 Cheon et al.
20110255555 October 20, 2011 Alexander
20110268178 November 3, 2011 Park et al.
20110305273 December 15, 2011 He et al.
20110310216 December 22, 2011 Lee et al.
20120017249 January 19, 2012 Ozawa
20120173751 July 5, 2012 Braness et al.
20120177101 July 12, 2012 van der Schaar
20120307883 December 6, 2012 Graves
20130031206 January 31, 2013 Miyazaki et al.
20130034166 February 7, 2013 Shiodera et al.
20130044821 February 21, 2013 Braness et al.
20130128970 May 23, 2013 Yu et al.
20130322517 December 5, 2013 Zurpal et al.
20140015922 January 16, 2014 Zhang
20140241420 August 28, 2014 Orton-Jay et al.
20140241421 August 28, 2014 Orton-Jay et al.
20150036758 February 5, 2015 Sato
20160065959 March 3, 2016 Stobaugh et al.
20160249109 August 25, 2016 Nelson
20160261879 September 8, 2016 Orton-Jay et al.
20160335754 November 17, 2016 Aaron et al.
20160360202 December 8, 2016 Xu et al.
20170094279 March 30, 2017 Piramanayagam et al.
20170359580 December 14, 2017 Su et al.
20180276538 September 27, 2018 Gopalan
20190075301 March 7, 2019 Chou et al.
20190132591 May 2, 2019 Zhang et al.
Foreign Patent Documents
1120140 29794 June 2017 BR
1684518 October 2005 CN
105359511 February 2016 CN
1585336 October 2005 EP
1596594 November 2005 EP
1626557 February 2006 EP
1659736 May 2006 EP
1732333 December 2006 EP
1770957 April 2007 EP
1786205 May 2007 EP
2856751 April 2015 EP
2962461 January 2016 EP
3038370 June 2016 EP
08195956 July 1996 JP
2001503225 March 2001 JP
2002142218 May 2002 JP
2003299064 October 2003 JP
2004320707 November 2004 JP
2005229593 August 2005 JP
2005260541 September 2005 JP
2005311459 November 2005 JP
2006033393 February 2006 JP
2009502099 January 2009 JP
2010258894 November 2010 JP
2010262255 November 2010 JP
2011029962 February 2011 JP
2013026724 February 2013 JP
2015523789 August 2015 JP
2016526336 September 2016 JP
201763495 March 2017 JP
20160021141 February 2016 KR
101823321 January 2018 KR
1997004585 February 1997 WO
9819460 May 1998 WO
2000018137 March 2000 WO
2000079801 December 2000 WO
2005083917 September 2005 WO
2006077526 July 2006 WO
2007018669 February 2007 WO
2007053957 May 2007 WO
2008108966 September 2008 WO
2010114092 October 2010 WO
2012009485 January 2012 WO
2013181013 December 2013 WO
2013181013 December 2013 WO
2014132098 September 2014 WO
2014190308 November 2014 WO
2016132148 August 2016 WO
2017036370 March 2017 WO
Other references
  • “Advanced video coding for generic audiovisual services”, International Telecommunication Union ITU-T Recommendation H.264, Telecommunication Standardization Sector of ITU, Series H: Audiovisual and Multimedia Systems, May 7, 2004, pp. 13-26.
  • Extended European Search Report for European Application No. 13798009.0, Search completed Jan. 18, 2017, dated Jan. 26, 2017, 16 Pgs.
  • Extended European Search Report for European Application No. 13876558.1, Search completed Jun. 13, 2017, dated Jun. 20, 2017, 12 Pgs.
  • International Preliminary Report on Patentability for International Application PCT/IB2013/001122, Report Issued Sep. 1, 2015, dated Sep. 11, 2015, 11 Pgs.
  • International Preliminary Report on Patentability for International Application PCT/US2013/041874, report completed Dec. 2, 2014, 6 pages.
  • International Search Report and Written Opinion for International Application PCT/IB2013/001122, Search Completed Oct. 21, 2013, dated Nov. 5, 2013, 12 pages.
  • International Search Report and Written Opinion for International Application PCT/US2013/041874, Search completed Jul. 16, 2013, dated Jul. 24, 2013, 7 pages.
  • Partial Supplemental European Search Report for European Application No. 13798009.0, Search completed Aug. 31, 2016, dated Sep. 7, 2016, 6 pages.
  • International Search Report and Written Opinion for International Application PCT/US14/39425, Report Completed Sep. 15, 2014, dated Oct. 17, 2014, 9 pages.
  • “H.265—HEVC, High Efficiency Video Coding, A Complete Guide to the Latest Standard in Video Compression”, Apr. 13, 2013, Retrieved from https://www.encoding.com/h-265/ on Jun. 6, 2017, 3 pgs.
  • “MPEG-2 Part 1, ISO/IEC Standard 13818-1”, Generic Coding of Moving Pictures and Associated Audio: Systems, Nov. 13, 1994, 161 pgs.
  • “MPEG-4, Part 14, ISO/IEC 14496-14”, Information technology—Coding of audio-visual objects, Nov. 15, 2003, 18 pgs.
  • Anonymous, “Method for the encoding of a compressed video sequence derived from the same video sequence compressed at a different bit rate without loss of data”, ip.com, ip.com No. IPCOM000008165D, May 22, 2002, 9 pages.
  • Crusty, “The Unofficial XviD Faq”, XP002338686, Mar. 4, 2004, Retrieved from http://ronald.vslcatena.nl/docs/Xvid-F AQ-Crusty-04-03-2004.pdf on Jul. 28, 2005, 50 pages.
  • Jang et al., “An adaptive non-linear motion vector resampling algorithm for down-scaling video transcoding”, Proceedings of The 2003 International Conference on Multimedia and Expo, vol. 2, Jul. 6-9, 2003, Baltimore, Maryland, USA, pp. 229-232.
  • Mitsumura et al., “Analytical study of feedback-based rate control at application layer for video streaming services”, Technical study report of the institute of electronics, information and Communication Engineers, vol. 111, No. 468, ISSN: 0913-5685, Mar. 8-9, 2012, pp. 203 to 208.
  • Senda, Yuzo, “Seamless video service and video transcoder”, NEC technical journal, NEC Creative, vol. 51, No. 8, ISSN: 0285-4139, Aug. 25, 1998, pp. 46 to 49.
  • Westerink et al., “Two-pass MPEG⋅2 variable-bit-rate encoding”, IBM Journal of Research and Development, International Business Machines Corporation, XP002395114, ISSN: 0018-8646, vol. 43, No. 4, Jul. 4, 1999, New York, NY, USA, pp. 471-488.
  • Extended European Search Report for European Application No. 14800901.2, Search completed Dec. 2, 2016, dated Dec. 13, 2016, 13 Pgs.
  • International Search Report for International Application PCT/JP2010/056013, Search completed Apr. 20, 2010, dated May 11, 2010, 2 pgs.
  • Search report for Application GB0709009.5, dated Aug. 31, 2007, 1 pg.
  • Search report for Application GB0709030.1 , dated Aug. 31, 2007, 2 pg.
  • Search report for Application GB0709030.1 , dated Jan. 31, 2008, 2 pg.
  • Search Report for Application GB0906721.6, dated Jul. 24, 2009, 1 pg.
  • Anonymous, “Any DVD Converter”, Dec. 30, 2009, XP55005595, Retrieved from the Internet: http://web.archive.org/web/20091230041557/http://any-video-converter.com/products/for_dvd/ retrieved on Aug. 25, 2011.
  • Anonymous, “How to rip DVDs in Windows/Mac using Handbrake”, Oct. 3, 2008, Retrieved from the Internet: http://www.my-guides.net/en/content/view/111/26/ , retrieved on Aug. 30, 2011.
  • Anonymous, “Screenshot—Any DVD Converter”, Dec. 30, 2009, XP55005596, Retrieved from the Internet: url:http://web.archive.org/web/20091230070300/http://any-video-converter.com/products/for_dvd/screenshot.php retrieved on Aug. 25, 2011.
  • Bhat et al., “An architectural approach towards achieving bandwidth efficient content delivery”, 2016 Spring Technical Forum Proceedings, 8 pgs.
  • Chiariotti, “Reinforcement learning algorithms for DASH video streaming”, Thesis, University of Padova, Academic year 2014-2015, 91 pgs.
  • Fernandez Escribano et al., “Video Encoding and Transcoding Using Machine Learning”, MDM/KDD '08, Aug. 24, 2008, pp. 53-62.
  • Haskell et al, “Digital Video: An Introduction to MPEG-2”, Chapman and Hall, 1997, pp. 33, 35, 230, 232-233.
  • Hickson, Ian et al., “HTML5—A vocabulary and associated APIs for HTML and XHTML”, Mar. 4, 2010, 1-569.
  • Hunter, “Machine Learning and AI dominate SMPTE 2017 conference”, https://www.thebroadcastbridge.com/content/entry/9902/machine-learning-and-ai-dominate-smpte-2017-conference, 4 pgs.
  • Knight, Tony, “Metadata & the Supply Chain”, Mar. 24, 2011.
  • Lawson, Bruce, “Accessible HTML5 Video with JavaScripted captions”, Jan. 13, 2010.
  • Oh et al., “Action-Conditional Video Prediction Using Deep Networks in Atari Games”, Neural Information Processing Systems, 2015, 9 pgs.
  • Pieters, Simon, “Everything you need to know about Htmls video and audio”, Mar. 3, 2010.
  • Pilgrim, Mark, “Video on the Web”, Nov. 16, 2009.
  • Sambe et al., Yasuo, “High-speed Distributed Video Transcoding for Multiple Rates and Formats”, IEICE Transactions on Information and Systems, vol. E88-D Issue 8, Aug. 2005, pp. 1923-1931, XP-002516536.
  • Shen et al., Bo, “Caching Strategies in Transcoding-enabled Proxy Systems for Streaming Media Distribution Networks”, HP Laboratories Technical Report, HPL-2003-261, Dec. 10, 2003—, pp. 1-11,XP-002293154.
  • Srivastava et al., “Unsupervised Learning of Video Representations using LSTMs”, arXiv:1502.04681, 2015, 12 pgs.
  • Steck, Chris, “Delivering Total Mobile TV: Combining Streaming and Broadcast for a Complete Mobile Television Solution”, Dated Apr. 2006. Rev. 1.0.2,12 pages (emphasizing p. 9).
  • International Preliminary Report on Patentability for International Application PCT/US2014/039425, Report Issued Nov. 24, 2015, dated Dec. 3, 2015, 6 pgs.
  • Colah et al., “Understanding LSTM Networks”, colah.github.io/posts/2015-08-Understanding-LSTMs/, Aug. 27, 2015, 8 pgs.
Patent History
Patent number: 11025902
Type: Grant
Filed: Dec 22, 2016
Date of Patent: Jun 1, 2021
Patent Publication Number: 20170104989
Assignee: NLD Holdings I, LLC (San Deigo, CA)
Inventors: Sergey Zurpal (Seversk), Leonid Volnin (Saint-Petersberg)
Primary Examiner: Dave Czekaj
Assistant Examiner: Berteau Joisil
Application Number: 15/388,466
Classifications
Current U.S. Class: Quantization (375/240.03)
International Classification: H04N 19/30 (20140101); H04N 19/103 (20140101); H04N 19/56 (20140101); H04N 19/61 (20140101); H04N 19/33 (20140101); H04N 19/53 (20140101); H04N 19/39 (20140101); H04N 19/87 (20140101); H04N 19/184 (20140101);