ADAPTIVE STREAMING TO MULTICAST AND CONSTRAINED-FIDELITY CONSTANT BIT RATE ENCODING
This disclosure describes and adaptive bit rate encoding and distribution techniques for conserving bandwidth usage in a channel. The invention comprises, an encoder or transcoder, a video fragmenter, a video-quality analyzer that output complexity values, a streaming server, a process by which individual fragments are selected for distribution, a video-quality threshold, and, optionally a bandwidth reclamation factor. A video-quality analyzer inspects any combination of the input and output of the encoder, transcoder, or fragmenter, and produces a video-quality metric for each fragment. A fragment-selection process responds to request from a client device. If the video-quality value of the fragment requested exceeds the video-quality threshold, a different fragment having a lower vide-quality value is selected instead. Otherwise, the fragment that would have been selected is selected. In some embodiments, the video-quality threshold can be dynamically adjusted to permit varying amounts of bandwidth reclamation.
Latest GENERAL INSTRUMENT CORPORATION Patents:
CROSS-REFERENCES TO RELATED APPLICATIONS
The present application claims the benefit and priority under 35 U.S.C. 119(e) of U.S. Provisional Application No. 61/537,057, filed Sep. 21, 2011, entitled “Adaptive Streaming to Multicast and Constrained-Fidelity Constant Bit Rate Encoding,” (Attorney Docket number CS39093), U.S. Provisional Application No. 61/537,058, filed Sep. 21, 2011, entitled “Constrained Fidelity Adaptive Bit Rate Encoding Systems and Methods,” (Attorney Docket number CS39094), and U.S. Provisional Application No. 61/537,054, filed Sep. 20, 2011, entitled “Video Quality of Experience Management and Constrained Fidelity Adaptive Bit Rate Encoding Systems and Methods,” (Attorney Docket number CS39095), the contents of all of which are incorporated herein by reference in their entirety.
Adaptive Bit Rate (ABR) streaming is an emerging technology that is rapidly being adopted for commercial real-time distribution of video media and is a technique used in streaming multimedia over computer networks. While in the past most video streaming technologies utilized streaming protocols such real-time transport protocol (RTP) with real-time streaming protocol (RTSP), today's adaptive streaming technologies are almost exclusively based on hypertext transport protocol (HTTP) and designed to work efficiently over large distributed HTTP networks such as the Internet.
Streaming media content can be divided into segments having a fixed duration. ABR streaming protocols have been also been developed. ABR is a method of streaming media content where sequential HTTP progressive downloads in which a continuous media program is delivered as a series of sequential media segments or chunks.
In view of the foregoing, alternative methods of ABR are needed to better accommodate performance and distribution needs of the media content and distributors.
BRIEF DESCRIPTION OF THE DRAWINGS
The accompanying figures where like reference numerals refer to identical or functionally similar elements throughout the separate views and which together with the detailed description below are incorporated in and form part of the specification, serve to further illustrate various embodiments and to explain various principles and advantages all in accordance with the present invention.
Before describing in detail embodiments that are in accordance with the present invention, it should be observed that the embodiments reside primarily in combinations of method steps and apparatus components related to method and apparatus related to variable duration media segments for streaming media content using adaptive bit rate streaming protocols. Accordingly, the apparatus components and method steps have been represented where appropriate by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the embodiments of the present invention so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.
In this document, relational terms such as first and second, top and bottom, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms “comprises,” “comprising,” or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by “comprises . . . a” does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.
It will be appreciated that embodiments of the invention described herein may be comprised of one or more conventional processors and unique stored program instructions that control the one or more processors to implement, in conjunction with certain non-processor circuits, some, most, or all of the functions of variable duration media segments used in streaming media content as described herein. The non-processor circuits may include, but are not limited to, a radio receiver, a radio transmitter, signal drivers, clock circuits, power source circuits, and user input devices. As such, these functions may be interpreted as steps of a method to perform streaming media content using variable duration segments. Alternatively, some or all functions could be implemented by a state machine that has no stored program instructions, or in one or more application specific integrated circuits (ASICs), in which each function or some combinations of certain of the functions are implemented as custom logic. Of course, a combination of the two approaches could be used. Thus, methods and means for these functions have been described herein. Further, it is expected that one of ordinary skill, notwithstanding possibly significant effort and many design choices motivated by, for example, available time, current technology, and economic considerations, when guided by the concepts and principles disclosed herein will be readily capable of generating such software instructions and programs and ICs with minimal experimentation.
Described herein are techniques for providing media content to client devices. In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of particular embodiments. Particular embodiments as defined by the claims may include some or all of the features in these examples alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.
Adaptive bit-rate streaming works by detecting a user's bandwidth and optionally other capabilities, such as CPU capacity, in real time and adjusting the quality of a video stream accordingly. It requires the use of an encoder which can encode a single source video at multiple bit rates. The client switches between streaming the different encodings depending on available resources. The result: very little buffering, fast start time and a good experience for both high-end and low-end connections.
However, as more players enter the field, a major emerging problem in adaptive streaming is that client devices are greedy. Client devices will request the maximum bit rate and quality fragments that are compatible with their current available resources thereby causing uneven, unpredictable, and fluctuating quality of experience for consumers.
Disclosed herein are novel solutions for improving efficiency of bandwidth in distribution of video content by means of adaptive streaming. Also disclosed are novel approaches to addressing the problem of delivering over-the-top (OTT) video content over legacy multicast networks and set top boxes. Finally, disclosed herein are novel approaches for addressing issues of managing the Quality of Experience for video distributed over unmanaged, best-effort networks.
Service providers and operators currently distribute digital video programming to fixed-location set top boxes using a managed pipe or multicast protocol. A managed pipe is an information channel (also referred to as the “pipe”) over which the amount of data allocated to individual programs and subscribers is controlled by the operator of the channel, a cable TV provider, for example. Multicast-based servers can support a larger audience by serving a single content stream simultaneously to multiple users. Multicast is a true broadcast. There is no direct relationship between the clients and media server. The server generates a file containing information that the client needs to listen for the multicast. This is similar to tuning into a station on a radio. Each client that listens to the multicast adds no additional overhead on the server. In fact, the server sends out only one stream per multicast station. The same load is experienced on the server whether only one client or 1,000 clients are listening.
On the other hand, in an unmanaged or unicast-based delivery, for example, over the Internet, media servers open and provide a stream for each unique user. In contrast to the managed or multi-cast pipe, the internet is a best-effort network which the service provider (cable internet, dsl, etc.) does not throttle content based on content type or subscriber identity. The effort to keep the internet an unmanaged pipe is at the center of “net neutrality”.
Returning now to
In contrast, the alternative model (New Model) 102 in
Service providers and operators currently distribute digital video programming to fixed-location set top boxes using a multicast protocol, as described above. As noted above, for mobile media platforms such as tablets and smartphones, adaptive streaming is gaining traction as a method of distributing digital content. The need to support both legacy set-top boxes and mobile platforms that are becoming popular create new challenges for service providers and operators. One challenge and opportunity addressed and solved herein, is to be able to deliver content to either fixed or mobile platforms using the best distribution option for each situation.
In contrast, in HTTP adaptive streaming,
Moving now to
The client device 152 requests media segments that are compatible with its performance profile 301. An encoder 112 formats and sends HTTP media segments, or chunks from the encoder 112 for various bit-rates and formats. An adaptive server 120 sends the media segments 302 to the client device 152 as requested.
As the encoder 112 is a multi-rate and multi-format encoder, the encoder 112 receives an input signal, which may be baseband video or pre-encoded media file or stream and encodes the input stream as output stream 302a. In an embodiment, the output encoded signals 125a-n are configured by the encoder as, for example, MPEG transport stream (MPEG TS) signals. The output stream 302a is stored in adaptive server 120 as chunks 122 a-n at a rate 1-n and a format 1-n. As shown, each rate 123a-n has a different format 121a-n although it is understood that different combinations can be used into media segments or chunks 122a-n. The result/output of the encoder 112 is output encoded signals 125a-n corresponding to the rate and format combinations of the encoder.
These encoded signals 125a-n are supplied to the adaptive bandwidth server 120 and stored, or optionally supplied to the adaptive bandwidth server in real time. The variable media segment signals 125a-n are supplied to the ABR server 120 that can also serve as the media storage and manager entity. Each of the variable duration segments 125a-n is accessible to the ABR server 120 so that the ABR server can stream the requested media content to the device 152.
In an embodiment, the bandwidth manager or adapter 320 can be coupled to the encoder 112. It is also possible to configure the encoder 112 to incorporate the bandwidth manager or adapter 320. This is in contrast with the current state of the art in which the bandwidth manager is associated with each client, therefore bringing about the overextending or competitive grab for bandwidth, a problem which is solved by the embodiments disclosed herein. When coupled to the encoder the bandwidth manager or adapter 320 provides a reasonable or accurate bandwidth requirement for the media segments within the encoded signals 316a-n. In another embodiment, the bandwidth manager or adapter 320 can be coupled to the server 120, or the server 120 is configured with the bandwidth manager or adapter 320. In this embodiment, the bandwidth manager or adapter 320 is provided within the server 120 and determines the analysis required for processing of the variable duration segments 125a-n for use by the ABR server 120.
Efficient Use of Bandwidth in Distribution of Video Content Via ABR Streaming
Delivering Over-the-Top (OTT) Video Content Over Legacy Multicast Networks and STBs
Service providers and operators currently distribute digital video programming to fixed-location set top boxes using a multicast protocol. For mobile media platforms such as tablets and smart phones, adaptive streaming is gaining traction as a method of distributing digital content. However, these existing methods and systems are limited to delivery only to the specifically-programmed client. The need to support both legacy set-top boxes and mobile platforms that are becoming popular create new challenges for service providers and operators. Following are novel embodiments for systems and methods for delivering over-the-top (OTT) video content over legacy multicast networks and set top boxes. In a standalone configuration, the Agile Streamer, as disclosed herein, can be used alongside any number of devices to apply its functionality as disclosed herein and interface with client device requests.
Moving now to
In this embodiment, video content is encoded by the encoder 915. The encoder sends HTTP media fragments or chunks to the server 910. The server 910 can also include or act as a bandwidth manager 916. The encoded chunks are stored and ready to be delivered via the Agile Streaming Adapter 920 to the end device. From the end device or client side, devices such as IPTV and Cable, etc 930, VOD servers 931, ATSC, DVB-T 932, and devices 935a-d, smart device 935a, computer 935b, phone, 935c, etc. are able to receive the same video content stream from the server 910 because the Agile Streaming Adapter 920 converts the media segments to single files in the format required by the end device.
The Agile Adapter 920 is a novel ABR-to-multicast adapter. In one embodiment, it is a module executing code from a memory in an ABR encoder (for creation and conversion of ABR chunks). In one embodiment the Adapter 920 is a module executing, via a processor, code stored in a streaming server that performs stitching and multiplexing (for collection and conversion of pre-existing ABR chunks). In yet another embodiment, it is a stand-alone server or software/hardware board in a server.
The Agile Adapter 920 receives the encoded stream and converts the chunks to single files. In files can be any number of formats as dictated by the client device. For example, some formats or protocols include MPEG-TS, MPEG4, etc. In both cases, targeted use cases would be those such as live over-the-air, IPTV, cable, satellite, video-on-demand, blu_ray discs, etc. (any place an encode-once-use-by-many transport streams could be used.)
In such embodiments, the notion of moving between ABR and multi-cast and vise versa begins to undo the distinctions between a stand-alone dedicated encoder and a HTTP server. In one embodiment, the Adapter 920 as described in the afore-mentioned figures, provides a method of turning an HTTP server into what was formerly thought of as a stand-alone encoder.
Prior art video encoders are designed to operate in one of two distinct ways: variable bit rate mode (VBR) or constant bit rate mode (CBR) mode. In basic terms, a CBR stream is created by adapting or varying the quantization parameters to produce a constant bit rate. With VBR mode, the quantization parameters are nearly static to produce a variable rate stream. DVDs, for instance, are encoded using VBR mode and produce very consistent quality. The fundamental principle is that simple scenes require less bandwidth than scenes with a lot of motion and detail.
In an ideal world, without bandwidth constraints, VBR would universally be used. However, most applications have bandwidth constraints and, therefore, they do not have the luxury of being able to support VBR. These applications require rate control mechanisms that constrain the bit rate within a predefined setting. Various methods such as statistical multiplexing have been developed to allow multiple VBR services to be delivered over a fixed channel by ensuring that the combined bit rate from the encoders does not exceed the bandwidth available from a channel of a fixed size, or, stated differently, does not over subscribe the fixed channel size. However, in a switched broadcast application such as IPTV, services are delivered to a home on an individual basis and statistical multiplexing (statmux) is not an option.
There exists a CBR/VBR hybrid implementation for encoding video data streams that can be thought of as Constrained Fidelity CBR. The system identifies low complexity content which requires less bandwidth to encode with acceptable quality, and reduces below the maximum bandwidth accordingly. This frees additional bandwidth to the channel for use by other services. This can also be described as CBR that will not encode video with more bits than it needs (or constrained fidelity CBR). Alternatively, in at least some implementations, the present invention may also be thought of as VBR with a cap (or capped VBR.)
In addition, for at least some embodiments, the ability to enable null packets may be included for, for example, applications that want to fix the bit rate at one point in the network so that bandwidth reclamation can be performed in downstream devices.
Returning now to
As shown in the above-described embodiments, systems and methods for techniques for delivering over-the-top (OTT) video content over legacy multi-cast networks and set top boxes are disclosed. There are numerous benefits realized as a result of implementation of the systems and methods disclosed. As discussed above, the disclosed methods and systems provide delivery of over-the-top (OTT) video content over legacy multicast networks and set top boxes via the Agile Adapter which converts a HTTP media segment chunk stream into any number of formats required by the end user client device. Also provided is the ability to provide adaptive streaming content over fixed channels that support multiple service types, such as video, data, and voice.
In one embodiment, converting adaptive streaming content to constrained-fidelity bit streams improves bandwidth usage and reduces network congestion because unneeded CF-CBR bandwidth is reclaimed and shared. This can result in an increase of the service operating range for bandwidth-constrained networks, thereby increasing the number of potential customers that could be served by a service provider. This method can also seamlessly allow people to share a network which can overcome inefficiency of unicast adaptive streaming, as compared to multicast streaming, when multiple users try to share a network.
Also provided are techniques by which to increase the service operating range for bandwidth-constrained networks, thereby increase the number of potential customers that could be served by a service provider.
Managing Quality of Experience for Video Distributed Over Unmanaged Networks
Delivery of video programming over the Internet does not typically provide the same quality of experience as delivery over traditional distribution channels, such as: cable, satellite, IPTV, etc. The internet is unmanaged, in the sense that certain resources, such as bandwidth, are not pre-assigned for certain services and users. As a result, use of HTTP adaptive streaming is less reliable and predictable than traditional distribution models. The lack of reliable, predictable quality could negatively impact content provider reputations and revenues.
Moreover, the rapid rise in use of over-the-top (OTT) and adaptive streaming creates new burdens for content-delivery networks (CDNs) that typically act as OTT delivery intermediaries between content providers and consumers. Adaptive streaming client devices are greedy in the sense that they request the maximum bit rate media fragments consistent with their available bandwidth and processing constraints, and without regard to other traffic. Accordingly, there exists a need to improve managing the quality of experience for video distributed over unmanaged, best-effort networks. The embodiments described herein provide a tool by which CDNs and content providers can manage the quality of experience of multiple users, multiple programs, and multiple service groups.
Referring back to
At block 1512, the Agile Streamer Server reviews the streaming content and at 1514 if the video-quality value of the fragment requested exceeds the video-quality threshold, at block 1516 a different fragment having a lower vide-quality value is selected instead, and otherwise, at block 1518 the fragment that would have been selected is selected at block 1510.
In one embodiment, the segments are selected via a weighted apportionment of total available bandwidth that specifies the size, in bits, of media segment to be selected and delivered to each client in the group within a particular interval of time. At block 1612 the Agile Streaming Server adapter combines the media fragments into a seamless multicast stream for delivery to legacy media distribution protocols, including, for example, over-the air broadcast, cable, IPTV and other formats. At block 1614 content is delivered in a single stream to end devices. In one embodiment, constrained-fidelity constant bit-rate (CF-CBR) multicast protocol can be combined with method 1600 resulting in the advantage that a service provider can share video bandwidth with other services as voice and data to improve overall efficiency while maintaining a minimum operational threshold for video quality.
The process begins with one or more devices requesting 1702 content from a media content provider. The media content provider supplies the media content to the device by streaming the media content. In order to stream the media content, an encoder encodes 1704 the streaming media content using known protocols. The encoder can use multirate and multiformat encoding. Then, at block 1708 the chunks can be optionally stored in a server or other repository. At block 1710 a fragment-selection process within the Agile Streaming Server inspects media fragments and selects fragments to deliver to individual clients based on several factors, including, for example: the content requested by the client; the total bandwidth available to all clients in a group; the video qualities of segments that correspond to the content requested by the clients; and the service provider's policies relating to the subscriber priority, program priority, and operation video quality targets.
In one embodiment, the segments are selected via a weighted apportionment of total available bandwidth that specifies the size, in bits, of media segment to be selected and delivered to each client in the group within a particular interval of time.
At block 1712 content is delivered to end devices. Any mixture of web-based protocols, for example HTTP, streaming, download-and-play, etc, is supported. In one embodiment, constrained-fidelity constant bit-rate (CF-CBR) multicast protocol can be combined with method 1700 resulting in the advantage that traditional service provider protocols such as multicast, MPEG transport stream, video-on-demand, over-the-air-mixed broadcast, etc, and any mixture of web-based and traditional protocols can be utilized. The various protocols can be mixed within and between groups. Service groups may be defined by the service or content provider according to any criteria, such as geography, subscription type, device type, connection type, or randomly, etc.
Particular embodiments may be implemented in a non-transitory computer-readable storage medium for use by or in connection with the instruction execution system, apparatus, system, or machine. The computer-readable storage medium contains instructions for controlling a computer system to perform a method described by particular embodiments. The instructions, when executed by one or more computer processors, may be operable to perform that which is described in particular embodiments.
As used in the description herein and throughout the claims that follow, “a”, “an”, and “the” includes plural references unless the context clearly dictates otherwise. Also, as used in the description herein and throughout the claims that follow, the meaning of “in” includes “in” and “on” unless the context clearly dictates otherwise.
The above description illustrates various embodiments along with examples of how aspects of particular embodiments may be implemented. The above examples and embodiments should not be deemed to be the only embodiments, and are presented to illustrate the flexibility and advantages of particular embodiments as defined by the following claims. Based on the above disclosure and the following claims, other arrangements, embodiments, implementations and equivalents may be employed without departing from the scope hereof as defined by the claims.
1. A method for joint management of video quality and bandwidth in adaptive bitrate streaming, the method comprising:
- encoding base band, via an encoder, or pre-encoding video to produce one or more bit streams;
- creating chunks or fragments, via a fragmenter, consistent with one or more adaptive streaming protocols;
- inspecting any combination of the input and output of the encoder, a transcoder, or fragmenter; and
- producing a video-quality metric for each fragment,
- wherein a fragment-selection process responds to request from a client device,
- wherein if the video-quality value of the fragment requested exceeds the video-quality threshold, a different fragment having a lower vide-quality value is selected instead,
- wherein otherwise, the fragment that would have been selected is selected.
2. A method of claim 1, wherein individual fragments are recombined and repackaged into an unfragmented real time delivery protocol.
3. The method of claim 1, wherein the video-quality threshold is dynamically adjusted to permit varying amounts of bandwidth reclamation.
4. The method of claim 1, wherein the video-quality metric can be determined by inspection of bitstreams or fragments to discover quantization coefficients.
5. The method of claim 1, wherein the video-quality metric can be determined by analysis of motion vectors.
6. The method of claim 1, wherein the video-quality metric can be determined by analysis of the level of spatial detail.
7. The method of claim 1, wherein the video-quality metric can be determined by one of full-reference, partial-reference, and no-reference video-quality assessment techniques.
8. The method of claim 1, wherein the video-quality metric can be determined by analysis of coding complexity
9. The method of claim 1, wherein the video-quality metric can be determined by a model of human visual perception
10. The method of claim 1, wherein the video-quality metric is determined on-the-fly (at the same time or near the same time that a request for a fragment is processed.).
11. The method claim 1, wherein the video-quality metric is determined ahead of time and stored in a manner that associates individual video-quality metrics with individual fragments.
12. The method of claim 1, wherein individual fragments are recombined and repackaged into an MPEG2 transport stream.
13. A system, the system comprising:
- an encoder or transcoder for receiving media content;
- a video fragmenter, for dividing the content into fragments;
- a video-quality analyzer that output complexity values for logically dividing the media content into fragments;
- a streaming server,
- a process by which individual fragments are selected for distribution,
- a video-quality threshold.
14. The system of claim 12, wherein the system further comprises a bandwidth reclamation factor.
15. The system of claim 12, wherein the system further comprises a process of storing and associating video-quality metrics and fragments.
16. The system of claim 12, the system further comprising a process for combining and/or repackaging individual fragments.
17. The system of claim 12, the system further comprising a process for distributing fragments to HTTP clients.
18. The system of claim 12, the system further comprising a process for distributing repackaged fragments to MPEG2 TS clients.
19. The system of claim 12, the system further comprising a process for distributing to HTTP clients and to MPEG2 TS clients.
International Classification: H04L 29/06 (20060101);