SYSTEM, SERVER, AND METHOD FOR VARIABLE BIT RATE MULTIMEDIA STREAMING
System, method, and computer program products for implementing techniques for efficient delivery of variable bit rate streaming media assets having a variety of data formats. An embodiment provides delivery system for use in client server computer architecture in which server provides streaming media assets to at least one client over computer network, wherein media assets can have plurality of data formats, including a packet producer that acquires a streaming media asset in packetized form and places time stamps on packets that specify delivery time for each packet, a time stamp packet queue containing the packets with time stamps in a first in, first out order; and a feeder module that removes packets from the time stamp packet queue and transmits the removed packets to a client via the computer network, transmission for each packet concluded at least by the specified delivery time in each packet.
This application is a Divisional Application of and claims the benefit of priority to U.S. patent application Ser. No. 09/917,198 filed 27 Jul. 2001, which itself claims priority to U.S. Provisional Patent Application Ser. No. 60/221,598, filed 28 Jul. 2000, with the United States Patent and Trademark Office, each of which applications are incorporated herein by reference.
FIELDThe present invention relates broadly to delivery of streaming media assets over a computer network having a client server computer architecture. Specifically, the present invention relates to delivery of streaming media assets having variable bit rates and multiple data formats.
BACKGROUNDMultimedia data can be streamed and encoded in many different ways. One aspect of encoding deals with uniformity or variation in the streaming bit rate. The characteristics of the data have implications on design on numerous components of the end to end streaming server system. These components include resources on the server such as disk bandwidth, CPU, memory and network bandwidth on the server, the CPU, memory and network bandwidth on the client as well as the network itself. Delivery of variable bit rate streaming multimedia data presents numerous challenges even with a single data format, but the level of difficulty is increased dramatically when streaming media assets of multiple data formats are to be delivered on a single system.
Another problem streaming media servers face is the resolution of streaming media assets that have to be delivered simultaneously. Scheduling simultaneous delivery has involved large, parallel infrastructure for delivering multiple streams simultaneously in previous approaches that results in idle system resources for an unacceptable amount of time. Idle resources translate directly into higher operating costs and reduced profits for commercial servers. Thus, there is a need for servers that can more efficiently deliver streaming media assets simultaneously to multiple users using infrastructure that is flexible to a variety of media formats.
SUMMARYThe present invention provides a system, methods, and computer program products that implement efficient techniques for variable bit multimedia streaming. In one aspect, the present invention provides a delivery system, methods, and computer executable instructions for use in a client server computer architecture in which the server provides streaming media assets to at least one client over a computer network, wherein the media assets can have a plurality of data formats, comprising a packet producer that acquires a streaming media asset in packetized form and places time stamps on the packets, the time stamps specifying a delivery time for each packet, a time stamp packet queue containing the packets with time stamps in a first in, first out order; and a feeder module that removes packets from the time stamp packet queue and transmits the removed packets to a client via the computer network, the transmission for each packet concluded at least by the specified delivery time in each packet.
In another aspect, the present invention provides a system, methods, and computer executable code for resolving delivery timestamp conflicts between packets from multiple streaming media assets by comparing the delivery times of the packets, and, when a conflict between two or more packets arises, adjusting the delivery timestamp on one or more packets to deliver the packets within the time allowed by the Pre-read size value on the client or by delivering the packets at a time other than the original delivery time specified by the timestamp if the MaxBufSize on the client is large enough to hold additional packets.
In another aspect, the present invention provides a system, methods, and computer executable code for performing admission control for streaming media assets delivered to a client in a client server computer architecture in which the server provides streaming media assets to at least one client over a computer network, by defining a time window in terms of a duration of time, computing a number of bytes that need to be delivered during the time window, the bytes comprising a streaming media asset, translating the computed number of bytes into a time to process value for the first streaming media asset, and admitting for delivery the streaming media asset if the first time to process value is smaller than the time window. Additional streaming media assets can be admitted if their respective time to process values, when added to the first admitted asset for the same time space, is still less than the time window.
Directing attention to
Computer 150 communicates with other computers via communication connection 166 and communication line 168 to allow the computer 150 to be operated remotely, or utilize files stored at different locations, such as content provider 108. Communication connection 156 can be a modem, network interface card, or other device that enables a computer to communicate with other computers. Communication line 168 can be a telephone line or cable, or any medium or channel capable of transferring data between computers. In alternative embodiments, communication connection 166 can be a wireless communication medium, thus eliminating the need for communication line 168. The components described above may be operatively connected by a communications bus 170.
While the use of packets with time stamps allows the delivery system 200 to support non-CBR delivery, the ability to handle a variety of formats, inputs sources, etc. is desirable. The delivery system 200 utilizes packet producer 202 that can service a variety of input sources such as data read from a file, data received from the network 106, data read from a circular disk buffer while synchronizing with another capture process, and the like. Packet producers 202 are implemented as software modules that acquire data to be streamed to the clients 104, parse the acquired data if necessary, and produce time stamped packets for delivery. The packet producers 202 can be specialized to handle specific formats by including, for example, code that parses Quicktime files, locates the hint tracks and constructs the realtime transport protocol (RTP) packets or code that parses ASF files and locates the index entries that are at the end of the file, etc. By providing a plurality of specialized packet producers 202 in the delivery system 200, the delivery system can handle data in any anticipated format.
The time stamped packets produced by the packet producer 202 are sent from the packet producer 202 to the time stamped packet queue 204, a data structure that organizes time stamped packet into a first in, first out queue. While the packet producer 202 is a producer of time stamped packets, a feeder software module 206 removes the packets from the queue and delivers them to the client according to the time stamp on each packet. In an embodiment, both packet producers and feeders are active entities (with an associated thread) and the time stamped packet queue 204 is a passive data structure. Each time stamped packet in the queue doesn't need to contain the packet data in the queue verbatim, but only a pointer to where the data is stored, such as in a buffer 208 that is shared with the packet producers 202.
In an embodiment, a packet producer 202 includes two software components: a stream reader 210 and a stream processor 212. The stream reader 210 produces the data stream by receiving data and sending it to the stream processor 212. The stream processor 212 takes the data from the stream reader 210, parses it if necessary, and produces time stamped packets. Both the stream reader 210 and the stream processor 212 are software components that run in a common thread, with the stream processor 212 calling the stream reader 210 whenever it needs more data.
Since the stream processors are format specific in an embodiment, they can be employed to modify the stream when needed. While only three packet producers are shown in
The packet producers 202 are responsible for dealing with any indexing information that the data might have. The index information may be part of the file itself, for example, such as with ASF, or it could be in a separate file, such as implemented in the current MediaBase system available from Kasenna, Inc. The packet producers can also hide how fast-forward/rewind is implemented and provide flexibility in different ways of supporting fast-forward/rewind. For example, the packet producer can decide whether to use a separate file for supporting FF/REW, or generate the FF/REW stream from the main file on-the-fly. It should be noted that the time stamp added to a packet by a packet producer is meant to be used by the feeder only, and doesn't necessarily correspond to the exact time of presentation as designated in the original media stream. In the case of reverse-play (rewind), the time stamps seen on the packets in the original media stream are decreasing when the stream is traversed in the reverse order. However, the time stamps produced by the packet producers for this reverse stream will always be increasing as these time stamps correspond to the delivery time to be used by the feeder.
In an embodiment, there is one packet queue per active stream in the delivery system 200. One packet producer is placing packets into a packet queue at a given time, and one feeder is removing packets from the packet queue at a given time. Typically, a feeder will deliver multiple streams, and hence will deliver packets from multiple packet queues. As shown in
Feeders can be configured to perform different types of transmission. As shown in
Special cases are accommodated by an embodiment of the feeder of the present invention. One special case involves a situation where two or more packets (possibly from different streams) need to be delivered at precisely the same time. This situation may occur in processing a stream of data conforming to the RTP standard, as the RTP standard allows multiple packets in one stream to have the same time stamp. Since only one packet can be delivered at any given instant, the feeder will miss the deadline for the remaining packets that have the same delivery time specified in its time stamp. A similar situation arises when a feeder ends up spending too much time in sending a large packet, and falling behind the delivery time for the next few packets for other streams handled by the same feeder.
A second special case involves the admission control of streams onto a feeder, namely determining the point beyond which the feeder cannot take up additional streams. If a feeder is currently delivering n streams, and adding an n+1th stream would cause it to miss its deadlines, then the n+1th stream is not added; instead, an additional feeder can be started to handle the stream if possible. In the non-CBR delivery framework, it is difficult to determine whether adding a stream to a feeder would cause it to miss its deadlines.
A scheduling window is defined herein as the time period in which all clients are streamed data that needs to be sent for that time interval. A time to process (TTP) value is defined herein as the time taken to write a data packet to the network. The TTP is calculated for each packet. In a scheduling window, one or more packets may need to be sent that corresponds to a stream. The time to process all these packets is defined as the stream-TTP. With CBR delivery, a stream has the same stream-TTP in every scheduling window. Hence it is easy to determine the point beyond which a feeder cannot take up additional streams. The sum of stream-TTP for all streams cannot exceed the scheduling window. With a constant stream-TTP, it is possible to arrange the delivery schedule such that the feeder is never put into a situation where it has to deliver two packets at precisely the same time (thus avoiding the possibility of missing a deadline). This is done by adjusting the starting time of a stream to make sure that no time clashes will occur between packet time stamps.
In embodiments of the present invention used in non-CBR delivery frameworks, since the time intervals between packets can vary, a packet time stamp conflict can occur rather easily.
The present invention solves conflicting packet time stamps by adjusting the time stamps of the packets within an acceptable range so that their time stamps don't conflict. For example, to resolve the conflict illustrated in
Traditionally, client side buffers located in memory 154 of the client 104 are used to smooth out the jitter in the arrival rate of data at the client side. There are two parameters that are critical in client side buffering. They are (i) the amount of data pre-read before the playout starts (pre-read size), and (ii) the size of the client side buffer (max buffer size). The pre-read size and max buffer size parameters impose the maximum limits on how late or how early a packet can arrive. When media is streamed at a fairly constant rate, if the arrival rate of data into the buffer matches the consumption rate of data by the decoder, then there should be pre-read size data left in the buffer. However, since the data can arrive late or early, buffering helps. The pre-read size data in the client's buffer protects against buffer underflow, if the data is not received in time. The max buffer size protects against overflow of the client's buffer if data starts arriving earlier than expected.
Embodiments of the present invention allow a packet's time stamp to be adjusted based on the client side pre-read size and/or max buffer size parameters. Directing attention to
It should be noted that the larger the pre-read size value, the longer a packet can be delayed. Also, if the max buffer size value is large, packets can be sent earlier than in cases where the max buffer size is small, thus allowing more flexibility in adjusting the time stamps. However, choosing a large pre-read size is not acceptable for video on demand (VOD) playouts where stream control is provided by means of stop, reposition, and fast forward/rewind controls. A larger pre-read size in such a case results in longer delay when a user does a reposition (i.e. moves the slider icon on a media player interface to access a different part of the stream), or switches from normal speed to fast forward. A delay of more than one second may not be acceptable for such transitions. However, for multicast playouts where stream controls are not provided, choosing a larger pre-read size (for example, pre-reading three seconds' worth of data instead of one second's worth of data) may be acceptable and allows the feeder more flexibility in adjusting the time stamps. As far as choosing a large max buffer size, the main limitation is how much memory a client can afford to allocate.
Also, one second's worth of data for 800 Kbps stream is different from one second's worth of data for 1.5 Mbps stream. That is, the parameters pre-read size and max buffer size are related to the bit rate of the media asset. For example, the pre-read size and max buffer size parameter value can vary between movies that are streamed over the network 106. A request for delivery of a media asset, such as an openMovie call made from the client to server returns these parameters from the server to the client indicating how much buffer should be allocated by the client, and how much data should be pre-read before the playout starts.
For variable bit rate (VBR) streams, computing the pre-read size and max buffer size are a bit complicated, but still possible. The table below shows an example of VBR data arrival at a client. The first row indicates kilobytes of data received during each second. The second and third rows show cumulative numbers of bytes received and consumed (respectively) each second assuming that the data is received and consumed at the same rate, with a three second pre-read delay. The fourth row shows the amount of data remaining in the buffer with this three second pre-read delay.
The number of bytes remaining in the buffer at any point in time is a cumulative number of bytes received during the preceding three second window. This means that the pre-read size can be computed as the maximum number of bytes that can be received during any three second time window in the stream. For stored videos, the pre-read size is computed by scanning a file containing the media to be streamed to the client with a moving three second time window and computing the maximum numbers of bytes that can be received during the window. In other words, given such a one or ten second time window the pre-read size for a stored video are computed by scanning the file with a time window of particular size. For live streams, the maximum or average bit rate is used to compute the buffer sizes.
Time windows and space windows are also useful for admission control in feeders. As referred to herein, a space window represents a contiguous amount of data in a media asset file. Space windows can be used when it is optimal to do a unit of work that is expressed in contiguous bytes of data. For example, an essential part of streaming involves retrieving the data to be streamed from a source such as a file system. In many file systems, applications retrieve data from the file system using an optimal I/O size (size S). There are also file systems such as the IRIX XFS file system that allow applications to specify the maximum bit rate at which the application requires to read data; once specified, the file system guarantees that the application will be able to read data at least at the specified rate (rate B). If the media data were delivered at a constant bit rate, it is easy to arrive at the above quantity (B). However, when the media data is to be delivered at a variable bit rate, coming up with the right number for B is difficult. One way to arrive at that number is to divide the total number of bytes in the file by the time in which the data has to be delivered (and therefore retrieved from the file system). However, such an approach can result in not enough data being read on time due to the fact that the media data is delivered with a variable bit rate. In this situation, space windows become useful. It is known that the optimal I/O size is S number of bytes. Scanning the file can reveal the shortest time in which contiguous S bytes have to be delivered. In other words, we scan the file using a space window of S bytes and find out the shortest time in which contiguous S bytes have to be delivered. Dividing the value for S by the time above gives the maximum bit rate requirement of the delivery service (the application) from the file system. Given a space window of some number of bytes (for example, 256K), the file can be scanned with a moving space window to compute the shortest amount of time occupied by such a space window (i.e., the shortest amount of time during which a space window bytes of data need to be delivered). Space windows can form the basis of reserving guaranteed rate input/output (GRIO) bandwidth by the storage manager. Two functions are defined:
(i) bytes timeWindowToBytes (in timeWindowInSeconds), and
(ii) seconds spaceWindowToSeconds (in spaceWindowInBytes)
The timeWindowToBytes function scans the file and return the maximum number of bytes that need to be delivered during a time window period. The spaceWindowToSeconds function scans the file and return the shortest duration of a space window amount of bytes in the stream.
For admission control of streams within a feeder, embodiments of the present invention utilize a time window for comparison with delivery requirements of streaming media assets. Given a time window of a particular size, a feeder can admit a stream if the stream can be delivered within the time window. For example, given a time window of ten minutes, the maximum numbers of bytes that need to be delivered during a ten minute time window for the stream is computed and then translated into the timeToProcess (in time units). The timeToProcess value can be defined as the time expected to be spent by the Feeder in delivering a certain number of bytes. For a stream to be admissible, the sum of the timeToProcess values for all admitted streams should be lower than the time window size chosen, or an overflow will result. Given a time window of size tw, for each stream, the period of intense activity (of duration tw) is identified and the work expected out of the feeder for such a window is computed. To express this concept in the form of an equation, streams that satisfy the following condition as follows are admitted:
VBR delivery can be performed as a sequence of CBR delivery runs. Given a VBR traffic of duration d and bit rate b, the server will arrive at a delivery schedule that best represents the original VBR traffic such as {<b1, d1>, b2, d2>, <b3, d3>, . . . }. With this schedule, the server delivers at a constant bit rate b1 for duration d1, then at constant bit rate b2 for duration d2, and so on, where d1+d2+ . . . =total duration of the media asset. Such delivery methods are well suited for a feeder that delivers the media asset from the beginning to the end, with no stream control.
It should be noted that the packet producer and feeder based design can easily support VBR delivery as a sequence of CBR runs. The feeders can use the computed delivery schedule to perform admission control. The packet producers can modify the time stamps on the packets in the original stream such that the produced packets have time stamps that conform to the computed delivery schedule.
The time window scheme is used during installation of the asset to coordinate optimal client buffering along with feeder admission control and timely delivery of data. The time window scheme is used by the server to specify the optimal pre-read size and maximum buffer size. The server then uses the knowledge of the client's pre-read and maximum buffer size to handle time stamp conflicts.
Some advanced operating systems such as IRIX provide advanced real time facilities. One such facility is Guaranteed Rate Input/Output (GRIO). File systems that provide such bandwidth support run applications that specify the disk bandwidth guarantee needed to satisfy real time schedules. For example, if a multimedia movie is encoded at 1.5 Mbps, then applications can request the same amount of bandwidth from the GRIO facility. GRIO ensures that the application is able to read the media asset at least at the rate of 1.5 Mbps. Disk reads in file systems that support a GRIO-like feature are issued at a multiple of size known as the optimal I/O size, which is typically 256K bytes in streaming media asset delivery servers. A stored video file can be scanned and the shortest time period during which 256K bytes of data need to be sent to the delivery system 200. The maximum bit rate during any 256K period (of space window) can be computed. This is the peak rate at which I/O requests are issued during playout of a streaming media asset.
Time windows can translate into space windows. A space window is a value associated with an amount of data, for example, 256K bytes. If data is delivered at a constant rate, a time window translates directly to a space window. The bit rates computed using space windows are different from (and usually much slower than) the peak delivery rate. Selecting a space window of size 256K averages the bit rate over the window because the space window is the same size as the optimal I/O size. In general, the larger the window size, the better the averaging behavior. The best average bit rate can be computed by taking the size of the entire file and dividing it by the total duration. That is the same as computing the bit rate over a window of size equal to the size of the entire file. The highest bit rate (also known as the peak bandwidth) of delivery occurs when using a shortest window (for example, one packet), which is the bit rate resulting from taking the largest packet of data that occupies the smallest amount of time in the stream's time line.
Having disclosed exemplary embodiments and the best mode, modifications and variations may be made to the disclosed embodiments while remaining within the scope of the present invention as defined by the following claims.
Claims
1. (canceled)
2. (canceled)
3. (canceled)
4. In a client server computer architecture, a method of resolving delivery conflicts between at least two streaming media assets delivered simultaneously by a server to at least one client, wherein the at least one client has a pre read size value that indicates a capability of the client to pre read data, the streaming media assets comprising data packets having delivery time stamps, the method comprising the steps of
- detecting a delivery conflict between the at least two streaming media assets;
- adjusting at least one of the time stamps to indicate an early delivery for at least one of the packets, wherein the adjusted time stamp is adjusted in accordance with the pre read size value; and
- delivering the packets at least by times specified by the time stamps.
5. The method of claim 4, further comprising the step of the client communicating the pre read size value to the server when the client requests delivery of a streaming media asset.
6. The method of claim 4, further comprising the steps of
- the server communicating to the client an optimum value for the pre read size value; and
- the client allocating sufficient resources to accommodate the optimum value for the pre read size value.
7. In a client server computer architecture, a method of resolving delivery conflicts between at least two streaming media assets delivered simultaneously by a server to at least one client, wherein the at least one client has a max buffer size value that indicates a capability of the client to accept delayed data, the streaming media assets comprising data packets having delivery time stamps, the method comprising the steps of
- detecting a delivery conflict between the at least two streaming media assets;
- adjusting at least one of the time stamps to indicate a delayed delivery for at least one of the packets, wherein the adjusted time stamp is adjusted in accordance with the max buffer size value; and
- delivering the packets at least by times specified by the time stamps.
8. The method of claim 7, further comprising the step of the client communicating the max buffer size value to the server when the client requests delivery of a streaming media asset.
9. The method of claim 7, further comprising the steps of
- the server communicating to the client an optimum value for the max buffer size value; and
- the client allocating sufficient resources to accommodate the max buffer size value having the optimum value.
10. (canceled)
11. A computer program product, which, when executed on a computer, resolves delivery conflicts between at least two streaming media assets delivered simultaneously by a server to at least one client, wherein the at least one client has a pre read size value that indicates a capability of the client to pre read data, the streaming media assets comprising data packets having delivery time stamps, by executing the steps of
- detecting a delivery conflict between the at least two streaming media assets;
- adjusting at least one of the time stamps to indicate an early delivery for at least one of the packets, wherein the adjusted time stamp is adjusted in accordance with the pre read size value; and
- delivering the packets at least by times specified by the time stamps.
12. A computer program product, which, when executed on a computer, resolves delivery conflicts between at least two streaming media assets delivered simultaneously by a server to at least one client, wherein the at least one client has a max buffer size value that indicates a capability of the client to accept delayed data, the streaming media assets comprising data packets having delivery time stamps, by executing the steps of:
- detecting a delivery conflict between the at least two streaming media assets;
- adjusting at least one of the time stamps to indicate a delayed delivery for at least one of the packets, wherein the adjusted time stamp is adjusted in accordance with the max buffer size value; and
- delivering the packets at least by times specified by the time stamps.
13. (canceled)
14. (canceled)
15. (canceled)
Type: Application
Filed: Nov 26, 2007
Publication Date: Jun 12, 2008
Inventors: Lakshminarayanan Gunaseelan (Milpitas, CA), Rammohan Kordale (Sunnyvale, CA)
Application Number: 11/945,229
International Classification: H04J 3/16 (20060101);