Systems and methods for seeking within multimedia content during streaming playback

- DIVX, LLC

A receiver driven approach for playback of remote content is described. One embodiment includes obtaining information concerning the content of the media file from the remote server, identifying a starting location within the media sequence, identifying byte ranges of the media file corresponding to media required to play the media sequence from the starting location, requesting the byte ranges required to play the media sequence from the starting location, buffering received bytes of information pending commencement of playback, playing back the buffered bytes of information, receiving a user instruction, identifying byte ranges of the media file corresponding to media required to play the media sequence in accordance with the user instruction, flushing previous byte range requests, and requesting the byte ranges required to play the media in accordance with the user instruction.

Skip to: Description  ·  Claims  ·  References Cited  · Patent History  ·  Patent History
Description
CROSS-REFERENCE TO RELATED APPLICATIONS

The current application is a continuation of U.S. patent application Ser. No. 16/136,149, entitled “Systems and Methods for Seeking Within Multimedia Content During Streaming Playback”, filed Sep. 19, 2018, which is a continuation of U.S. patent application Ser. No. 15/682,379, entitled “Video Distribution System Including Progressive Playback”, filed Aug. 21, 2017, which is a continuation of U.S. patent application Ser. No. 14/632,670, entitled “Video Distribution System Including Progressive Playback”, filed Feb. 26, 2015, which is a continuation of U.S. patent application Ser. No. 12/982,413, entitled “Video Distribution System Including Progressive Playback”, filed Dec. 30, 2010 and issued as U.S. Pat. No. 8,977,768 on Mar. 10, 2015, which is a continuation of U.S. patent application Ser. No. 11/970,493, entitled “Video Distribution System Including Progressive Playback”, filed Jan. 7, 2008 and issued as U.S. Pat. No. 7,886,069 on Feb. 8, 2011, which claims priority to U.S. Provisional Application Ser. No. 60/883,659, entitled “Video Distribution System Including Progressive Playback”, filed Jan. 5, 2007, the disclosures of which are incorporated herein by reference.

BACKGROUND

The present invention relates generally to playing multimedia files over a network and more specifically to the progressive playback of multimedia files as they are downloaded over a network.

Progressive playback is the idea of playing back remote content as it is being downloaded. With this feature a user can select a remote movie and commence watching it before it is fully downloaded. Even with a fast Internet connection, waiting for a movie to fully download can range from minutes to hours depending on the size of the media file. With progressive playback a user only has to wait a couple of seconds before playback can begin.

Current implementations of receiver or player driven progressive playback, while suitable for the short video clips that are dominant in many current applications, are typically limited in the scope and flexibility of the progressive playback they provide. Players typically download files linearly from the beginning to the end. Playback then begins when the player has buffered enough data to provide a likelihood that the media will play without interruption. The buffering requirement can either be a fixed amount suitable for a large percentage of content, or a dynamic amount, where the player infers how much data is required to play the entire content without suffering buffer under-run. Although suitable for playback of short video clips, these methods typically do not support random seeking, trick-play and playback of remotely stored longer content such as feature length movies.

Some systems are implemented with a server driven approach. Examples of server driven approaches include the systems described in U.S. patent applications Ser. Nos. 11/323,044, 11/323,062, 11/327,543, and 11/322,604, the disclosure of which is incorporated herein by reference in its entirety. In these systems, the server parses the data file and determines which data to send. Network efficiency and flexibility in playback becomes a much easier task. Standard HTTP web servers however do not typically provide this functionality, and custom web servers providing this functionality often scale poorly when called upon to deliver content simultaneously to a large number of players.

Browser based players often implement receiver driven playback by parsing the video file as it is downloaded linearly. When a long clip is started, it is impossible to seek or fast-forward to a point in the file that has not already been downloaded. Samba (open source software available at http://us2.samba.org/samba/) can be used to give any application access to a remote file as if it were a local file. It tries to minimize the access latency by pre-caching data from the current file position, which can be randomly set. This may be insufficient when trying to perform “trick play” functions (e.g. performing functions such as rewinding, fast forwarding and skipping between scenes that require non-sequential access of media content). The video frames to be delivered to the player in these scenarios can be spaced far apart or require more complex ordering, greatly diminishing the utility of traditional pre-caching methods which are based on assumptions regarding the subsequent video frames to be viewed.

SUMMARY OF THE INVENTION

Systems and methods are described for performing progressive playback and “trick play” functions on partially downloaded media files. Many embodiments of the invention include a receiver or player driven system supporting features such as the maintenance at all times of a full capacity download stream of only certain required data, including data in certain byte ranges, the discarding of previous requests, and the issuing of new requests for data at the highest priority. Additionally, several embodiments of the invention include features such as random file access at any point in a file, and asynchronous requests, which provide users flexibility in the playback of a file. In a number of embodiments, the systems and processes support scalability for implementation on Internet servers that store files that can contain multiple titles, titles that include multiple audio tracks, and/or titles that include one or more subtitle tracks.

In several embodiments, the ability to provide full featured progressive playback is due in part to the tight coupling of the playback engine for the media sequence (i.e., the system that decodes and plays back the encoded media) with a transport protocol that provides random access to the remote file. Interfacing of the playback engine with the transport protocol via a file parser can reduce latency and enable the client and media server to operate in parallel improving download efficiency and interactivity. In a number of embodiments, the system and processes are configured for use with files that are formatted to include an index to the data within the file and a transport protocol that allows for downloading specific byte ranges within a file.

One embodiment of the method of the invention includes, obtaining information concerning the content of the media file from the remote server, identifying a starting location within the media sequence, identifying byte ranges of the media file corresponding to media required to play the media sequence from the starting location, requesting the byte ranges required to play the media sequence from the starting location, buffering received bytes of information pending commencement of playback, playing back the buffered bytes of information, receiving a user instruction, identifying byte ranges of the media file corresponding to media required to play the media sequence in accordance with the user instruction, flushing previous byte range requests, and requesting the byte ranges required to play the media in accordance with the user instruction.

A further embodiment of the method of the invention includes, maintaining a mask of the portions of the media file that have been downloaded, identifying that at least a portion of a byte range required to play the media in accordance with the user instruction has already been downloaded using the mask, and requesting only the portions of byte ranges that have not already been downloaded from the media server.

Another embodiment of the method of the invention includes storing downloaded bytes in a data file, and outputting the downloaded media file when all bytes of the media file have been downloaded.

In a still further embodiment of the method of the invention, the data file is a sparse data file.

In still another embodiment of the method of the invention, the media file contains a plurality of media sequences and menu information, and identifying a starting location within the media sequence further includes displaying menu information, receiving a user instruction indicative of the selection of the media sequence, and receiving a user instruction indicative of a starting location within the media sequence.

In a yet further embodiment of the method of the invention, the media sequence includes a plurality of interchangeable audio tracks, identifying a starting location within the media sequence further comprises selecting an audio track, and identifying byte ranges of the media file corresponding to media required to play the media sequence from the starting location further comprises selecting byte ranges that do not include the audio tracks that were not selected.

In yet another embodiment of the method of invention, the media sequence includes a plurality of interchangeable subtitle tracks, identifying a starting location within the media sequence further comprises selecting a subtitle track, and identifying byte ranges of the media file corresponding to media required to play the media sequence from the starting location further includes selecting byte ranges that do not include the subtitle tracks that were not selected.

In a further embodiment again of the method of the invention, the sequence includes key frames, and identifying byte ranges of the media file corresponding to media required to play the media in accordance with the user instruction further includes identifying a sequence of key frames in response to a predetermined user instruction, and identifying byte ranges of the media file corresponding to the identified key frames.

One embodiment of the invention includes a media server, a client, and a network. In addition, the client and the media server are configured to communicate via the network, the client is configured to send requests for at least one portion of the media file to the media server, the server is configured to provide requested portions of the media file to the client, and the client is configured to receive user instructions concerning the playback of the media file and to request portions of the media file that have not been downloaded and that are required to comply with the playback instructions from the media server.

In a further embodiment of the invention, proximate portions of the media file grouped and the groups are requested on an earliest deadline first basis.

In another embodiment of the invention, the client is configured to maintain a queue of requested portions of the media file.

In a still further embodiment of the invention, the client and the server are configured to communicate via at least one connection, and the client is configured to flush the queue of requested portions of the media file and break at least one of the connections in response to the receipt of a predetermined user instruction.

In still another embodiment of the invention, the client is configured to store a file map and a data file, the file map contains a mask indicating the portions of the media file that have been downloaded, and the data file contains the downloaded portions of the media file.

In a yet further embodiment of the invention, the data file is a sparse file.

In yet another embodiment of the invention, the media file includes a media sequence and an index, and the client includes a playback engine configured to obtain the index and determine the portions of the media sequence required to comply with user playback instructions, a file parser configured to use the index to map the portions of the media sequence to portions of the media file and a download manager configured to communicate with the media server to download portions of the media file.

A further embodiment again of the invention includes a user interface configured to receive user instructions, a storage device configured to store at least one media file, a network connection, a download manager configured to asynchronously request at least one byte range of a file from a remotely stored media file via the network connection, a playback engine configured to determine portions of a remotely stored media file that must be downloaded in response to user instructions received via the user interface, and a file parser configured to translate requests for portions of a remotely stored media file to byte ranges and to provide the byte ranges to the download manager.

In another embodiment again of the invention, the download manager is configured to create a status file containing a map of blocks of a media file that have been downloaded, and the download manager is configured to create a data file in which to store blocks of a downloaded media file.

In a further additional embodiment of the invention, the download manager is configured to maintain a queue of requested byte ranges.

In another additional embodiment of the invention, the download manager is configured to flush the queue.

In a still yet further embodiment of the invention, the playback engine is configured to generate a menu using menu information obtained from a remote media file.

In still yet another embodiment of the invention, the playback engine is configured to receive a selection of one of a plurality of media sequences in a remote media file via the menu.

In a still further embodiment again of the invention, the playback engine is configured to receive a selection of one of a plurality of audio tracks for a media sequence in a remote media file via the menu.

In still another embodiment again of the invention, the playback engine is configured to receive a selection of a subtitle track for a media sequence in a remote media file via the menu.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a semi-schematic network diagram of progressive playback system in accordance with an embodiment of the invention.

FIG. 2 is a flow chart showing a process for progressively playing back a remotely stored media file in accordance with an embodiment of the invention.

FIG. 3 is a conceptual illustration of a client application configured to request byte ranges from a remote server and to support “trick play” functions in accordance with an embodiment of the invention.

FIG. 4 is a conceptual illustration of a download manager in accordance with an embodiment of the invention.

FIG. 5 is a flow chart showing a process for requesting byte ranges from a media server in accordance with an embodiment of the invention.

FIG. 6 is a flow chart showing a process for flushing a connection with a media server in accordance with an embodiment of the invention.

FIG. 7 is a flow chart showing a process for building a data file during the non-sequential downloading of byte ranges of the data file in accordance with an embodiment of the invention.

FIG. 8 is a flow chart showing a process that can be used by a file parser to identify menu information and media sequences within a remote media file and to extract information from the file in accordance with an embodiment of the invention.

FIG. 9 is a flow chart showing a process used by a playback engine to obtain data chunks from a remote media file formatted using a container format that utilizes chunks in accordance with an embodiment of the invention.

 DETAILED DESCRIPTION OF THE INVENTION

Turning now to the drawings, a system for progressively downloading and playing media is shown. In many embodiments, the media is stored in a file on a remote server and a device configured with a client application retrieves portions of the media file and plays the media. The client application typically does not possess the entire media file when it commences playing and can request non-sequential portions of the media file. In this way, the client application can support “trick play” functions. “Trick play” functions impact the playing of a media file such as non-sequential functions including pausing, rewinding, fast forwarding and skipping between scenes. Instead of sequentially downloading a media file and waiting until the required information has been downloaded to perform a “trick play” function, client applications in accordance with embodiments of the invention can determine portions of a media file that are required to support a specific “trick play” function and request those portions of the file from the remote server. When a “trick play” function involves skipping to portions of the media that have not been downloaded, such as fast forwarding and skipping between chapters, latency can be significantly reduced compared to sequential download.

The configuration of a progressive playback system in accordance with an embodiment of the invention can depend upon the container formats supported by the progressive playback system. Examples of container formats include the AVI 1.0 file format specified by Microsoft Corporation of Redmond, Wash., the OpenDML AVI or AVI 2.0 format, container formats similar to the formats specified in U.S. patent application Ser. Nos. 11/016,184 and 11/198,142, the disclosure of which is incorporated herein by reference in its entirety, MPEG-4 Part 15 (MP4) and the open source format known as Matroska (see www.matroska.org). Depending upon the container file format used, a media file can include multiple titles (i.e. media sequences) and each title can include multiple audio tracks and/or one or more subtitle tracks. The container format of a media file influences the manner in which media information within a media file is located. Therefore, the configuration of a progressive playback system is typically determined based upon the container formats supported in a specific application. Although numerous embodiments are discussed below, other variations appropriate to different container formats can be constructed in accordance with embodiments of the invention.

A progressive playback system in accordance with an embodiment of the invention is shown in FIG. 1. The progressive playback system 10 includes a media server 12 connected to a network 14. Media files are stored on the media server 14 and can be accessed by devices configured with a client application. In the illustrated embodiment, devices that access media files on the media server include a personal computer 16, a consumer electronics device such as a set top box 18 connected to a playback device such as a television 20, and a portable device such as a personal digital assistant 22 or a mobile phone handset. The devices and the media server 12 can communicate over a network 14 that is connected to the Internet 24 via a gateway 26. In other embodiments, the media server 14 and the devices communicate over the Internet.

The devices are configured with client applications that can request portions of media files from the media server 12 for playing. The client application can be implemented in software, in firmware, in hardware or in a combination of the above. In many embodiments, the device plays media from downloaded media files. In several embodiments, the device provides one or more outputs that enable another device to play the media. When the media file includes an index, a device configured with a client application in accordance with an embodiment of the invention can use the index to determine the location of various portions of the media. Therefore, the index can be used to provide a user with “trick play” functions. When a user provides a “trick play” instruction, the device uses the index to determine the portion or portions of the media file that are required in order to execute the “trick play” function and requests those portions from the server. In a number of embodiments, the client application requests portions of the media file using a transport protocol that allows for downloading of specific byte ranges within the media file. One such protocol is the HTTP 1.1 protocol published by The Internet Society or BitTorrent available from www.bittorrent.org. In other embodiments, other protocols and/or mechanisms can be used to obtain specific portions of the media file from the media server.

A flow chart showing a process for requesting media from a media server in accordance with an embodiment of the invention is shown in FIG. 2. The process 40 includes obtaining (42) the index of the media file from the media server. A location from which to start playing the media file is then determined (44). In a number of embodiments, all files commence playing at the start of a media sequence. In several embodiments, the media file can include one or more menus that enable a user to select different locations from which to commence viewing one or more media sequences. Once a location has been determined, the media information required to commence playing the media from the determined location is requested (46) and played back (48) upon receipt. The process involves listening (50) for user instructions. In the event that a user does not provide an instruction, the system continues playing the media in accordance with previous instructions received from the user. When a user provides an instruction, the process determines (52) whether the instruction is to cease playing. Otherwise, the process involves determining (54) the media required to comply with the instruction and requesting (46) the required media. The process continues until the user provides an instruction to stop playing the media or the end of the media sequence is reached.

Media servers in accordance with embodiments of the invention can support progressive playback and trick play functions by simply storing media files and receiving requests for specific byte ranges within the media file. The client application can determine the appropriate byte ranges and the media server simply responds to the byte range requests. A client application that is configured to determine appropriate byte ranges in response to user instructions can be implemented in a variety of ways.

A client application implemented using three abstraction layers in accordance with an embodiment of the invention is illustrated in FIG. 3. The client application 60 includes a download manager 62 that is responsible for coordinating the downloading of specific byte ranges of a file from a remote server. The playback engine 64 is a high level process that coordinates the playback of a media file in response to user interactions. When a media file is being played, the playback engine uses an index of the media file to determine the portions of the media file required to continue playing the media and/or to respond to user instructions. A file parser 66 interfaces between the playback engine 64 and the download manager 62. The file parser maps high level data requests from the playback engine to specific byte ranges that can then be requested using the download manager. The implementation of download managers, file parsers and playback engines in accordance with embodiments of the invention is discussed below. In many embodiments, client applications are configured using alternative architectures that are configured to use an index to a media file to convert user instructions into byte requests that are provided to a remote media server.

A download manager in accordance with an embodiment of the invention is illustrated in FIG. 4. As discussed above, the download manager is responsible for communicating with one or more media servers and obtaining specific byte ranges of media from media files stored on the media servers. The download manager 70 shown in FIG. 4 is configured to instantiate a remote file object 72 and a partial file object 74 to assist with the downloading of media files. The remote file object 72 handles the communications associated with requesting byte ranges of a file from a media server and maintains a queue of the byte ranges that have been requested. The partial file object 74 handles storage of the data downloaded from the media server. The partial file object 74 establishes a temporary data path 75 for a file being downloaded by the download manager.

The temporary data path 75 includes a data file 78 and a status file 76. The data file 76 contains data received from the media server. The status file contains a mask of the data file, where each bit within the mask corresponds to a block of fixed size within the data file. As blocks are downloaded, bits within the mask are set. A status file can also include a region for external data, which can include information, such as the last modified server timestamp, that can be used by the download manager to determine if any partially downloaded data has expired. When the entire media file has been downloaded, the download manager creates an output file path 80 and fully downloaded version of the remote file 82 is output to the download path. At which point, the client application can use the local file to play the media and support “trick play” functions in a conventional manner.

Depending upon the size of the file being downloaded, the data file can be several gigabytes in length. A common file allocation approach is to allocate zeros for every byte within the file, which can take several minutes to complete for large files. Latency during data file allocation can be reduced by allocating the file as a sparse file that only uses the number of bytes actually written to the file. When a sparse file is used, the file allocation process requires very little time. In other embodiments, other file allocation approaches can be used that weight latency against the needs of the download manager.

The block size of the data file (as represented in the status file) determines the granularity by which data can be downloaded. A small block size is typically more efficient in terms of downloading only needed bytes. However small block sizes can lead to a large mask size. In many embodiments, a block size of 128 is used to compromise between efficiency and mask size. In other embodiments, other block sizes determined based upon the requirements of the application are utilized.

A process for requesting data using the download manager in accordance with an embodiment of the invention is shown in FIG. 5. The process 90 commences when a request is received (91) to download a byte range from a media file stored on a remote server. When a download manager similar to the download manager shown in FIG. 4 is used, the download manager instantiates a remote file object and a partial file object and creates the necessary supporting files. A connection is established (92) with the remote server, the requested byte range is placed (94) in a request queue and is then requested (96). As more byte ranges are received, the process determines whether any of the bytes within a requested byte range have been previously downloaded and only places portions of the byte range that have not been previously downloaded in the request queue.

When a download manager similar to that shown in FIG. 4 is used to implement the process 90 in FIG. 5, the mask in the status file 76 is used to determine the requested bytes that have already been stored in the data file 78 and the remaining bytes that should be requested. Each byte range request has associated overhead, therefore, a number of embodiments of the invention include multiple byte ranges in a byte range request and/or search the request queue for byte ranges proximate the byte range at the front of the queue and request a large byte range that encompasses all of the proximate byte range. In several embodiments, the process opens multiple connections to increase download data rate and/or accommodate servers that limit the number of byte requests that can be made via a connection. Again, opening connections has associated overhead. Therefore, the number of connections can be limited based upon a limit appropriate to a particular application (e.g. 5).

When a determination (100) is made that there are no more byte ranges in the request queue, the process determines (102) whether the entire file has been downloaded. In the event that the entire file has not been downloaded, the process requests missing bytes from the partially downloaded file object. Once the entire file is downloaded, the downloaded file is exported to its output directory and the connection with the remote server is closed (104) and the process is complete. In many embodiments, the data file is exported only after playback is complete.

Although a specific process for downloading byte ranges is shown in FIG. 5, variations on this process and/or alternative processes that enable the downloading of specific byte ranges and assembly of a data file can be used in accordance with embodiments of the invention. In addition, processes can involve any of a variety of optimizations to minimize the impact of communication overhead on media playback.

When a user provides a “trick play” instruction, previously requested byte ranges may no longer be required in order to continue playing media in the manner instructed by the user. Download managers in accordance with a number of embodiments of the invention possess the ability to flush the queue of pending byte range requests and establish a new queue of byte range requests. An advantage of flushing a request queue is that there is no latency associated with waiting until previously requested byte ranges have been requested prior to downloading the now higher priority byte ranges. In a number of embodiments, closing the connection with the remote server and opening a new connection further reduces latency. Closing the connection can remove latency associated with waiting for the media server to respond to pending download requests prior to the media server responding to new download requests.

A process for flushing a request queue in accordance with an embodiment of the invention is shown in FIG. 6. The process 110 includes receiving (112) a flush request, flushing (114) the request queue and closing (116) the connection with the media server. In many embodiments, other processes can be used to reduce latency when “trick play” requests are received that eliminate the immediate need for portions of a media file previously requested and create an immediate need for portions of a remote media file that have not been previously requested.

When data is received by the download manager, the status file and the data file are both updated to reflect the received bytes. A process for handling receiving bytes from a remote media server in accordance with an embodiment of the invention is shown in FIG. 7. The process 130 includes receiving (132) bytes, updating (134) the mask in the status file and updating (136) the data file. A determination (138) is then made as to whether the entire file has been downloaded. In the even that the entire file has not been downloaded, the process waits to receive additional bytes. When the entire file has been downloaded, the downloaded media file is exported (140) to its output directory. In other embodiments, other processes are used to organize received byte ranges.

A file parser in accordance with embodiments of the invention is used to convert high level requests from a playback engine into byte range requests for the download manager and to pass byte ranges downloaded by the download manager to the playback engine. When a device commences progressively playing a media file stored on a remote media server, the file parser accesses the file and downloads information concerning the content of the media file. Media files such as the media files described in U.S. patent application Ser. Nos. 11/016,184 and 11/198,142, incorporated by reference above include menu information and/or information from multiple media sequences (i.e. distinct media presentations). The file parser obtains menu information and information concerning the media sequences. When a media sequence is selected by the user, the file parser obtains an index to the selected media sequence and the index is used to identify the byte ranges within the remote media file to request as the media sequence is played.

A process in accordance with an embodiment of the invention for determining the media sequences contained within a remote media file and extracting selected media in accordance with an embodiment of the invention is shown in FIG. 8. The process 150 includes obtaining (152) any menu information contained within the file and information concerning the number of distinct media sequences within the media file. In embodiments where a file parser is used in conjunction with a download manager, the file parser uses knowledge concerning media formats to select bytes of information to request from the media file using the download manager. The menu information and/or the information concerning the number of media sequences can be used to obtain (154) information concerning each of the media sequences. Information that can be useful includes information concerning the title of the media sequence, the format of the media sequence, the number of alternate audio tracks in the media sequence, the presence of one or more subtitle tracks in the media sequence and/or any additional information that could be useful to a user in the selection of a media sequence or to a decoder in the decoding of the media sequence. When the media is formatted in an AVI format or in a format similar to any of the file formats described in U.S. patent application Ser. Nos. 11/016,184 and 11/198,142, information concerning each of the media sequences can be downloaded by downloading the RIFF header for each media sequence. Once information concerning the media sequences has been obtained, a selection (154) can be made concerning the media sequence that is to be played. When the media file contains a single media sequence, the decision can be automatic. When the media file contains multiple sequences, the decision can be made based upon a user instruction that is obtained via a menu interface generated using menu information obtained from the remote media file by the file parser. The file parser uses the information obtained concerning the media sequence to direct the download manager to download a byte range corresponding to an index (156) for the media sequence. The file parser can use the index to extract (158) data from the remote file. The player engine determines the data that is extracted by the file parser. The manner in which the data is extracted depends upon the format of the media file. When the media file is formatted in a media format that utilizes chunks, the file parser uses the index to convert a chunk reference into specific byte ranges that can be retrieved using the download manager. When other formats are used, the file parser uses byte mappings appropriate to the file descriptive information available to the file parser. In addition to requesting byte ranges, file parsers in accordance with embodiments of the invention can communicate with a download manager to check on the status of a particular request and can provide downloaded bytes to the playback engine.

The primary goal of the playback engine, when progressively playing a remote file, is to always maintain a queue of media information required to play the file in the manner requested by the user. When a media file includes an index, the playback engine can refer to the index to determine the media information required to play the media file in the manner requested by the user. A process in accordance with an embodiment of the invention that is used to obtain media from a file that is formatted to represent the media as chunks of information is shown in FIG. 9. The process 190 includes obtaining (192) an index from the remote media file. In embodiments where the playback engine requests information via a file parser, the playback engine can provide an instruction to the file parser to obtain the index and the file parser can extract the necessary information using the download manager. The playback engine then selects (194) chunks based upon instructions, including “trick play” instructions, received from the user and provides instructions to the file parser to download (196) the selected chunks. In a number of embodiments, the playback engine selects chunks based upon an earliest deadline first selection strategy. Chunks from unused audio tracks and unused subtitle tracks multiplexed within the media sequence can be ignored. In many embodiments, media chunks are requested prior to the downloading of the entire index. Media is typically played form the start of a media sequence, therefore, chunks from the start of the media sequence can be downloaded as the index is downloaded. When the playback engine receives the chunks from the file parser, the chunks are queued and provided to an appropriate decoder to enable the playing (198) of the media. Playback of the movie can begin once enough of the movie has been downloaded. The buffered length can be determined by the length of the playback list shared with the chunk download component.

The chunk selection process described above with respect to FIG. 9 maintains a queue of requested chunks. The queue can be maintained as a list of index entries for the requested chunks. The chunk download process polls the download status of the requested chunks. Once downloaded, a chunk is removed from the queue of requested chunks and the downloaded chunk is delivered to the chunk playback process.

When a “trick play” instruction is received, the playback engine selects media information appropriate to the “trick play” instruction. For example, a playback engine that receives a fast-forward or rewind instruction can request only key frames (i.e. complete frames) that are spaced throughout the media sequence at a timing determined by the rate of the trick play function. In many embodiments, the spacing in time is 0.1× the trick frame rate to provide a playback rate during trick play of 10 key frames per second. In other embodiments, various other algorithms are used to determine the media to request. Once the chunks containing the key frames have been identified, the playback engine requests the chunks using the file parser and download manager.

While the above description contains many specific embodiments of the invention, these should not be construed as limitations on the scope of the invention, but rather as an example of one embodiment thereof. Much of the discussion provided above assumes a media file having an index identifying the location of different pieces of media information within the media file. In many embodiments, hierarchical indexes and/or other index formats are included in media files and the playback engine and file parser are configured to accommodate the particular index structure. In several embodiments, the client application is configured to accommodate multiple file formats including file formats that do not possess indexes, but utilize other information to describe the content of the media file. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.

Claims

1. A playback device, comprising:

a processor; and
a non-volatile storage containing an application for causing the processor to perform the steps of: obtaining information concerning the content of a media file from a remote server system; identifying, by a client application, a starting location within the media file; identifying, by the client application, byte ranges of the media file corresponding to media required to play the media file from the starting location; requesting the byte ranges required to play the media file from the starting location; buffering received bytes of information pending commencement of playback; playing back the buffered bytes of information; receiving a user instruction; identifying byte ranges of the media file corresponding to media required to play the media file in accordance with the user instruction; when previously requested byte ranges are no longer required, flushing the previous byte range requests; maintaining a mask indicating byte ranges of the media file that have been downloaded; identifying, using the mask, the byte ranges required to play the media in accordance with the user instruction that have not already been downloaded; and requesting from the remote server in accordance with the user instruction the byte ranges required to play the media file that have been identified to not already been downloaded.

2. The playback device of claim 1, wherein flushing previous byte range requests comprises receiving a flush request, flushing the previous byte requests, closing connection with the remote server.

3. The playback device of claim 1, wherein the application further causes the processor to perform the steps of, after flushing previous byte range requests:

closing the connection with the remote server; and
opening a new connection with the remote server.

4. The playback device of claim 1, wherein the information concerning the content of the media file comprises an index of the media file.

5. The playback device of claim 1, wherein the client application and the remote server are configured to communicate via a network connection and wherein in response to the user instruction, the application further causes the processor to break the network connection.

6. The playback device of claim 1, wherein the user instruction is an instruction for skipping between scenes.

7. The playback device of claim 1, wherein the user instruction is a fast forward or rewind instruction.

8. The playback device of claim 1, wherein the application further causes the processor to store a file map containing the mask indicating the portions of the media file that have been downloaded and a data file containing the downloaded portions of the media file.

9. The playback device of claim 1, wherein the application further causes the processor to store downloaded bytes in a data file; and output the downloaded media file when all bytes of the media file have been downloaded.

10. The playback device of claim 1, wherein the application further causes the processor to instantiate a remote file object and a partial file object, wherein the remote file object handles communications associated with requesting byte ranges of the media file from the remote server and maintains a queue of requested byte ranges; and the partial file object handles storage of the data downloaded from the remote server and establishes a temporary data path for the media file being downloaded.

11. The playback device of claim 1, wherein the media file includes key frames, and identifying byte ranges of the media file corresponding to media required to play the media in accordance with the user instruction comprises identifying a sequence of key frames in response to the user instruction, and identifying byte ranges of the media file corresponding to the identified key frames.

12. The playback device of claim 1, wherein the application further causes the processor to send requests for at least one portion of the media file to the remote server.

13. The playback device of claim 1, wherein the client application is configured to maintain a queue of requested portions of the media file.

14. The playback device of claim 1, wherein the remote server is a standard HTTP server.

15. The playback device of claim 1, wherein the information concerning the content of the media file comprises an index.

16. The playback device of claim 15, wherein the media file is formatted using a container format that utilizes chunks so that each of the identified byte ranges corresponds to a chunk, wherein requesting the byte ranges required to play the media in accordance with the user instruction comprises maintaining a list of index entries for requested chunks.

17. A method of playing back content on a playback device, comprising:

obtaining information concerning the content of a media file from a remote server system using the playback device;
identifying, by a client application, a starting location within the media file;
identifying, by the client application, byte ranges of the media file corresponding to media required to play the media file from the starting location;
requesting the byte ranges required to play the media file from the starting location;
buffering received bytes of information pending commencement of playback;
playing back the buffered bytes of information;
receiving a user instruction;
identifying byte ranges of the media file corresponding to media required to play the media file in accordance with the user instruction;
when previously requested byte ranges are no longer required, flushing previous byte range requests;
maintaining a mask indicating byte ranges of the media file that have been downloaded;
identifying, using the mask, the byte ranges required to play the media in accordance with the user instruction have not already been downloaded; and
requesting from the remote server in accordance with the user instruction the byte ranges required to play the media file that have been identified to not already been downloaded.
Referenced Cited
U.S. Patent Documents
3609227 September 1971 Kuljian
4694491 September 15, 1987 Horne et al.
5132992 July 21, 1992 Yurt et al.
5341474 August 23, 1994 Gelman et al.
5400401 March 21, 1995 Wasilewski et al.
5477263 December 19, 1995 Ocallaghan et al.
5544318 August 6, 1996 Schmitz et al.
5550863 August 27, 1996 Yurt et al.
5574785 November 12, 1996 Ueno et al.
5600721 February 4, 1997 Kitazato
5614940 March 25, 1997 Cobbley et al.
5621794 April 15, 1997 Matsuda et al.
5630005 May 13, 1997 Ort
5642338 June 24, 1997 Fukushima et al.
5761417 June 2, 1998 Henley et al.
5805700 September 8, 1998 Nardone et al.
5813010 September 22, 1998 Kurano et al.
5828370 October 27, 1998 Moeller et al.
5838791 November 17, 1998 Torii et al.
5852664 December 22, 1998 Iverson et al.
5854873 December 29, 1998 Mori et al.
5874986 February 23, 1999 Gibbon et al.
5878135 March 2, 1999 Blatter et al.
5892915 April 6, 1999 Duso et al.
5907658 May 25, 1999 Murase et al.
5923869 July 13, 1999 Kashiwagi et al.
5973679 October 26, 1999 Abbott et al.
6002834 December 14, 1999 Hirabayashi et al.
6009237 December 28, 1999 Hirabayashi et al.
6016381 January 18, 2000 Taira et al.
6038316 March 14, 2000 Dwork et al.
6057832 May 2, 2000 Lev et al.
6065050 May 16, 2000 DeMoney
6108422 August 22, 2000 Newby et al.
6151634 November 21, 2000 Glaser et al.
6199107 March 6, 2001 Dujari
6266483 July 24, 2001 Okada et al.
6282320 August 28, 2001 Hasegawa et al.
6320905 November 20, 2001 Konstantinides
6347145 February 12, 2002 Kato et al.
6351538 February 26, 2002 Uz
6373803 April 16, 2002 Ando et al.
6389473 May 14, 2002 Carmel et al.
6415031 July 2, 2002 Colligan et al.
6445877 September 3, 2002 Okada et al.
6453115 September 17, 2002 Boyle
6453116 September 17, 2002 Ando et al.
6504873 January 7, 2003 Vehvilaeinen
6512883 January 28, 2003 Shim et al.
6516064 February 4, 2003 Osawa et al.
6535920 March 18, 2003 Parry et al.
6578200 June 10, 2003 Takao et al.
6594699 July 15, 2003 Sahai et al.
6654933 November 25, 2003 Abbott et al.
6671408 December 30, 2003 Kaku
6690838 February 10, 2004 Zhou
6721794 April 13, 2004 Taylor et al.
6724944 April 20, 2004 Kalevo et al.
6742082 May 25, 2004 Lango et al.
6751623 June 15, 2004 Basso et al.
6810131 October 26, 2004 Nakagawa et al.
6813437 November 2, 2004 Ando et al.
6871006 March 22, 2005 Oguz et al.
6912513 June 28, 2005 Candelore
6931531 August 16, 2005 Takahashi
6931543 August 16, 2005 Pang et al.
6957350 October 18, 2005 Demos
6965646 November 15, 2005 Firestone
6970564 November 29, 2005 Kubota et al.
6983079 January 3, 2006 Kim
7006757 February 28, 2006 Ando et al.
7007170 February 28, 2006 Morten
7020287 March 28, 2006 Unger
7023992 April 4, 2006 Kubota et al.
7043021 May 9, 2006 Graunke et al.
7051110 May 23, 2006 Hagai et al.
7058177 June 6, 2006 Trimberger et al.
7073191 July 4, 2006 Srikantan et al.
7110542 September 19, 2006 Tripathy
7120250 October 10, 2006 Candelore
7124303 October 17, 2006 Candelore et al.
7130908 October 31, 2006 Pecus et al.
7139868 November 21, 2006 Parry et al.
7143289 November 28, 2006 Denning et al.
7143433 November 28, 2006 Duan et al.
7151832 December 19, 2006 Fetkovich et al.
7167560 January 23, 2007 Yu
7188183 March 6, 2007 Paul et al.
7203313 April 10, 2007 England et al.
7212726 May 1, 2007 Zetts
7231516 June 12, 2007 Sparrell et al.
7233669 June 19, 2007 Candelore
7233948 June 19, 2007 Shamoon et al.
7242772 July 10, 2007 Tehranchi
7274861 September 25, 2007 Yahata et al.
7295673 November 13, 2007 Grab et al.
7302490 November 27, 2007 Gupta et al.
7315829 January 1, 2008 Tagawa et al.
7346163 March 18, 2008 Pedlow, Jr. et al.
7349886 March 25, 2008 Morten et al.
7349976 March 25, 2008 Glaser et al.
7352956 April 1, 2008 Winter et al.
7363647 April 22, 2008 Fakharzadeh
7376233 May 20, 2008 Candelore et al.
7382879 June 3, 2008 Miller
7397853 July 8, 2008 Kwon et al.
7400679 July 15, 2008 Kwon et al.
7406176 July 29, 2008 Zhu et al.
7418132 August 26, 2008 Hoshuyama
7443449 October 28, 2008 Momosaki et al.
7457415 November 25, 2008 Reitmeier et al.
7499930 March 3, 2009 Naka et al.
7539213 May 26, 2009 Guillemot et al.
7546641 June 9, 2009 Robert et al.
7577980 August 18, 2009 Kienzle et al.
7623759 November 24, 2009 Shimoda
7624337 November 24, 2009 Sull et al.
7627750 December 1, 2009 Chan
7627888 December 1, 2009 Ganesan et al.
7639921 December 29, 2009 Seo et al.
7640358 December 29, 2009 Deshpande
7640435 December 29, 2009 Morten
7644172 January 5, 2010 Stewart et al.
7653686 January 26, 2010 Yoneda
7664872 February 16, 2010 Osborne et al.
7697686 April 13, 2010 Puiatti et al.
7702925 April 20, 2010 Hanko et al.
7711052 May 4, 2010 Hannuksela et al.
7734806 June 8, 2010 Park
7756270 July 13, 2010 Shimosato et al.
7756271 July 13, 2010 Zhu et al.
7787622 August 31, 2010 Sprunk
7797720 September 14, 2010 Gopalakrishnan et al.
7840693 November 23, 2010 Gupta et al.
7853980 December 14, 2010 Pedlow, Jr. et al.
7864186 January 4, 2011 Robotham et al.
7877002 January 25, 2011 Ikeda et al.
7881478 February 1, 2011 Derouet
7885405 February 8, 2011 Bong
7886069 February 8, 2011 Osborne
7895311 February 22, 2011 Juenger
7907833 March 15, 2011 Lee
7945143 May 17, 2011 Yahata et al.
7962942 June 14, 2011 Craner
7970835 June 28, 2011 St
7974714 July 5, 2011 Hoffberg
8001471 August 16, 2011 Shaver et al.
8015491 September 6, 2011 Shaver et al.
8073900 December 6, 2011 Guedalia et al.
8131875 March 6, 2012 Chen
8135041 March 13, 2012 Ramaswamy
8160157 April 17, 2012 Lamy-Bergot et al.
8169916 May 1, 2012 Pai et al.
8170210 May 1, 2012 Manders et al.
8213607 July 3, 2012 Rose et al.
8213768 July 3, 2012 Morioka et al.
8218439 July 10, 2012 Deshpande
8243924 August 14, 2012 Chen et al.
8286213 October 9, 2012 Seo
8311094 November 13, 2012 Kamariotis et al.
8312079 November 13, 2012 Newsome et al.
8369421 February 5, 2013 Kadono et al.
8380041 February 19, 2013 Barton et al.
8397265 March 12, 2013 Henocq et al.
8472792 June 25, 2013 Butt et al.
8514926 August 20, 2013 Ro et al.
8526610 September 3, 2013 Shamoon et al.
8543842 September 24, 2013 Ginter et al.
8555329 October 8, 2013 Fröjdh et al.
8571993 October 29, 2013 Kocher et al.
8649669 February 11, 2014 Braness et al.
8650599 February 11, 2014 Shindo et al.
8683066 March 25, 2014 Hurst et al.
8731369 May 20, 2014 Li et al.
8782268 July 15, 2014 Pyle et al.
8818896 August 26, 2014 Candelore
8819116 August 26, 2014 Tomay et al.
8849950 September 30, 2014 Stockhammer et al.
8977768 March 10, 2015 Osborne
9015782 April 21, 2015 Acharya et al.
9038116 May 19, 2015 Knox et al.
9038121 May 19, 2015 Kienzle et al.
9485469 November 1, 2016 Kahn et al.
9615061 April 4, 2017 Carney et al.
9674254 June 6, 2017 Pare et al.
9761274 September 12, 2017 Delpuch et al.
9794318 October 17, 2017 Osborne
9967521 May 8, 2018 Kahn et al.
10171873 January 1, 2019 Krebs
10412141 September 10, 2019 Osborne
10574716 February 25, 2020 Osborne
20010021276 September 13, 2001 Zhou
20010052077 December 13, 2001 Fung et al.
20010052127 December 13, 2001 Seo et al.
20010053222 December 20, 2001 Wakao et al.
20020048450 April 25, 2002 Zetts
20020067432 June 6, 2002 Kondo et al.
20020075572 June 20, 2002 Boreczky et al.
20020107802 August 8, 2002 Philips
20020114330 August 22, 2002 Cheung et al.
20020135607 September 26, 2002 Kato et al.
20020138619 September 26, 2002 Ramaley et al.
20020141503 October 3, 2002 Kobayashi et al.
20020154779 October 24, 2002 Asano et al.
20020161797 October 31, 2002 Gallo
20020164024 November 7, 2002 Arakawa et al.
20020169926 November 14, 2002 Pinckney et al.
20020169971 November 14, 2002 Asano et al.
20030002577 January 2, 2003 Pinder
20030043847 March 6, 2003 Haddad
20030044080 March 6, 2003 Frishman et al.
20030051237 March 13, 2003 Sako et al.
20030053541 March 20, 2003 Sun et al.
20030061369 March 27, 2003 Aksu et al.
20030063675 April 3, 2003 Kang et al.
20030077071 April 24, 2003 Lin et al.
20030079222 April 24, 2003 Boykin et al.
20030081776 May 1, 2003 Candelore
20030135633 July 17, 2003 Dror et al.
20030135742 July 17, 2003 Evans
20030142594 July 31, 2003 Tsumagari et al.
20030152224 August 14, 2003 Candelore et al.
20030169815 September 11, 2003 Aggarwal
20030206717 November 6, 2003 Yogeshwar et al.
20030236907 December 25, 2003 Stewart et al.
20040001594 January 1, 2004 Krishnaswamy et al.
20040003008 January 1, 2004 Wasilewski et al.
20040022391 February 5, 2004 Obrien
20040028227 February 12, 2004 Yu
20040037421 February 26, 2004 Truman
20040047592 March 11, 2004 Seo et al.
20040047607 March 11, 2004 Seo et al.
20040049690 March 11, 2004 Candelore et al.
20040049694 March 11, 2004 Candelore
20040073917 April 15, 2004 Pedlow et al.
20040076237 April 22, 2004 Kadono et al.
20040081333 April 29, 2004 Grab et al.
20040088557 May 6, 2004 Malcolm et al.
20040093494 May 13, 2004 Nishimoto et al.
20040101059 May 27, 2004 Joch et al.
20040101142 May 27, 2004 Nasypny
20040107356 June 3, 2004 Shamoon et al.
20040213094 October 28, 2004 Suzuki
20040243714 December 2, 2004 Wynn et al.
20040267952 December 30, 2004 He et al.
20050005143 January 6, 2005 Lang et al.
20050013494 January 20, 2005 Srinivasan et al.
20050063541 March 24, 2005 Candelore
20050066063 March 24, 2005 Grigorovitch et al.
20050076232 April 7, 2005 Kawaguchi
20050102371 May 12, 2005 Aksu
20050120132 June 2, 2005 Hutter
20050138655 June 23, 2005 Zimler et al.
20050144468 June 30, 2005 Northcutt
20050177741 August 11, 2005 Chen et al.
20050190911 September 1, 2005 Pare et al.
20050192904 September 1, 2005 Candelore
20050198364 September 8, 2005 Val et al.
20050207442 September 22, 2005 Zoest
20050207578 September 22, 2005 Matsuyama et al.
20050216752 September 29, 2005 Hofmeyr et al.
20050227773 October 13, 2005 Lu et al.
20050243912 November 3, 2005 Kwon et al.
20050262257 November 24, 2005 Major et al.
20050265555 December 1, 2005 Pippuri
20060013568 January 19, 2006 Rodriguez
20060026654 February 2, 2006 An et al.
20060037057 February 16, 2006 Xu
20060059223 March 16, 2006 Klemets et al.
20060093318 May 4, 2006 Cohen et al.
20060095472 May 4, 2006 Krikorian et al.
20060109856 May 25, 2006 Deshpande
20060129909 June 15, 2006 Butt et al.
20060161635 July 20, 2006 Lamkin et al.
20060165163 July 27, 2006 Burazerovic et al.
20060168291 July 27, 2006 Van Zoest et al.
20060168298 July 27, 2006 Aoki et al.
20060174021 August 3, 2006 Osborne et al.
20060174026 August 3, 2006 Robinson et al.
20060195884 August 31, 2006 Van Zoest et al.
20060200744 September 7, 2006 Bourke et al.
20060210245 September 21, 2006 Mccrossan et al.
20060212370 September 21, 2006 Shear et al.
20060218251 September 28, 2006 Tanabe
20060235883 October 19, 2006 Krebs
20060294212 December 28, 2006 Kikkawa et al.
20070044010 February 22, 2007 Sull et al.
20070047645 March 1, 2007 Takashima
20070055982 March 8, 2007 Spilo
20070067472 March 22, 2007 Maertens et al.
20070083467 April 12, 2007 Lindahl et al.
20070083663 April 12, 2007 Tanabe et al.
20070106863 May 10, 2007 Bonwick et al.
20070157267 July 5, 2007 Lopez-Estrada
20070162568 July 12, 2007 Gupta
20070162981 July 12, 2007 Morioka et al.
20070166000 July 19, 2007 Nallur et al.
20070180051 August 2, 2007 Kelly et al.
20070201502 August 30, 2007 Abramson
20070204003 August 30, 2007 Abramson
20070204011 August 30, 2007 Shaver et al.
20070204115 August 30, 2007 Abramson
20070209005 September 6, 2007 Shaver et al.
20070220118 September 20, 2007 Loyer
20070250536 October 25, 2007 Tanaka et al.
20080008455 January 10, 2008 De Lange et al.
20080022005 January 24, 2008 Wu et al.
20080071838 March 20, 2008 Moriya et al.
20080082576 April 3, 2008 Bodin et al.
20080086570 April 10, 2008 Dey et al.
20080101718 May 1, 2008 Yang et al.
20080137847 June 12, 2008 Candelore et al.
20080155615 June 26, 2008 Craner et al.
20080160911 July 3, 2008 Chou et al.
20080168133 July 10, 2008 Osborne
20080177793 July 24, 2008 Epstein et al.
20090010622 January 8, 2009 Yahata et al.
20090013195 January 8, 2009 Ochi et al.
20090067367 March 12, 2009 Buracchini et al.
20090077143 March 19, 2009 Macy, Jr.
20090106082 April 23, 2009 Senti et al.
20090132599 May 21, 2009 Soroushian et al.
20090178090 July 9, 2009 Oztaskent
20090249081 October 1, 2009 Zayas
20090282162 November 12, 2009 Mehrotra et al.
20090310819 December 17, 2009 Hatano
20100142915 June 10, 2010 Mcdermott et al.
20100185854 July 22, 2010 Burns et al.
20100198943 August 5, 2010 Harrang et al.
20110010466 January 13, 2011 Fan et al.
20110035517 February 10, 2011 Minnick et al.
20110058675 March 10, 2011 Brueck et al.
20110083009 April 7, 2011 Shamoon et al.
20110096828 April 28, 2011 Chen et al.
20110099225 April 28, 2011 Osborne
20110103374 May 5, 2011 Lajoie et al.
20110135090 June 9, 2011 Chan et al.
20110145858 June 16, 2011 Philpott et al.
20110158470 June 30, 2011 Martin et al.
20110170687 July 14, 2011 Hyodo et al.
20110173345 July 14, 2011 Knox et al.
20110179185 July 21, 2011 Wang et al.
20110197261 August 11, 2011 Dong et al.
20110246661 October 6, 2011 Manzari et al.
20110296048 December 1, 2011 Knox et al.
20110314130 December 22, 2011 Strasman
20120005312 January 5, 2012 Mcgowan et al.
20120042090 February 16, 2012 Chen et al.
20120047542 February 23, 2012 Lewis et al.
20120110120 May 3, 2012 Willig et al.
20120167132 June 28, 2012 Mathews et al.
20120311174 December 6, 2012 Bichot et al.
20120331167 December 27, 2012 Hunt
20130013803 January 10, 2013 Bichot et al.
20130080267 March 28, 2013 McGowan
20140140253 May 22, 2014 Lohmar et al.
20140149557 May 29, 2014 Lohmar et al.
20150172351 June 18, 2015 Osborne
20150288530 October 8, 2015 Oyman
20170011055 January 12, 2017 Pitts
20170353520 December 7, 2017 Osborne
20180046949 February 15, 2018 Kahn et al.
20180255366 September 6, 2018 Lockett et al.
20190020704 January 17, 2019 Osborne
Foreign Patent Documents
2237293 July 1997 CA
2306524 September 2001 CA
1575595 February 2005 CN
1581971 February 2005 CN
1596403 March 2005 CN
1801929 July 2006 CN
101636726 January 2010 CN
101636726 October 2013 CN
103559165 February 2014 CN
103561278 February 2014 CN
103559165 August 2016 CN
103561278 April 2017 CN
1158799 November 2001 EP
1453319 September 2004 EP
1534013 May 2005 EP
1536646 June 2005 EP
1283640 October 2006 EP
2122482 November 2009 EP
2180664 April 2010 EP
2360923 August 2011 EP
2122482 November 2018 EP
3467666 April 2019 EP
3467666 March 2021 EP
2398210 August 2004 GB
H1175178 March 1999 JP
2003504984 February 2003 JP
2003111048 April 2003 JP
2003111048 April 2003 JP
2004295568 October 2004 JP
2004362099 December 2004 JP
2005149029 June 2005 JP
2005173241 June 2005 JP
2005518726 June 2005 JP
2005284041 October 2005 JP
2005341334 December 2005 JP
2006074511 March 2006 JP
4516082 May 2010 JP
2010516123 May 2010 JP
2014197879 October 2014 JP
5894220 March 2016 JP
20040039852 May 2004 KR
20060030164 April 2006 KR
20060106250 October 2006 KR
20060116967 November 2006 KR
20070020727 February 2007 KR
2328040 June 2008 RU
153621 May 2012 SG
199800973 January 1998 WO
199834405 August 1998 WO
1998047290 October 1998 WO
2000049762 August 2000 WO
2000049763 August 2000 WO
2001006788 January 2001 WO
200223315 March 2002 WO
2003028293 April 2002 WO
2002035832 May 2002 WO
2002054776 July 2002 WO
2002073437 September 2002 WO
2002087241 October 2002 WO
2003046750 June 2003 WO
2003047262 June 2003 WO
2003061173 July 2003 WO
2003071800 August 2003 WO
2003088665 October 2003 WO
2004012378 February 2004 WO
2004100158 November 2004 WO
2005008385 January 2005 WO
2005015935 February 2005 WO
2005057906 June 2005 WO
2005125214 December 2005 WO
2006045334 May 2006 WO
2007072257 June 2007 WO
2007093923 August 2007 WO
2007101182 September 2007 WO
2008032908 March 2008 WO
2008086313 July 2008 WO
2009006302 January 2009 WO
2009109976 September 2009 WO
2011087449 July 2011 WO
2011101371 August 2011 WO
2011103364 August 2011 WO
Other references
  • Extended European Search Report for European Application No. 18206048.3, Search completed Feb. 8, 2019 , dated Feb. 21, 2019, 11 Pgs.
  • Adobe—Development Center: Flash video learning guide, printed Jan. 13, 2009 from. http://www.adobe.com/devnet/flash/articles/video_guide_02.html, 5 pgs.
  • International Preliminary Report on Patentability for International Application No. PCT/US2008/050440, Report Completed Aug. 7, 2009, dated Aug. 11, 2009, 8 pgs.
  • International Search Report for International Application No. PCT/US2008/050440, International Filing Date Jan. 7, 2008, Search completed Apr. 23, 2008, dated May 16, 2008, 2 pgs.
  • RedOrbit News, New DivX Web Player Hits 1 Milling Downloads in One Week, printed Jan. 12, 2009 from http://www.redorbit.com/modules/news/tools.php?tool=print&id=421307, 2 pgs.
  • Vuze HD Network, printed Jun. 1, 2009 from http://www.vuze.com/Index.html, 1 pg.
  • Written Opinion of international Application No. PCT/US2008/050440; International filing date Jan. 7, 2008, Opinion completed Apr. 23, 2008, dated May 16, 2008, 9 pgs.
  • Extended European Search Report for European Application No. 08705745.1, Search completed Dec. 16, 2011, dated Dec. 28, 2011, 9 pgs.
  • Collet, “Delivering Protected Content, An Approach for Next Generation Mobile Technologies”, Thesis, 2010, 84 pgs.
  • Diamantis et al., “Real Time Video Distribution using Publication through a Database”, Proceedings SIBGRAPI'98. International Symposium on Computer Graphics, Image Processing, and Vision (Cat. No. 98EX237), Oct. 1990, 8 pgs.
  • Dworkin, “Recommendation for Block Cipher Modes of Operation: Methods and Techniques”, NIST Special Publication 800-38A, 2001, 66 pgs.
  • Fang et al., “Real-time deblocking filter for MPEG-4 systems”, Asia-Pacific Conference on Circuits and Systems, Oct. 28-31, 2002, Bail, Indonesia, 4 pgs.
  • Fecheyr-Lippens, “A Review of HTTP Live Streaming”, Jan. 2010, 38 pgs.
  • Fukuda et al., “Reduction of Blocking Artifacts by Adaptive DCT Coefficient Estimation in Block-Based Video Coding”, Proceedings 2000 International Conference on Image Processing, Sep. 10-13, 2000, Vancouver, BC, Canada, 4 pgs.
  • Huang, U.S. Pat. No. 7,729,426, U.S. Appl. No. 11/230,794, filed Sep. 20, 2005.
  • Huang et al., “Adaptive MLP post-processing for block-based coded images”, IEEE Proceedings—Vision, Image and Signal Processing, vol. 147, No. 5, Oct. 2000, pp. 463-473.
  • Huang et al., “Architecture Design for Deblocking Filter in H.264/JVT/AVC”, 2003 International Conference on Multimedia and Expo., Jul. 6-9, 2003, Baltimore, MD, 4 pgs.
  • Jain et al., U.S. Appl. No. 61/522,623, filed Aug. 11, 2011.
  • Jung et al., “Design and Implementation of an Enhanced Personal Video Recorder for DTV”, IEEE Transactions on Consumer Electronics, vol. 47, No. 4, Nov. 2001, 6 pgs.
  • Kalva, Hari “Delivering MPEG-4 Based Audio-Visual Services”, 2001, 113 pgs.
  • Kang et al., “Access Emulation and Buffering Techniques for Steaming of Non-Stream Format Video Files”, IEEE Transactions on Consumer Electronics, vol. 43, No. 3, Aug. 2001, 7 pgs.
  • Kim et al, “A Deblocking Filter with Two Separate Modes in Block-based Video Coding”, IEEE transactions on circuits and systems for video technology, vol. 9, No. 1, 1999, pp. 156-160.
  • Kim et al., “Tree-Based Group Key Agreement”, Feb. 2004, 37 pgs.
  • Laukens, “Adaptive Streaming—A Brief Tutorial”, EBU Technical Review, 2011, 6 pgs.
  • Legault et al., “Professional Video Under 32-bit Windows Operating Systems”, SMPTE Journal, vol. 105, No. 12, Dec. 1996, 10 pgs.
  • Li et al., “Layered Video Multicast with Retransmission (LVMR): Evaluation of Hierarchical Rate Control”, Proceedings of IEEE INFOCOM'98, the Conference on Computer Communications. Seventeenth Annual Joint Conference of the IEEE Computer and Communications Societies. Gateway to the 21st Century, Cat. No. 98, vol. 3, 1998, 26 pgs.
  • List et al., “Adaptive deblocking filter”, IEEE transactions on circuits and systems for video technology, vol. 13, No. 7, Jul. 2003, pp. 614-619.
  • Massoudi et al., “Overview on Selective Encryption of Image and Video: Challenges and Perspectives”, EURASIP Journal on Information Security, Nov. 2008, 18 pgs.
  • Mccanne et al., “Receiver-driven Layered Multicast”, Conference proceedings on Applications, technologies, architectures, and protocols for computer communications, Aug. 1996, 14 pgs.
  • Meier, “Reduction of Blocking Artifacts in Image and Video Coding”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 9, No. 3, Apr. 1999, pp. 490-500.
  • Nelson, “Smooth Streaming Deployment Guide”, Microsoft Expression Encoder, Aug. 2010, 66 pgs.
  • Newton et al., “Preserving Privacy by De-identifying Facial Images”, Carnegie Mellon University School of Computer Science, Technical Report, CMU-CS-03-119, Mar. 2003, 26 pgs.
  • O'Brien, U.S. Appl. No. 60/399,846, filed Jul. 30, 2002.
  • O'Rourke, “Improved Image Decompression for Reduced Transform Coding Artifacts”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 5, No. 6, Dec. 1995, pp. 490-499.
  • Park et al., “A postprocessing method for reducing quantization effects in low bit-rate moving picture coding”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 9, No. 1, Feb. 1999, pp. 161-171.
  • Richardson, “H.264 and MPEG-4 Video Compression”, Wiley, 2003, 306 pgs. (presented in 2 parts).
  • Sima et al., “An Efficient Architecture for Adaptive Deblocking Filter of H.264 AVC Video Coding”, IEEE Transactions on Consumer Electronics, vol. 50, No. 1, Feb. 2004, pp. 292-296.
  • Spanos et al., “Performance Study of a Selective Encryption Scheme for the Security of Networked, Real-Time Video”, Proceedings of the Fourth International Conference on Computer Communications and Networks, IC3N'95, Sep. 20-23, 1995, Las Vegas, NV, pp. 2-10.
  • Srinivasan et al., “Windows Media Video 9: overview and applications”, Signal Processing: Image Communication, 2004, 25 pgs.
  • Stockhammer, “Dynamic Adaptive Streaming over HTTP—Standards and Design Principles”, Proceedings of the second annual ACM conference on Multimedia, Feb. 2011, pp. 133-145.
  • Timmerer et al., “HTTP Streaming of MPEG Media”, Proceedings of Streaming Day, 2010, 4 pgs.
  • Tiphaigne et al., “A Video Package for Torch”, Jun. 2004, 46 pgs.
  • Trappe et al., “Key Management and Distribution for Secure Multimedia Multicast”, IEEE Transaction on Multimedia, vol. 5, No. 4, Dec. 2003, pp. 544-557.
  • Van Deursen et al., “On Media Delivery Protocols in the Web”, 2010 IEEE International Conference on Multimedia and Expo, Jul. 19-23, 2010, 6 pgs.
  • Ventura, Guillermo Albaida “Streaming of Multimedia Learning Objects”, AG Integrated Communication System, Mar. 2003, 101 pgs.
  • Waggoner, “Compression for Great Digital Video”, 2002, 184 pgs.
  • Watanabem et al., “MPEG-2 decoder enables DTV trick plays”, esearcher System LSI Development Lab, Fujitsu Laboratories Ltd., Kawasaki, Japan, Jun. 2001, 2 pgs.
  • Wiegand, “Joint Video Team (JVT) of ISO/IEC MPEG and ITU-T VCEG”, Jan. 2002, 70 pgs.
  • Willig et al., U.S. Appl. No. 61/409,285, filed Nov. 2, 2010, 43 pgs.
  • Yang et al., “Projection-Based Spatially Adaptive Reconstruction of Block-Transform Compressed Images”, IEEE Transactions on Image Processing, vol. 4, No. 7, Jul. 1995, pp. 896-908.
  • Yang et al., “Regularized Reconstruction to Reduce Blocking Artifacts of Block Discrete Cosine Transform Compressed Images”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 3, No. 6, Dec. 1993, pp. 421-432.
  • Yu et al., “Video deblocking with fine-grained scalable complexity for embedded mobile computing”, Proceedings 7th International Conference on Signal Processing, Aug. 31-Sep. 4, 2004, pp. 1173-1178.
  • Zakhor, “Iterative Procedures for Reduction of Blocking Effects in Transform Image Coding”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 2, No. 1, Mar. 1992, pp. 91-95.
  • Information Technology—MPEG Systems Technologies—Part 7: Common Encryption in ISO Base Media File Format Files (ISO/IEC 23001-7), Apr. 2015, 24 pgs.
  • ISO/IEC 14496-12 Information technology—Coding of audio-visual objects—Part 12: ISO base media file format, Feb. 2004 (“MPEG-4 Part 12 Standard”), 62 pgs.
  • ISO/IEC 14496-12:2008(E) Informational Technology—Coding of Audio-Visual Objects Part 12: ISO Base Media File Format, Oct. 2008, 120 pgs.
  • ISO/IEC FCD 23001-6 MPEG systems technologies Part 6: Dynamic adaptive streaming over HTTP (DASH), Jan. 28, 2011, 86 pgs.
  • Microsoft Corporation, Advanced Systems Format (ASF) Specification, Revision 01.20.03, Dec. 2004, 121 pgs.
  • MPEG-DASH presentation at Streaming Media West 2011, Nov. 2011, 14 pgs.
  • Pomelo, LLC Tech Memo, Analysis of Netflix's Security Framework for ‘Watch Instantly’ Service, Mar.-Apr. 2009, 18 pgs.
  • Server-Side Stream Repackaging (Streaming Video Technologies Panorama, Part 2), Jul. 2011, 15 pgs.
  • Text of ISO/IEC 23001-6: Dynamic adaptive streaming over HTTP (DASH), Oct. 2010, 71 pgs.
  • Universal Mobile Telecommunications System (UMTS), ETSI TS 126 233 V9.1.0 (Jun. 2011) 3GPP TS 26.233 version 9.1.0 Release 9, 18 pgs.
  • Universal Mobile Telecommunications Systems (UMTS); ETSI TS 126 244 V9.4.0 (May 2011) 3GPP TS 26.244 version 9.4.0 Release 9, 58 pgs.
  • “Apple HTTP Live Streaming specification”, Aug. 2017, 60 pgs.
  • “Data Encryption Decryption using AES Algorithm, Key and Salt with Java Cryptography Extension”, Available at https://www.digizol.com/2009/10/java-encrypt-decrypt-jce-salt.html, Oct. 200, 6 pgs.
  • “Delivering Live and On-Demand Smooth Streaming”, Microsoft Silverlight, 2009, 28 pgs.
  • “HTTP Based Adaptive Streaming over HSPA”, Apr. 2011, 73 pgs.
  • “HTTP Live Streaming”, Mar. 2011, 24 pgs.
  • “HTTP Live Streaming”, Sep. 2011, 33 pgs.
  • “Information Technology—Coding of Audio Visual Objects—Part 2: Visual”, International Standard, ISO/IEC 14496-2, Third Edition, Jun. 1, 2004, pp. 1-724. (presented in three parts).
  • “Java Cryptography Architecture API Specification & Reference”, Available at https://docs.oracle.com/javase/1.5.0/docs/guide/security/CryptoSpec.html, Jul. 25, 2004, 68 pgs.
  • “Java Cryptography Extension, javax.crypto.Cipher class”, Available at https://docs.oracle.com/javase/1.5.0/docs/api/javax/crypto/Cipher.html, 2004, 24 pgs.
  • “JCE Encryption—Data Encryption Standard (DES) Tutorial”, Available at https://mkyong.com/java/jce-encryption-data-encryption-standard-des-tutorial/, Feb. 25, 2009, 2 pgs.
  • “Live and On-Demand Video with Silverlight and IIS Smooth Streaming”, Microsoft Silverlight, Windows Server Internet Information Services 7.0, Feb. 2010, 15 pgs.
  • “Microsoft Smooth Streaming specification”, Jul. 22, 2013, 56 pgs.
  • “OpenDML AVI File Format Extensions Version 1.02”, OpenDMLAVI MJPEG File Format Subcommittee. Last revision: Feb. 28, 1996. Reformatting: Sep. 1997, 42 pgs.
  • “Single-Encode Streaming for Multiple Screen Delivery”, Telestream Wowza Media Systems, 2009, 6 pgs.
  • “The MPEG-DASH Standard for Multimedia Streaming Over the Internet”, IEEE MultiMedia, vol. 18, No. 4, 2011, 7 pgs.
  • “Windows Media Player 9”, Microsoft, Mar. 23, 2017, 3 pgs.
  • Abomhara et al., “Enhancing Selective Encryption for H.264/AVC Using Advanced Encryption Standard”, International Journal of computer Theory and Engineering, Apr. 2010, vol. 2, No. 2, pp. 223-229.
  • Alattar et al., A.M. “Improved selective encryption techniques for secure transmission of MPEG video bit-streams”, In Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348), vol. 4, IEEE, 1999, pp. 256-260.
  • Antoniou et al., “Adaptive Methods for the Transmission of Video Streams in Wireless Networks”, 2015, 50 pgs.
  • Apostolopoulos et al., “Secure Media Streaming and Secure Transcoding”, Multimedia Security Technologies for Digital Rights Management, 2006, 33 pgs.
  • Asai et al., “Essential Factors for Full-Interactive VOD Server: Video File System, Disk Scheduling, Network”, Proceedings of Globecom '95, Nov. 14-16, 1995, 6 pgs.
  • Beker et al., “Cipher Systems, The Protection of Communications”, 1982, 40 pgs.
  • Bocharov et al, “Portable Encoding of Audio-Video Objects, the Protected Interoperable File Format (PIFF)”, Microsoft Corporation, First Edition Sep. 8, 2009, 30 pgs.
  • Bulterman et al., “Synchronized Multimedia Integration Language (SMIL 3.0)”, W3C Recommendation, Dec. 1, 2008, https://www.w3.org/TR/2008/REC-SMIL3-20081201/, 321 pgs. (presented in five parts).
  • Cahill et al., “Locally Adaptive Deblocking Filter for Low Bit Rate Video”, Proceedings 2000 International Conference on Image Processing, Sep. 10-13, 2000, Vancouver, BC, Canada, 4 pgs.
  • Candelore, U.S. Appl. No. 60/372,901, filed Apr. 17, 2002.
  • Chaddha et al., “A Frame-work for Live Multicast of Video Streams over the Internet”, Proceedings of 3rd IEEE International Conference on Image Processing, Sep. 19, 1996, Lausanne, Switzerland, 4 pgs.
  • Cheng, “Partial Encryption for Image and Video Communication”, Thesis, Fall 1998, 95 pgs.
  • Cheng et al., “Partial encryption of compressed images and videos”, IEEE Transactions on Signal Processing, vol. 48, No. 8, Aug. 2000, 33 pgs.
  • Cheung et al., “On the Use of Destination Set Grouping to Improve Fairness in Multicast Video Distribution”, Proceedings of IEEE INFOCOM'96, Conference on Computer Communications, vol. 2, IEEE, 1996, 23 pgs.
  • International Standard, Information technology—Generic coding of moving pictures and associated audio information: Systems, ISO/IEC 13818-1:2000(E), Dec. 1, 2000 174 pgs.
  • Common Interface Specification for Conditional Access and other Digital Video Broadcasting Decoder Application, European Standard, EN 50221, Feb. 1997, 86 pgs.
  • “Server ‘Trick Play’ support for MPEG-2 Transport Stream Files”, www.live555.com/liveMedia/transport-stream-trick-play.html, 2006, Dec. 31, 2020, 1 pg.
  • “The LIVE555 Media Server”, www.live555.com/mediaServer/#about, 2006, printed Dec. 31, 2020, 3 pgs.
  • ADB, “ADB-3800W Datasheet”, 2007, 2 pgs.
  • Agi et al., “An Empirical Study of Secure MPEG Video Transmissions”, IEEE, Mar. 1996, 8 pgs., DOI: 10.1109/NDSS.1996.492420.
  • Conklin et al., “Video coding for streaming media delivery on the Internet”, IEEE Transactions on Circuits and Systems for Video Technology, Mar. 2001, vol. 11, No. 3, pp. 269-281.
  • Deshpande et al., “Scalable Streaming of JPEG2000 Images Using Hypertext Transfer Protocol”, Multimedia '01: Proceedings of the Ninth ACM International Conference on Multimedia, Oct. 2001, pp. 372-381. https://doi.org/10.1145/500141.500197.
  • ETSI, “Digital Video Broadcasting (DVB) Support for use of scrambling and Conditional Access (CA) within digital broadcasting systems”, Oct. 1996, 13 pgs.
  • Etsi, “Digital Video Broadcasting (DVB); Implementation guidelines for the use of Video and Audio Coding in Contribution and Primary Distribution Applications based on the MPEG-2 Transport Stream”, ETSI TS 102 154 V1.2.1, May 2004, 73 pgs.
  • Fahmi et al., “Proxy Servers for Scalable Interactive Video Support”, Computer, Sep. 2001, vol. 45, No. 9, pp. 54-60, https://doi.org/10.1109/2.947092.
  • Fielding et al., “Hypertext Transfer Protocol—HTTP1.1”, Network Working Group, RFC 2616, Jun. 1999, 114 pgs.
  • Fitzek et al., “A Prefetching Protocol for Continuous Media Streaming in Wireless Environments”, IEEE Journal on Selected Areas in Communications, Oct. 2001, vol. 19, No. 10, pp. 2015-2028, DOI:10.1109/49.957315.
  • Ho, “Digital Video Broadcasting Conditional Access Architecture”, Report prepared for CS265-Section 2, Fall 2002, Prof Stamp, 7 pgs.
  • ISMA, “ISMA Encryption and Authentication, Version 1.1, Area / Task Force: DRM”, Internet Streaming Media Alliance, Sep. 15, 2006, pp. 1-64.
  • ITU-T, “Series J: Cable Networks and Transmission of Television, Sound Programme and Other Multimedia Signals”, Technical method for ensuring privacy in long-distance international MPEG-2 television transmission conforming to ITU-T J.89, ITU-T Recommendation J.96, Mar. 2001, 34 pgs.
  • Kabir, “Scalable and Interactive Multimedia Streaming Over the Internet”, Thesis, 2005, 207 pgs.
  • Lian et al., “Selective Video Encryption Based on Advanced Video Coding”, PCM, Nov. 2005, Part II, LNCS 3768, pp. 281-290.
  • Lievaart, “Characteristics that differentiate CA Systems”, Irdeto access, Nov. 2001, 5 pgs.
  • Lloyd, “Supporting Trick Mode Playback Universally Across the Digital Television Industry”, Thesis, 2005, 111 pgs.
  • Macaulay et al., “Whitepaper—IP Streaming of MPEG-4: Native RTP vs MPEG-2 Transport Stream”, Envivio, Oct. 2005, 12 pgs.
  • Meyer et al., “Security mechanisms for Multimedia-Data with the Example MPEG-I-Video”, SECMPEG, 1992, 10 pgs.
  • Molavi et al., “A Security Study of Digital TV Distribution Systems”, Thesis, Jun. 2005, 112 pgs.
  • NCITS/ISO/IEC, “Information Technology—Generic Coding of Moving Pictures and Associated Audio Information: Video (Formerly ANSI/ISO/IEC 13818-2-2000)”, Second edition, Dec. 15, 2000, 220 pgs.
  • Nelson, “The Data Compression Book”, M&T Publishing, 1992, 533 pgs., (presented in two parts).
  • Qiao et al., “Comparison of MPEG Encryption Algorithms”, Comput. & Graphics, 1998, vol. 22, No. 4, pp. 437-448.
  • Senoh et al., “DRM Renewability & Interoperability”, IEEE Xplore, Conference: Consumer Communications and Networking Conference, 2004, Feb. 2004, pp. 424-429, DOI: 10.1109/CCNC.2004.1286899 Conference: Consumer Communications and Networking Conference, 2004. CCNC 2004. First IEEE.
  • Shojania et al., “Experiences with MPEG-4 Multimedia Streaming”, CiteSeer, Jan. 2001, 3 pgs., DOI: 10.1145/500141.500221.
  • Symes, “Video Compression Demystified”, McGraw-Hill, 2001, 353 pgs., (presented in two parts).
  • Taymans et al., “GStreamer Application Development Manual (1.6.0)”, 2007, 159 pgs.
  • Thomas et al., “A Novel Secure H.264 Transcoder Using Selective Encryption”, Proceedings in International Conference on Image Processing, Jan. 2007, vol. 4, pp. IV-85-IV-88, DOI: 10.1109/ICIP.2007.4379960.
  • Tosun et al., “Efficient multi-layer coding and encryption of M Peg video streams”, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia, Jul. 30-Aug. 2, 2000, pp. 119-122, DOI: 10.1109/ICME.2000.869559.
  • Um, “Selective Video Encryption of Distributed Video Coded Bitstreams and Multicast Security over Wireless Networks”, Thesis, Aug. 2006, 142 pgs.
  • Wang, “Lightweight Encryption in Multimedia”, Thesis, Jun. 2005, 184 pgs.
  • Wong, “Web Client Programming with Perl”, 1997, printed Jan. 8, 2021 from: https://www.oreilly.com/openbook-webclientch03.html, 31 pgs.
  • Wu, “A Fast MPEG Encryption Algorithm and Implementation of AES on CAM”, Thesis, Oct. 6, 2003, 91 pgs.
  • Yuksel, “Partial Encryption of Video for Communication and Storage”, Thesis, Sep. 2003, 78 pgs.
  • Order No. 40: Construing Certain Terms of the Asserted Claims of the Patent at Issue (Markman Claim Construction), Inv. No. 337-TA-1222, Mar. 12, 2021, 97 pgs.
Patent History
Patent number: 11050808
Type: Grant
Filed: Sep 9, 2019
Date of Patent: Jun 29, 2021
Patent Publication Number: 20200007601
Assignee: DIVX, LLC (San Diego, CA)
Inventor: Roland Osborne (San Francisco, CA)
Primary Examiner: Glenford J Madamba
Application Number: 16/565,375
Classifications
Current U.S. Class: Synchronization Of Presentation (715/203)
International Classification: G06F 16/71 (20190101); H04L 29/06 (20060101); G06F 16/738 (20190101); H04N 5/76 (20060101); H04N 5/783 (20060101); H04N 21/234 (20110101); H04N 21/44 (20110101); H04N 21/472 (20110101); H04N 21/6587 (20110101); H04N 7/173 (20110101);