System and method for organizing multimedia content
In one embodiment, a method of organizing multimedia content includes obtaining, by a server, a first video program. Also, the method includes selecting a first video, where the first video is a first subset of the first video program, and where the first video forms a first branch of the first video program and selecting a second video, where the second video is a second subset of the first video program. Additionally, the method includes selecting a first partial video, where the first partial video is a first subset of the first video and selecting a second partial video, where the second partial video is a second subset of the first video. The method also includes establishing a multi-dimensional index tree including a first video program level, a first video level, a first partial video level.
Latest Futurewei Technologies, Inc. Patents:
This application claims the benefit of U.S. Provisional Application No. 61/593,728 filed on Feb. 1, 2012, entitled “System and Method for Multimedia Content Storage in Media Cloud) which application is hereby incorporated herein by reference.
TECHNICAL FIELDThe present invention relates to a system and method for organizing media, and, in particular, to a system and method for organizing multimedia content.
BACKGROUNDVideo is an important part of people's digital life, with a strong impact on entertainment, enterprise solutions, education, and medical services. The amount of internet video is growing at an astonishing rate. In 2011, 82.5% of the U.S. internet audience viewed a video online, with an average 816.4 minutes of video viewing per user. By 2013, it is estimated that 90% of internet traffic will be video. Video may be continuously streamed to users in real time. For example, a video of a sporting event may be streamed live.
Cloud computing may be used to distribute video content over the internet. Cloud computing involves the use of hardware and software to deliver content as a service over a network, such as the internet. The content is stored on the cloud, not on user equipment. Cloud computing allows for the delivery of multimedia content with agility, reliability, scalability, security, a high performance, and a low cost. Cloud content may be available on demand.
SUMMARYAn embodiment method of organizing multimedia content includes obtaining, by a server, a first video program. Also, the method includes selecting a first video, where the first video is a first subset of the first video program, and where the first video forms a first branch of the first video program and selecting a second video, where the second video is a second subset of the first video program, and where the second video forms a second branch of the first video program. Additionally, the method includes selecting a first partial video, where the first partial video is a first subset of the first video, and where the first partial video forms a first branch of the first video and selecting a second partial video, where the second partial video is a second subset of the first video, and where the second partial video forms a second branch of the first video. The method also includes establishing a multi-dimensional index tree including a first video program level for the first video program, a first video level for the first video and the second video, a first partial video level for the first partial video and the second partial video.
An embodiment method of searching a tree index includes receiving, by a server, a first search phrase. Also, the method includes determining if the first search phrase matches metadata in a program index and indicating a program match if the first search phrase matches metadata in the program index. Additionally, the method includes determining if the first search phrase matches metadata in a video index, where the video index forms a branch of the program index and indicating a video match if the first search phrase matches metadata in the video index.
An embodiment method of adding an input video to a tree index includes receiving, by a server, the input video and extracting features of the input video. Also, the method includes comparing the extracted features of the input video with metadata in a metadata database and incorporating the input video into a matched video corresponding to metadata that matches the extracted features of the input video when the extracted features of the input video match metadata in the metadata database.
An embodiment server includes a processor and a computer readable storage medium storing programming for execution by the processor. The programming includes instructions to receive a first search phrase. Also, the programming includes instructions to determine if the first search phrase matches metadata in a program index and indicate a program match if the first search phrase matches metadata in the program index. Additionally, the programming includes instructions to determine if the first search phrase matches metadata in a video index, where the video index forms a branch of the program index and indicate a video match if the first search phrase matches metadata in the video index.
Another embodiment server includes a processor and a computer readable storage medium storing programming for execution by the processor. The programming includes instructions to obtain a first video program. Also, the programming includes instructions to select a first video, where the first video is a first subset of the first video program, and where the first video forms a first branch of the first video program and select a second video, where the second video is a second subset of the first video program, and where the second video forms a second branch of the first video program. Additionally, the programming includes instructions to select a first partial video, where the first partial video is a third subset of the first video, and where the first partial video forms a third branch of the first video and select a second partial video, where the second partial video is a fourth subset of the first video, and where the second partial video forms a fourth branch of the first video. The programming also includes instructions to establish a multi-dimensional index tree including a video program level for the first video program, a video level for the first video and the second video, a partial video index for the first partial video and the second partial video.
An additional embodiment server includes a processor and a computer readable storage medium storing programming for execution by the processor. The programming includes instructions to receive an input video and extract features of the input video. Also, the programming includes instructions to compare the extracted features of the input video with metadata in a metadata database and incorporate the input video into a matched video corresponding to metadata that matches the extracted features of the input video when the extracted features of the input video match metadata in the metadata database.
The foregoing has outlined rather broadly the features of an embodiment of the present invention in order that the detailed description of the invention that follows may be better understood. Additional features and advantages of embodiments of the invention will be described hereinafter, which form the subject of the claims of the invention. It should be appreciated by those skilled in the art that the conception and specific embodiments disclosed may be readily utilized as a basis for modifying or designing other structures or processes for carrying out the same purposes of the present invention. It should also be realized by those skilled in the art that such equivalent constructions do not depart from the spirit and scope of the invention as set forth in the appended claims.
For a more complete understanding of the present invention, and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawing, in which:
Corresponding numerals and symbols in the different figures generally refer to corresponding parts unless otherwise indicated. The figures are drawn to clearly illustrate the relevant aspects of the embodiments and are not necessarily drawn to scale.
DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTSIt should be understood at the outset that although an illustrative implementation of one or more embodiments are provided below, the disclosed systems and/or methods may be implemented using any number of techniques, whether currently known or in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.
Multimedia content may be organized in a particular configuration. For example, an embodiment method organizes content in a tree index.
Next, in step 226, the system selects a video that is a subset of the video program. A video program may have multiple videos with different quality levels. For example, one high density (HD) video, one standard density (SD) video, and one video for smart phones. Then, in step 228, attributes of the video, such as video features, audio features, and textual features are extracted and added to a video level index. Next, in step 230, a partial video, which is a subset of the video, is selected. A partial video may be a scene where the video is a HD movie. A partial video may be from about a second to several minutes. After selecting a partial video, in step 232, partial video attributes, such as video features, audio features, and textual features, are extracted and added to a partial video index. In step 234, a segment or key frame, which is a subset of the partial video, is selected. A key frame is one event, for example a touchdown. In an example, the key frame is one frame of the partial video that is most representative. A key frame may be from about a single frame to about several seconds. A segment is a video segment that is a set period of time, such as two seconds. In an example, a segment is an arbitrary two second clip of the partial video. The system may index both segments and key frames, just segments, just key frames, or neither segments nor key frames. Next, in step 236, the system extracts features of the key frame or segment and adds the extracted features to a key frame segment level. The features may be video features, audio features, and/or textual features.
Then, in step 238, the system determines if it will search for more key frames or segments. For example, if the system is determining segments, it may index every two second segment in the partial video. In an example, only one key frame is extracted per partial video. If there are more key frames or segments, the system goes to step 234 to select another segment or key frame. However, if there will not be another key frame or segment in this partial video, the system goes to step 240, where it decides if there will be another partial video in the video. If there will be another partial videos in this video, the system goes to step 230, to select another partial video. However, if there will not be another partial video in this video, the system goes to step 242, where it determines if there will be more videos in this program. For adaptive streaming, a video program may have multiple different quality levels of videos. If there will be another video in the video program, the system goes to step 226, and it selects another video. However, if there will not be another video in this video program, the system goes to step 244, and finishes adding the video program to the tree index.
If no match is found, the input video is sent to the key frame extraction engine 182, of media processing engine 178, where a plurality of key frames of the input video is extracted. Then, in key frame fingerprint generation engine 184 of media processing engine 178, a plurality of key frame fingerprints is generated, one for each key frame. In an example, video segmentation may be used to extract the head stream and the tail stream of the input video. Then, the key frame may be generated from the tail stream and the head stream. Next, in comparison block 190, the plurality of key frame fingerprints is compared with key frame fingerprints stored in metadata database 174. If there is no match, a new index item is added to the index table. Once a match is found, indexing engine 186 generates additional metadata of the input video. Next, the input video may be stored in a video database (not pictured). In an example, the video database resides in the cache. In an embodiment, the key frame extraction utilizes modeling and fast video segmentation algorithms. Then, a tree index of the input video and its metadata are generated by indexing engine 186 if no match of the input video's metadata is found in the metadata database. The indexes may include a program level, video level, partial video level, and key frame segment level metadata. Global textual information, such as the title, header, and format, and local textual information, such as subtitles, visual information such as visual fingerprints and visual content information, and audio information such as audio fingerprinting and audio content are organized via a tree index.
Subsets of a video program are organized into a program family tree. Multiple versions of the same video or partial video of the same video are organized into the index tree with links to all versions and parts available for user access. A new video is processed by a processing engine. If the family of the new video, such as its ascendants, descendants, or siblings, is found in tree index 200, the new video is added to the family tree. If no ascendants, descendants, or siblings are found, a new family tree is created with the new video as the video program.
When a query is received by a portal of an IP video system, the query processing engine parses the query, generates a query search stream, and looks up in a metadata database to find a match. The family tree of each matching result is returned with the matching result. In an embodiment, a weighting function considers services rules and regulation and facilitates the control of searches and query retrieval.
The bus may be one or more of any type of several bus architectures including a memory bus or memory controller, a peripheral bus, video bus, or the like. CPU 274 may comprise any type of electronic data processor. Memory 276 may comprise any type of system memory such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), a combination thereof, or the like. In an embodiment, the memory may include ROM for use at boot-up, and DRAM for program and data storage for use while executing programs.
Mass storage device 278 may comprise any type of storage device configured to store data, programs, and other information and to make the data, programs, and other information accessible via the bus. Mass storage device 278 may comprise, for example, one or more of a solid state drive, hard disk drive, a magnetic disk drive, an optical disk drive, or the like.
Video adaptor 280 and I/O interface 288 provide interfaces to couple external input and output devices to the processing unit. As illustrated, examples of input and output devices include the display coupled to the video adapter and the mouse/keyboard/printer coupled to the I/O interface. Other devices may be coupled to the processing unit, and additional or fewer interface cards may be utilized. For example, a serial interface card (not pictured) may be used to provide a serial interface for a printer.
The processing unit also includes one or more network interface 284, which may comprise wired links, such as an Ethernet cable or the like, and/or wireless links to access nodes or different networks. Network interface 284 allows the processing unit to communicate with remote units via the networks. For example, the network interface may provide wireless communication via one or more transmitters/transmit antennas and one or more receivers/receive antennas. In an embodiment, the processing unit is coupled to a local-area network or a wide-area network for data processing and communications with remote devices, such as other processing units, the Internet, remote storage facilities, or the like.
Advantages of an embodiment include a lightweight processing scheme for a scalable and flexible video organizer in a media cloud. An embodiment employs multiple modality-based video processing and organization schemes to offer quality of experience (QoE) enhanced video indexing, browsing, and searching. An example embodiment is implemented in IP video and media cloud applications, such as video management systems and video distribution systems. An embodiment includes multiple levels of indexes providing convenience and search capability. In an example, a tree based indexing scheme is scalable and easy to use. An embodiment includes better service control with multimodality based indexing.
While several embodiments have been provided in the present disclosure, it should be understood that the disclosed systems and methods might be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the intention is not to be limited to the details given herein. For example, the various elements or components may be combined or integrated in another system or certain features may be omitted, or not implemented.
In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items shown or discussed as coupled or directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediate component whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and could be made without departing from the spirit and scope disclosed herein.
Claims
1. A method of organizing multimedia content, the method comprising:
- receiving, by a server, a first video program;
- selecting, by the server, a first video, wherein the first video is a first portion of the first video program, and wherein the first video forms a first branch of the first video program;
- selecting, by the server, a second video, wherein the second video is a second portion of the first video program, and wherein the second video forms a second branch of the first video program;
- selecting, by the server, a first partial video, wherein the first partial video is a first portion in time of the first video, and wherein the first partial video forms a first branch of the first video;
- selecting, by the server, a second partial video, wherein the second partial video is a second portion in time of the first video, and wherein the second partial video forms a second branch of the first video; and
- storing, by the server in a metadata database, a first multi-dimensional tree index including a first video program level comprising an electronic link to the first video program, a first video level comprising an electronic link to the first video and an electronic link to the second video, and a first partial video level comprising an electronic link the first partial video and an electronic link to the second partial video, wherein the first video program level of the first multi-dimensional tree index and the first video level of the first multi-dimensional tree index are related to the first video program.
2. The method of claim 1, further comprising determining a video program index in accordance with the first video program, wherein the first multi-dimensional tree index includes the video program index at the first video program level.
3. The method of claim 2, wherein determining the video program index comprises extracting video features from the first video program.
4. The method of claim 2, wherein determining the video program index comprises extracting audio features from the first video program.
5. The method of claim 2, wherein determining the video program index comprises extracting textual features from the first video program.
6. The method of claim 2, further comprising determining a video index in accordance with the first video and the second video, wherein the first multi-dimensional tree index further includes the video index at the first video level.
7. The method of claim 6, further comprising determining a partial video index in accordance with the first partial video and the second partial video, wherein the first multi-dimensional tree index further includes the partial video index at the first partial video level.
8. The method of claim 1, wherein the first video program comprises a live streamed video.
9. The method of claim 1, further comprising selecting a first key frame, wherein the first key frame is a first portion of the first partial video, wherein the first key frame forms a first branch of the first partial video, and wherein the first multi-dimensional tree index further includes a key frame level for the first key frame.
10. The method of claim 9, further comprising determining a key frame index in accordance with the first key frame, and wherein the first multi-dimensional tree index further includes the key frame index at the key frame level.
11. The method of claim 1, further comprising:
- determining a first segment, wherein the first segment is a first portion of the first partial video, and wherein the first segment forms a first branch of the first partial video; and
- determining a second segment, wherein the second segment is a second portion of the first partial video, wherein the second segment forms a second branch of the first partial video, and wherein the first multi-dimensional tree index further comprises a first segment level for the first segment and the second segment.
12. The method of claim 11, further comprising determining a segment index in accordance with the first segment and the second segment, wherein the first multi-dimensional tree index further includes the segment index at the first segment level.
13. The method of claim 1, further comprising:
- receiving, by the server, a second video program;
- determining a third video, wherein the third video is a first portion of the second video program, and wherein the third video forms a first branch of the second video program;
- determining a fourth video, wherein the fourth video is a second portion of the second video program, and wherein the fourth video forms a second branch of the second video program;
- determining a third partial video, wherein the third partial video is a first portion of the third video, and wherein the third partial video forms a first branch of the third video;
- determining a fourth partial video, wherein the fourth partial video is a second portion of the third video, and wherein the fourth partial video forms a second branch of the third video;
- storing a second tree index comprising a first indicator of the third video, a second indicator of the fourth video, a third indicator of the third partial video, and a fourth indicator of the fourth partial video; and
- determining a second multi-dimensional tree index comprising a second video program level for the second video program, a second video level for the third video and the fourth video, and a second partial video level for the third partial video and the fourth partial video.
14. A server comprising:
- a processor;
- a metadata database; and
- a non-transitory computer readable storage medium storing programming for execution by the processor, the programming including instructions to: receive a first video program, select a first video, wherein the first video is a first portion of the first video program, and wherein the first video forms a first branch of the first video program, select a second video, wherein the second video is a second portion of the first video program, and wherein the second video forms a second branch of the first video program, select a first partial video, wherein the first partial video is a first portion in time of the first video, and wherein the first partial video forms a first branch of the first video, select a second partial video, wherein the second partial video is a second portion in time of the first video, and wherein the second partial video forms a second branch of the first video, and store, in the metadata database, a multi-dimensional tree index including a video program level comprising an electronic link to the first video program, a video level comprising an electronic link to the first video and an electronic link to the second video, and a partial video index comprising an electronic link to the first partial video and an electronic link to the second partial video, wherein the video program level of the multi-dimensional tree index and the video level of the multi-dimensional tree index are related to the first video program.
15. The method of claim 1, further comprising:
- receiving, by the server, a first search phrase;
- determining whether the first search phrase matches metadata in a program index of the first multi-dimensional tree index;
- indicating a program match when the first search phrase matches metadata in the program index;
- determining whether the first search phrase matches metadata in a video index of the first multi-dimensional tree index, wherein the video index forms a branch of the program index; and
- indicating a video match when the first search phrase matches metadata in the video index.
16. The method of claim 15, further comprising:
- determining whether the first search phrase matches metadata in a partial video index of the first multi-dimensional tree index, wherein the partial video index forms a branch of the video index; and
- indicating a partial video match whether the first search phrase matches metadata in the partial video index.
17. The method of claim 15, further comprising determining whether the first search phrase matches metadata in a segment key frame index of the first multi-dimensional tree index.
18. The method of claim 1, further comprising:
- receiving, by the server, an input video;
- extracting features of the input video to produce extracted features;
- comparing the extracted features of the input video with metadata in the metadata database of the first multi-dimensional tree index; and
- incorporating the input video into a matched video corresponding to metadata that matches the extracted features of the input video when the extracted features of the input video match metadata in the metadata database.
19. The method of claim 18, wherein incorporating the input video into the matched video comprises:
- creating an indicator denoting that the input video matches the matched video;
- extracting additional metadata from the input video; and
- incorporating the additional metadata with the metadata in the metadata database that matches the extracted features.
20. The method of claim 18, further comprising:
- extracting a plurality of key frames from the input video when the extracted features of the input video do not match metadata in the metadata database; and
- generating a plurality of key frame fingerprints in accordance with the plurality of key frames.
21. The method of claim 20, wherein generating the plurality of key frame fingerprints comprises:
- generating a plurality of head streams in accordance with the plurality of key frames; and
- generating a plurality of tail streams in accordance with the plurality of key frames.
22. The method of claim 21, further comprising:
- comparing the plurality of key frame fingerprints with key frame fingerprints in the metadata database;
- generating additional metadata from the input video when one of the plurality of key frame fingerprints matches one of the key frame fingerprints in the metadata database; and
- storing the input video.
23. The server of claim 14, wherein the programming further comprises instructions to:
- receive a first search phrase;
- determine whether the first search phrase matches metadata in a program index of the multi-dimensional tree index;
- indicate a program match when the first search phrase matches metadata in the program index;
- determine whether the first search phrase matches metadata in a video index of the multi-dimensional tree index, wherein the video index forms a branch of the program index; and
- indicate a video match when the first search phrase matches metadata in the video index.
24. The server of claim 14, wherein the programming further comprises instructions to:
- receive an input video;
- extract features of the input video to produce extracted features;
- compare the extracted features of the input video with metadata in the metadata database of the multi-dimensional tree index; and
- incorporate the input video into a matched video corresponding to metadata that matches the extracted features of the input video when the extracted features of the input video match metadata in the metadata database.
5835667 | November 10, 1998 | Wactlar et al. |
6122628 | September 19, 2000 | Castelli et al. |
6278446 | August 21, 2001 | Liou et al. |
6859455 | February 22, 2005 | Yazdani et al. |
20080052739 | February 28, 2008 | Logan |
20080133464 | June 5, 2008 | Shin |
20090132918 | May 21, 2009 | Deyo et al. |
20120011544 | January 12, 2012 | Vaysman et al. |
20120163770 | June 28, 2012 | Kaiser et al. |
- Notification of Transmittal of the International Search Report and the Written Opinion of the International Searching Authority, or the Declaration for Application No. PCT/US2013/024507, Mailed Jun. 10, 2013, 11 pages.
- Das, M., et al., “A New Hybrid Approach to Video Organization for Content-Based Indexing,” IEEE ICMCS, Jun. 28, 1998 to Jul. 1, 1998, 10 pages.
- Kuchinsky, A., et al., “FotoFile: A Consumer Multimedia Organization and Retrieval System,” Proceedings of the SIGCHI conference on Human factors in computing systems, May 1999, 8 pages.
Type: Grant
Filed: Feb 1, 2013
Date of Patent: Aug 15, 2017
Patent Publication Number: 20130198783
Assignee: Futurewei Technologies, Inc. (Plano, TX)
Inventors: Yue Chen (Fremont, CA), Hong Heather Yu (West Windsor, NJ), Hongbing Li (Skillman, NJ), Jianbin Dai (Sunnyvale, CA), Zhenhua Liu (Beijing)
Primary Examiner: Pankaj Kumar
Assistant Examiner: Franklin Andramuno
Application Number: 13/757,614
International Classification: H04N 5/445 (20110101); H04N 21/27 (20110101); H04N 21/8541 (20110101); H04N 21/845 (20110101); H04N 21/235 (20110101); G06F 17/30 (20060101);