System and method for exploiting scene graph information in construction of an encoded video sequence

A system method and computer program product for creating a composited video frame sequence for an application. A current scene graph state for the application is compared to a previous scene graph state wherein each scene graph state includes a plurality of hierarchical nodes that represent one or more objects at each node. A video construction engine determines if one or more objects have moved based upon a comparison of the scene graph states. If one or more objects have moved based upon the scene graph comparison, motion information about the objects is determined and the motion information is forwarded to a stitcher module. The motion information is used to define motion vectors for use by the stitcher module in construction of the composited video frame sequence.

Skip to: Description  ·  Claims  ·  References Cited  · Patent History  ·  Patent History
Description
TECHNICAL FIELD

The present invention relates to the creation of an encoded video sequence, and more particularly to using scene graph information for encoding the video sequence.

BACKGROUND ART

It is known in the prior art to encode and transmit multimedia content for distribution within a network. For example, video content may be encoded as MPEG video wherein pixel domain data is converted into a frequency domain representation, quantized and entropy encoded and placed into an MPEG stream format. The MPEG stream can then be transmitted to a client device and decoded and returned to the spatial/pixel domain for display on a display device.

The encoding of the video may be spatial, temporal or a combination of both. Spatial encoding generally refers to the process of intraframe encoding wherein spatial redundancy (information) is exploited to reduce the number of bits that represent a spatial location. Spatial data is converted into a frequency domain over a small region. In general for small regions it is expected that the data will not drastically change and therefore there much of the information will be stored at DC and low frequency components with the higher frequency components being at or near zero. Thus, the lack of high frequency information of small area is used to reduce the representative data size. Data may also be compressed using temporal redundancy. One method for exploiting temporal redundancy is through the calculation of motion vectors. Motion vectors establish how objects or pixels move between frames of video. Thus, a ball may move between a first frame and a second frame by a number of pixels in a given direction. Thus, once a motion vector is calculated, the information about the spatial relocation of the ball information from the first frame to the second frame can be used to reduce the amount of information that is used to represent the motion in an encoded video sequence.

Motion vector calculation is perhaps the most time consuming and processor intensive step in compressing video content. Typically, a motion vector search algorithm is employed to attempt to match elements within the video frames and to define motion vectors that point to the new location that objects or portions of objects. This motion vector search compares macroblocks between video frames to find a best match. If a best match is identified that is above a certain threshold, the process assumes that a motion vector match has been found and uses this information in the compression process. It would be beneficial if a mechanism existed in which motion vectors could be more easily calculated.

SUMMARY OF THE EMBODIMENTS

In a first embodiment of the invention there is provided a method for creating a composited video frame sequence for an application wherein the video frame sequence is encoded according to a predetermined specification, such as MPEG-2, H264 or other block based encoding protocol or variant thereof. A current scene graph state for the application is compared to a previous scene graph state wherein each scene graph state includes a plurality of hierarchical nodes that represent one or more objects at each node. A video construction engine determines if one or more objects have moved based upon a comparison of the scene graph states. If one or more objects have moved based upon the scene graph state comparison, motion information about the objects is determined and the motion information is forwarded to a stitcher module. The motion information is used to define motion vectors for use by the stitcher module in construction of the composited video frame sequence. In certain embodiments of the invention, the motion vectors are two-dimensional motion vectors.

In order to determine the motion information, the current scene graph state and the previous scene graph state may be converted from a three dimensional representation into a two dimensional representation. The three dimensional representation may be for a worldview of the objects to be rendered and displayed on a display device. The two dimensional representation may be a screen view for displaying the objects on a display device. In addition to the motion information, in general there will be residual graphical information because the edges of moving objects generally do not map exactly on macroblock boundaries or objects are partially translucent, overlay or have quantization effects etc.

Embodiments of the invention may construct an MPEG encoded video sequence using the motion information including the corresponding motion vectors and residual graphical information that can be encoded. The scene graph states (previous and current) may result as the output of an application engine such as an application execution engine. The application execution engine may be a web browser, a script interpreter, operating system or other computer-based environment that is accessed during operation of the application. The scene graph state may be transcoded using a library such as an API. One example of a library is an OpenGL API (application programming interface). The transcoded scene graph state results in a graphical processing schema that accesses the library.

The current scene graph state includes a plurality objects having associated parameters. Some examples of parameters are the location of objects to be rendered, lighting effects, textures, and other graphical characteristics that may be used in rendering the object(s). A hash may be created for objects within a scene. The hash may be compared to a table of hashes that represent objects from previous scenes. If the current hash matches a hash within the table of hashes, MPEG encoded elements for the identified object are retrieved. The MPEG encoded elements can then be sent to a stitcher that can stitch together the MPEG encoded elements to form one or more MPEG encoded video frames in a series of MPEG encoded video frames.

In order to create the hash for the objects, the scene graph state is converted to a 2D or display representation. It is then determined which non-overlapping rectangles of the display represent state changes of the scene graph state. A hash is created for each rectangle i.e. object; the previous and current state of the objects within these rectangles is hashed. These hashes are compared to hashes available in the table of hashes.

If the current hash does not match a hash in the table and no motion information can be determined by the scene graph state comparison for an object, the spatial data from the hashing process where the object is converted from a three dimensional representation to a two dimension screen representation is provided to an encoder wherein the encoder compresses the data using at least spatial techniques to produce one or more encoded elements. The encoder may encode according to a predetermined protocol such as MPEG, H264 or another block based encoding protocol. The encoded elements are passed to a stitching module. The stitching module forms an encoded MPEG frame from the encoded elements where the encoded MPEG frame is part of an MPEG video sequence.

The methodology may be embodied as a computer program product where the computer program product includes a non-transitory computer readable medium having computer code thereon for creating an encoded video sequence. The above-described method may be embodied as a system that includes one or more processors that perform specified functions in the creation of the encoded video sequence.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing features of embodiments will be more readily understood by reference to the following detailed description, taken with reference to the accompanying drawings, in which:

FIG. 1 shows a detailed embodiment showing components that are used in processing application environment data and constructing an encoded video sequence from the data;

FIG. 2 shows a flow chart for implementing the functionality of relevant components of an embodiment of the invention;

FIG. 3 shows an environment for implementing the present invention;

FIG. 4 shows an exemplary screen shot of an application;

FIG. 5 shows a representative DOM tree for the application of FIG. 4;

FIG. 6 shows an exemplary scene graph model of the image of FIG. 4.

FIG. 7 shows a scene graph state with associated screen position information;

FIG. 8 shows a previous scene graph state and a current scene graph state

FIG. 9 shows a motion field between a first scene graph state and a second scene graph state; and

FIG. 10 shows a motion field for the rotation of each macroblock of an image.

 DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS Definitions

As used in this description and the accompanying claims, the following terms shall have the meanings indicated, unless the context otherwise requires:

The term “application” refers to an executable program, or a listing of instructions for execution, that defines a graphical user interface (“GUI”) for display on a display device. An application may be written in a declarative language such as HTML or CSS, a procedural language such as C, JavaScript, or Perl, any other computer programming language, or a combination of languages.

“Application execution environment” is an environment that receives in an application including all of its components and manages the components and execution of the components to define a graphical layout and manages the interactions with the graphical layout. For example, Trident, WebKit, and Gecko are software layout engines that convert web pages into a collection of graphical objects (text strings, images, and so on) arranged, according to various instructions, within a page display area of a web browser. The instructions may be static, as in the case of parts of HTML, or dynamic, as in the case of JavaScript or other scripting languages, and the instructions may change as a function of user input. Trident is developed by Microsoft Corporation and used by the Internet Explorer web browser; WebKit is developed by a consortium including Apple, Nokia, Google and others, and is used by the Google Chrome and Apple Safari web browsers; Gecko is developed by the Mozilla Foundation, and is used by the Firefox web browser. Operating systems such as Google's Android and Apple's iOS may be considered application execution environment because these operating systems can execute applications. The output of an application execution environment is a screen state (either absolute or relative to a previous screen state). The screen state may be presented as a scene graph state.

“Rendering Engine” transforms a model of an image to actual data that can generate the image on a display device. The model of the image may contain two-dimensional or three-dimensional data as would be represented in a world space and the rendering engine takes the data and transforms the data into a screen-space representation wherein the data may be represented as pixels.

“Encoding Engine” transforms data representative on an image or a portion of an image into one or more encoded elements. An encoding engine may take pixel data and perform spatial/frequency and temporal transformations to represent the data in an encoded format.

“Stitching Engine” receives as input one or more encoded elements (e.g. MPEG encoded elements) along with layout information and then constructs complete video frames for a video sequence (e.g. MPEG video frames for an MPEG elementary stream).

“Scene graph” is a hierarchical model of an image (i.e. a scene) generated by an application including nodes and the attributes for the nodes;

“Scene graph state” is a scene graph a particular moment in time including nodes and the attributes for the nodes at that time.

“DOM” (document object model) is a convention for representing and interacting with objects in markup languages such as HTML and XML documents.

“DOM tree” is a representation of a DOM (document object model) for a document (e.g. an HTML file) having nodes wherein the topmost node is the document object.

“CSS” (cascading style sheets) provide the graphical layout information for a document (e.g. an HTML document) and how each object or class of objects should be represented graphically. The combination of a DOM object and the corresponding CSS files (i.e. layout) is referred to as a rendering object.

“Render layer” is a representation of the rendering objects according to one or more nodes within a scene graph state, which may be a modified scene graph state. For example, a group of objects that have a geographical relationship such as an absolute or a relative position to each other may form a layer. An object may be considered to be a separate render layer if the object is transparent, has an alpha mask or has a reflection, for example. A render layer may be defined by a screen area, such as a screen area that can be scrolled. A render layer may be designated for an area that may have an overlay (e.g. a pop-up). A render layer could be defined for a portion of a screen area if that area is to have an applied graphical filter such as a blur, color manipulation or shadowing. A layer could be defined by a screen area that has associated video content. Thus, a render layer may be a layer within a scene graph state or a modification of a scene graph state layer in which objects are grouped according to a common characteristic

“MPEG element” is one or more MPEG-encoded macroblocks, as disclosed in U.S. patent application Ser. No. 12/443,571, filed Oct. 1, 2007, the contents of which are incorporated by reference in their entirety. An MPEG element may be intra-encoded (within a frame) or inter-encoded (between frames).

Embodiments of the present invention provide for the extraction of spatial information as well as other graphical information from an application execution environment by using software integration points that are (for example) intended for communication between the application execution environment and Graphical Processing Unit (GPU) driver software. This spatial information can then be used for the creation of motion vectors for encoding of graphical content in a frequency-based encoding format, such as MPEG, AVS, VC-1, H.264 and other block-based encoding formats and variants that employ motion vectors.

Embodiments of the invention use the motion information exposed by an Application Execution Environment's GPU interface (or another suitable interface that allows access to the scene graph state) to obtain spatial and temporal information of the screen objects to be rendered, and to use that information to more efficiently encode the screen objects into a stream of MPEG frames.

In order to determine the motion information, the application execution Environment may access Z-ordering information from a scene graph for the rendering of objects. For example, the application execution environment can separate a background layer from a foreground image layer and the scene graph state may specify objects that are partially translucent. This information can be used to determine what information will be rendered from a 3-dimensional world view in a 2-dimensional screen view. Once the visible elements are determined, motion information can be determined and the motion information can be converted into motion vectors. Multiple motion vectors may be present for a particular screen area. For example, if two different layers (on different Z indices) are moving in different directions, the area would have different associated motion vectors. The encoder will determine a dominant vector given its knowledge on what is being rendered, including translucency, surface area of the moving object, texture properties (i.e. is it a solid or a pattern) etc.

FIG. 1 shows a detailed embodiment showing components that are used in processing application environment data and constructing an encoded video sequence from the data. The application environment data provides information about visual content to be rendered on a display device of a client. The data from an application execution environment 110 may be processed through one of a plurality of possible paths. The first path is a prior art path wherein the data from the application execution environment 110, which may be OpenGL library function calls is passed to a hardware-based graphics accelerator 120 and presented on a display 130. In an alternative path, the data from the application execution environment 110 is passed to a video construction engine 170. The video construction engine 170 exploits information within the data from the application execution engine to improve the encoding process and reduce the number of calculations that need to be performed. This path will be explained in greater detail below with respect to embodiments of the invention.

FIG. 1 is now explained in more detail. An application is constructed in an application editor 100. The application editor 100 may be an integrated development environment (IDE) or a text editor for example. The output of the application editor may include one or more sections. The application may be composed of one or more of the following: HTML (hypertext markup language) data, CSS (cascading style sheets) data, script(s) from various scripting languages such as JavaScript and Perl, program code, such as, JAVA for execution in an application execution environment and/or executable programs (*.exe). The components of the application may then be executed in an application execution environment 110 in response to a request for the application by a client device operating remotely from the application execution environment. An application execution environment receives in the application including its various components and creates an output file that can be used for display on a display device of the client. For example, the application execution environment may create a program referencing a number of OpenGL library functions/objects. OpenGL is a specification that describes an abstract API for drawing 2D and 3D graphics and is known to one of ordinary skill in the art.

As shown, the Application Execution Engine 110 may produce an output for graphical processing. Examples of application execution environments include both computer software and hardware and combinations thereof for executing the application. Applications can be written for certain application execution environments including WebKit, JAVA compilers, script interpreters (Perl etc.) and various operating systems including iOS and Android OS for example.

The video construction engine 170 takes advantage of the data that it receives from the application execution environment in order to exploit redundancies in requests for the presentation of information within user sessions and between user sessions as well as determining motion changes of objects from a previous video frame or scene graph state to a current frame or scene graph state. The present system may be used in a networked environment wherein multiple user sessions are operational simultaneously wherein requested applications may be used by multiple users simultaneously.

The video construction engine 170 may receive in OpenGL data and can construct a scene graph from the OpenGL data. The video construction engine 170 can then compare the current scene graph state to one or more previous scene graph states to determine if motion occurs between objects within the scene. If motion occurs between the objects, this motion can be translated into a motion vector and this motion vector information can be passed to an encoding module 150. Thus, the encoding module 150 need not perform a motion vector search and can add the motion vectors into the video frame format (e.g. MPEG video frame format). The MPEG elements can be constructed that are encoded MPEG macroblocks that are inter-frame encoded. These macroblocks are passed to the stitching module 160 that receives stitching information about the video frame layout and stitches together encoded MPEG elements to form complete MPEG encoded video frames in accordance with the scene graph. Either simultaneously or in sequence, the MPEG video construction engine may hash the parameters for objects within the scene graph according to a known algorithm. The construction engine 170 will compare the hash value to hash values of objects from previous scene graphs and if there is a match within the table of hashes, the construction engine 170 will locate MPEG encoded macroblocks (MPEG elements) that are stored in memory and are related to the hash. These MPEG elements can be passed directly to the stitching engine 160 wherein the MPEG elements are stitched together to form complete MPEG encoded video frames. Thus, the output of the stitching module 160 is a sequence of encoded video frames that contain both intra-frame encoded macroblocks and inter-frame encoded macroblocks. Additionally, the video construction engine 170 outputs pixel based information to the encoder. This pixel-based information may be encoded using spatial based encoding algorithms including the standard MPEG DCT processes. This pixel-based information occurs as a result of changes in the scene (visual display) in which objects represented by rectangles are altered. The encoded macroblocks can then be passed to the stitcher. The processes of the video construction engine 170 will be explained in further detail with respect to the remaining figures.

FIG. 2 shows a flow chart for implementing the functionality of relevant components of an embodiment of the invention. A user of the system at a client device interacts with the application through the application execution engine. The user makes a request for content through a key press or other input that generates a control signal that is transmitted from the client device to the application execution engine that indicates that there should be a screen update of one or more screen elements (e.g. rectangles). Thus, the rectangles to be updated can be defined as a dirty rectangle that will need either to be retrieved from memory if the dirty rectangle has previously been rendered and encoded or provided to an encoder. The encoder may receive motion vector information, which will avoid motion vector calculations and the encoder may receive spatial data for dirty rectangles, which need to be spatially encoded.

The application execution engine may be proximate to the client device, operational on the client device, or may be remote from the client device, such as in a networked client/server environment. The control signal for the dirty rectangle causes the application execution engine to generate a scene graph having a scene graph state that reflects the changes to the screen (e.g. dirty rectangles of the screen display). For example, the application execution environment may be a web browser operating within an operating system. The web browser represents a page of content in a structured hierarchical format such as a DOM and corresponding DOM tree. Associated with the DOM tree is a CSS that specifies where and how each object is to be graphically rendered on a display device. The web browser creates an output that can be used by a graphics engine. The output that is produced is the scene graph state which may have one or more nodes and objects associated with the nodes forming a layer (i.e. a render layer) 200. As requests occur from a client device for updates or updates are automatically generated as in a script, a new or current scene graph state is generated. Thus, the current scene graph state represents a change in the anticipated output video that will be rendered on a display device. An exemplary scene graph state is shown in FIG. 6 described below.

Once the current scene graph state is received by the video construction engine 200, the scene graph state can be compared with a previous scene graph state 210. The comparison of scene graph states can be performed hierarchically by layer and by object. For each object associated with a node differences in the positions of objects from the scene graph states can be identified as well as differences in characteristics, such as translucence and lighting.

For example, in a simple embodiment, a circle may be translated by a definable distance between the current scene graph state and a previous scene graph state. The system queries whether one or more objects within the scene graph state have moved. If one or more objects have been identified as moving between scene graph states information about the motion translation are determined 220. This information may require the transformation of position data from a three dimensional world coordinate view to a two-dimensional screen view so that pixel level motion (two dimensional motion vectors) can be determined. This motion information can then be passed on to an encoder in the form of a motion vector 230. Thus, the motion vector information can be used by the encoder which to create interframe encoded video frames. For example, the video frames may be P or B frame MPEG encoded frames.

In addition to objects moving, scene elements may also change. Thus, a two dimensional representation of information to be displayed on a screen can be ascertained from the three-dimensional scene graph state data. Rectangles can be defined as dirty rectangles, which identify data on the screen that has changed 240. These rectangles can by hashed according to a known formula that will take into account properties of the rectangles 250. The hash value can then be compared to a listing of hash values associated with rectangles that were updated from previous scene graph states 260. The list of hash values may be for the current user session or for other user sessions. Thus, if a request for a change in the content being displayed in an application is received from multiple parties, the redundancy in information being requested can be exploited and processing resources conserved. More specifically, if the hash matches a hash within the searchable memory, encoded graphical data (e.g. either a portion of an entire video frame of encoded data or an entire frame of encoded data) that is linked to the hash value in the searchable memory is retrieved and the data can be combined with other encoded video frames 270.

Additionally, if a rectangle is identified as being dirty and a hash is not identified, the spatial information for that rectangle can be passed to the encoder and the MPEG encoder will spatially encode the data for the rectangle. As used herein, the term content, may refer to a dirty rectangle or an object from a scene graph state.

FIG. 3 shows an embodiment of the present invention showing the data flow between an application execution environment 300 and the data flow internal to the video construction engine 310. As previously indicated an application execution environment 300 receives as input an application and the application execution environment 300 executes the application and receives as input user requests for changes to the graphical content that is displayed on a display device associated with the user.

The application execution environment 300 creates a current scene graph 320. The current scene graph may be translated using a library of functions, such as the OpenGL library 330. The resulting OpenGL scene graph state 340 is passed to the video construction engine 310. The OpenGL scene graph state 340 for the current scene graph is compared to a previous scene graph state 350 in a comparison module 360. This may require the calculation and analysis of two-dimensional projections of three-dimension information that are present within the scene graph state. Such transformation are known by one of ordinary skill in the art. It should be recognized that OpenGL is used herein for convenience and that only the creation of a scene graph state is essential for the present invention. Thus, the scene graph state need not be converted into OpenGL before a scene graph state comparison is performed.

Differences between the scene graphs are noted and dirty rectangles can be identified 370. A dirty rectangle 370 represents a change to an identifiable portion of the display (e.g. a button changing from an on-state to an off-state). There may be more than one dirty rectangle that is identified in the comparison of the scene graph states. Thus, multiple objects within a scene may change simultaneously causing the identification of more than one dirty rectangle.

From the list of dirty rectangles 370, a list of MPEG fragment rectangles (i.e. spatially defined fragments, such as a plurality of macroblocks on macroblock boundaries) can be determined for the dirty rectangle. It should be recognized that the term MPEG fragment rectangle as used in the present context refers to spatial data and not frequency transformed data and is referred to as an MPEG fragment rectangle because MPEG requires a block-based formatting schema i.e. macroblocks that are generally 16×16 pixels in shape. Defining dirty rectangles as MPEG fragment rectangles can be achieved by defining an MPEG fragment rectangle for a dirty rectangle wherein the dirty rectangle is fully encompassed within a selection of macroblocks. Thus, the dirty rectangle fits within a rectangle composed of spatially defined macroblocks. Preferably the dirty rectangles are combined or split to limit the number of MPEG fragment rectangles that are present or to avoid small changes in large rectangles.

For each MPEG fragment rectangle, a listing of nodes according to z-order (depth) in the scene graph that contributed to the rectangle contents is determined. This can be achieved by omitting nodes that are invisible, have a low opacity, or have a transparent texture.

For each MPEG fragment rectangle, a hash value 382 is created based upon relevant properties of all nodes that have contributed to the rectangle contents (for example absolute position, width, height, transformation matrix, hash of texture bitmap, opacity). If the cache contains an encoded MPEG fragment associated with that hash value, then the encoded MPEG fragment is retrieved from the cache. In the present context, the term encoded MPEG fragment, refers to a portion of a full frame of video that has been encoded according to the MPEG standard. The encoding may simply be DCT encoding for blocks of data or may also include MPEG specific header information for the encoded material. If the calculated hash value does not match an MPEG fragment in the cache, then the dirty rectangle contents (using the scene graph state) are rendered from a three dimensional world view to a two dimensional screen view and the rendered pixel data (i.e. spatial data) are encoded in an encoder, such as an MPEG encoder 385. The encoded MPEG data (e.g. encoded MPEG fragment(s)) for the scene is stored into the cache.

As part of the encoding process, the fragment is analyzed to determine whether the encoding can best be performed as ‘inter’ encoding (an encoding relative to the previous screen state) or whether it is encoded as ‘intra’ encoding (an independent encoding). Inter-encoding is preferred in general because it results in less bandwidth and may result in higher quality streams. All changes in nodes between scene graphs are determined including movement, changes of opacity, and changes in texture for example. The system then evaluates whether these changes contribute to a fragment, and whether it is possible to express these changes efficiently into the video codec's primitives. If the evaluation indicates that changes to dominant nodes can be expressed well in the videocodec's primitives, then the fragment is inter-encoded. These steps are repeated for every screen update. Since the ‘new scene graph’ will become ‘previous scene graph’ in a next screen update, intermediate results can be reused from previous frames.

FIG. 4 shows an exemplary screen shot 400 of an application that may be rendered on a display device according to the previously described methodology. As shown, the display shows a video frame of the application that has the title “Movie Catalogue.” 410 The video frame also includes a static background 420 and also shows a plurality of selectable movie frames 431, 432, 433, 434. Each movie frame is selectable and associated with a separate underlying movie. The movie frames may include one or more full-motion elements (e.g. may display a clip from the movie or a transition of multiple images, or may be movable in a scripted fashion) the video frame 400 includes the titles (431a, 432a, 433a, 434a) for each of the displayed movies. In the present example, there are four movie frames and associated titles displayed on the current screen. Additionally, the video frame includes a right pointing arrow 440 and a left pointing arrow 450 that when selected provides the user with additional movies that may be selected. This screen shot may be displayed using an application such as a web-browser or another graphical display application such as an application execution environment. It should be understood that the application may reside remote from the client device wherein video content, such as a sequence of MPEG video frames such as an MPEG elementary stream is sent from a server to the client device. The video content represents the output display of the application and the server may include the environment for executing the application and the graphical output is transformed to an MPEG elementary stream in accordance with disclosed embodiments.

FIG. 5 shows a representative DOM tree 500 for the application of FIG. 4. The DOM tree is a document object model representation of the hierarchical objects in a tree structure with associated nodes. A document object model is a cross-platform and language-independent convention for representing and interacting with objects in HTML, XHTML and XML documents. The document object model does not include position information, fonts or effects. This information would be included in an associated CSS document (cascading style sheet document). As shown, there are four levels (501-504) to the DOM tree and the nodes entitled “Body” 502 and “list” 503 each include multiple sub-nodes. Thus, the Body node 502 includes the, Title, l-arrow, list, r-arrow objects 510, 511, 512,513 and also the list objects of cover1, cover2, cover3, and cover4 objects 520, 521, 522, 523. The construction of DOM trees are well known in the art and are typically performed by applications, such as, web browsers.

FIG. 6 shows an exemplary scene graph model of the application screen shot of FIG. 4 that can be built based upon the DOM tree of FIG. 5. A scene graph is a data structure used for representing both logical and spatial objects for a graphical scene. The complete “scene graph state” includes also the textures, spatial information that describes how the texture is positioned into a 2D or 3D space (e.g. a transformation matrix), and all other attributes that are necessary to render the screen. In an exemplary embodiment using the OpenGL API to interface to WebKit, the spatial information for the present example is a 4×4 matrix that specifies translation (i.e. position of the texture in space), rotation, slanting, shearing, shrinking etc. For simplicity, the following examples use only 2D coordinates, but it should be understood that this could be extended to a 3D transformation matrix. Programs that employ scene graphs include graphics applications (e.g. WebKit, Adobe Acrobat, AutoCAD, CorelDraw, VRML97 etc., graphics acceleration programs and corresponding graphics acceleration hardware and additionally 3D applications and games.

The tree like structure provides a hierarchical representation wherein attributes of parent objects can be attributed to the child objects. The root object represents the entire scene 610, while child nodes of a certain node may contain a decomposition of the parent node into smaller objects. The nodes contain may contain a texture (bitmap object), a 3D transformation matrix that specifies how the texture is positioned in a 3D space, and/or other graphical attributes such as visibility and transparency. A child node inherits all attributes, transformations, filters, from the parent node.

For example, movement between scene graphs for an object such as the “cover list” 620 would indicate that each of the child objects (cover1, cover2, cover3, and cover4) 621, 622, 623, 624 would also move by an equal amount. As shown, the screen shot of FIG. 4 includes a hierarchy wherein there is a static layer 615, a cover list layer 620, and a background layer 630 and cover1, cover2, cover3, and cover4 are at a sub-layer for the cover list layer. The choice of objects that are associated with a specific layer is performed by the application execution environment, such as in a web browser.

FIG. 7 shows a scene graph state with associated screen position information. As shown, the upper left position of each object is provided in scene graph (i.e. world coordinates). For example, the cover list layer 620 begins at (30, 400), which is 30 pixels in the X direction (assuming standard video X, Y coordinates) and 400 pixels down in the Y direction. This scene graph state allows a web browser or other application that produces a scene graph state to instruct a graphical processing unit or other program, such as embodiments of the invention that include a video construction engine, such as that shown and discussed with respect to FIGS. 1-3 to render the movie covers 621, 622, 623, 624 including certain effects (shadows, reflections) and to be able to manipulate the position of these objects. The web browser or other application execution environment would then pass the scene graph state and request rendering of the screen. Often the standardized OpenGL API is used for this communication to be able to interface to many different GPUs. The OpenGL API is not only used by web browsers, but by many applications in general, across many Operating Systems (Linux, Windows, Android).

FIG. 8 shows a previous scene graph state 800 and a current scene graph state 810 where the previous scene graph state is on the left and the current scene graph state is on the right. As shown, in both scene graph states there are three layers, a static layer, a cover list layer, and a background layer that are all coupled to the head node. The cover list layer has an additional four objects (cover1, cover2, cover3 and cover 4) at a lower sub-layer. According to embodiments the invention, the scene graph states are compared, where for example the previous transformation matrix is subtracted from the current transformation matrix. This yields the motion of the objects relative to their previous position. It is thus discovered that cover1, cover2, cover3, and cover4 have moved 10 units in the ‘x’ axis direction (e.g. cover1 moves from 60,430 to 70,430 etc.). It is then determined which macroblocks are covered by the new positions of the covers, and a motion vector is set to (10, 0) for each of these macroblocks.

The scene graph comparison between the previous scene graph and the current scene graph may be performed in the following manner wherein the scene graph is transformed from a 3D to a 2D space. A node in a scene graph consists of an object having a texture (2D bitmap) and a transformation how that object is floating in space. It also contains the z-order (absolute order to render things). In OpenGL the transformation consists of a matrix:

m[0] m[4] m[8] m[12] m[1] m[5] m[9] m[13] m[2] m[6] m[10] m[14] m[3] m[7] m[11] m[15]

This transformation is applied to an element ‘a’ in a 3D space by matrix multiplication. The element ‘a’ is identified by four points: the origin and the three top positions of the object in x, y and z direction. The bottom row, i.e. elements m[12], m[13] and m[14] specify translation in 3D space. Elements m[0], m[4], m[8], m[1], m[5], m[9], m[2], m[6], m[10] specify the three top positions of an object (i.e. furthest point out in x, y, z direction) where that particular point will end up by using matrix multiplication. This allows for object or frame rotation, slanting, shearing, shrinking, zooming, and translation etc. and repositioning of the object in world space at any time.

When two transformations have been applied to an object according to matrix ‘m’ (from the previous scene graph) and ‘n’ (from the current scene graph) then the “difference” between the two is m-n: matrix subtraction. The result of the matrix subtraction gives the amount of rotation, slanting, shearing, shrinking, zooming, translation etc. that has been performed to the object between the previous frame and the current frame.

Projecting a 3D image to a 2D surface is well known in the art. In one embodiment, the system first calculates projections of the 3D scene graphs onto a 2D plane, where the transformation matrices also become 2D. The motion vector (obtained by subtracting the transformation matrices) is then 2D and can be directly applied by the MPEG encoder. One motion vector per (destination) macroblock is passed, if motion was detected. The motion vector has a defined (x, y) direction, having a certain length that indicates direction and distance covered between the current frame and the previous frame. The encoder then assumes that the reference information for a macroblock is located in the reverse direction of the motion vector. If no motion was detected, then either the macroblock did not change, or it changed entirely and then it is intra-encoded.

FIG. 9 is an exemplary motion field that shows all of the motion vectors for macroblocks in a scene wherein all of the macroblocks have moved 10 units to the right. This might happen in a scrolling scenario where a user provides user input wanting to move elements on the display screen to the right. The user may be viewing a television or other device and may send a control signal to the server that is indicative of a right arrow key or a right-ward swipe. This control signal is received by the system and the control signal is used to generate a scene graph update within the Application Execution Environment. Once a scene graph is created, the video construction engine and the internal components of the video construction engine create an encoded video signal that is transmitted from the server to the client device and then displayed on the client device. The provided motion field is the result of the scene graph state comparison between the previous and current scene graph states wherein the transformation matrices are subtracted.

FIG. 10 shows a motion field for the rotation of an image. For this example, the transformation matrices of the previous and current scene graph states are subtracted and the motion vectors indicate that there is a rotation of the objects within the image. Note that the macroblocks themselves are not rotated; consequently, there will be a residual error after the motion has been compensated. Thus, residual error calculations as are known in the art for motion vectors may be calculated. The residual error may be considered to be graphical information This may be performed by the MPEG encoder or by the video construction engine. Slanting, shearing, and other movements will result in other motion fields.

Hashing and caching of dirty rectangles on individual layers of a scene graph state is more efficient compared to hashing and caching of 2D projection of these layers, because the layers represent independent changes.

It should be noted that some Application Execution Environments might use one ‘background’ layer where it renders objects for which it chooses not to create a separate render layer. This could be a wall clock, for example. Changes to this layer are analyzed resulting in one or more dirty rectangles. In principle all rectangles depend on the background (if the background changes, parts of the background are likely visible in the rectangle due to the macroblock snapping). To avoid the background being part of every rectangle's hash function, and thus to avoid a re-rendering and re-encoding of all rectangles when the background changes (e.g. when the seconds hand moves in the wall clock object), the background is excluded from the scene graph and it is not available as an MPEG fragment.

The present invention may be embodied in many different forms, including, but in no way limited to, computer program logic for use with a processor (e.g., a microprocessor, microcontroller, digital signal processor, or general purpose computer), programmable logic for use with a programmable logic device (e.g., a Field Programmable Gate Array (FPGA) or other PLD), discrete components, integrated circuitry (e.g., an Application Specific Integrated Circuit (ASIC)), or any other means including any combination thereof. In an embodiment of the present invention, predominantly all of the reordering logic may be implemented as a set of computer program instructions that is converted into a computer executable form, stored as such in a computer readable medium, and executed by a microprocessor within the array under the control of an operating system.

Computer program logic implementing all or part of the functionality previously described herein may be embodied in various forms, including, but in no way limited to, a source code form, a computer executable form, and various intermediate forms (e.g., forms generated by an assembler, compiler, networker, or locator.) Source code may include a series of computer program instructions implemented in any of various programming languages (e.g., an object code, an assembly language, or a high-level language such as Fortran, C, C++, JAVA, or HTML) for use with various operating systems or operating environments. The source code may define and use various data structures and communication messages. The source code may be in a computer executable form (e.g., via an interpreter), or the source code may be converted (e.g., via a translator, assembler, or compiler) into a computer executable form.

The computer program may be fixed in any form (e.g., source code form, computer executable form, or an intermediate form) either permanently or transitorily in a tangible storage medium, such as a semiconductor memory device (e.g., a RAM, ROM, PROM, EEPROM, or Flash-Programmable RAM), a magnetic memory device (e.g., a diskette or fixed disk), an optical memory device (e.g., a CD-ROM), a PC card (e.g., PCMCIA card), or other memory device. The computer program may be fixed in any form in a signal that is transmittable to a computer using any of various communication technologies, including, but in no way limited to, analog technologies, digital technologies, optical technologies, wireless technologies, networking technologies, and internetworking technologies. The computer program may be distributed in any form as a removable storage medium with accompanying printed or electronic documentation (e.g., shrink wrapped software or a magnetic tape), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the communication system (e.g., the Internet or World Wide Web.)

Hardware logic (including programmable logic for use with a programmable logic device) implementing all or part of the functionality previously described herein may be designed using traditional manual methods, or may be designed, captured, simulated, or documented electronically using various tools, such as Computer Aided Design (CAD), a hardware description language (e.g., VHDL or AHDL), or a PLD programming language (e.g., PALASM, ABEL, or CUPL.)

While the invention has been particularly shown and described with reference to specific embodiments, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended clauses. As will be apparent to those skilled in the art, techniques described above for panoramas may be applied to images that have been captured as non-panoramic images, and vice versa.

Embodiments of the present invention may be described, without limitation, by the following clauses. While these embodiments have been described in the clauses by process steps, an apparatus comprising a computer with associated display capable of executing the process steps in the clauses below is also included in the present invention. Likewise, a computer program product including computer executable instructions for executing the process steps in the clauses below and stored on a computer readable medium is included within the present invention.

Claims

1. A method for creating a composited video frame sequence, the method comprising:

at a system including one or more processors and memory storing instructions for execution by the processor:
comparing a current scene graph state with a previous scene graph state, wherein the current scene graph state and the previous scene graph state are respective states of a tree data structure that includes a plurality of hierarchical nodes associated with one or more objects within a graphical scene and that includes spatial information for the one or more objects representing respective positions of the one or more objects in the graphical scene, wherein the comparing comprises comparing, for at least some of the one or more objects, the spatial information of the current scene graph state with the spatial information of the previous scene graph state;
determining if at least one object of the one or more objects has moved based upon the comparison of the scene graph states;
in accordance with a determination that at least one object of the one or more objects has moved, determining motion information about the at least one object based on the comparison of the scene graph states; and
defining motion vectors for the composited video frame sequence based on the motion information.

2. The method according to claim 1 wherein determining motion information also includes determining graphical information about the objects and the graphical information is used in an encoding process.

3. The method according to claim 1 wherein the motion vectors are two-dimensional motion vectors.

4. The method according to claim 1, further comprising:

converting the current scene graph state and the previous scene graph state from a three dimensional representation to a two dimensional representation.

5. The method according to claim 1, further comprising:

constructing an MPEG encoded video sequence using the motion vectors.

6. The method according to claim 1, wherein the current scene graph state and the previous scene graph state are created by an application engine.

7. The method according to claim 1, wherein the current scene graph state includes OpenGL function calls.

8. The method according to claim 1 further comprising:

creating a hash of parameters for objects that have changed between the current and previous scene graph states.

9. The method according to claim 8, further comprising:

comparing the hash to a table of hashes representative of previous scene graph states;
if the current hash matches a hash within the table of hashes, retrieving MPEG encoded elements associated with the hash within the table of hashes; and
providing the MPEG encoded elements to a stitcher for stitching the MPEG encoded elements with other MPEG encoded elements to form one or more MPEG encoded video frames in a series of MPEG encoded video frames.

10. The method according to claim 9 further comprising:

providing two dimension screen representation data for objects that have changed between the current and previous scene graph states to an encoder wherein the encoder compresses the data using at least spatial techniques to produce one or more encoded elements.

11. The method according to claim 10, wherein the encoded elements are passed to a stitching block forming an encoded MPEG frame from the encoded elements where the encoded MPEG frame is part of an MPEG video sequence.

12. A non-transitory computer readable storage medium storing one or more programs for creating an encoded video frame sequence, the one or more programs comprising instructions for:

comparing a current scene graph state with a previous scene graph state, wherein the current scene graph state and the previous scene graph state are respective states of a tree data structure that includes a plurality of hierarchical nodes associated with one or more objects within a graphical scene and that includes spatial information for the one or more objects representing respective positions of the one or more objects in the graphical scene, wherein the comparing comprises comparing, for at least some of the one or more objects, the spatial information of the current scene graph state with the spatial information of the previous scene graph state;
determining if at least one object of the one or more objects has moved based upon the comparison of the scene graph state;
in accordance with a determination that at least one object of the one or more objects has moved, determining motion information about the at least one object based on the comparison of the scene graph states; and
defining motion vectors for the composited video frame sequence based on the motion information.

13. The non-transitory computer readable storage medium according to claim 12, the one or more programs further comprising instructions for:

constructing an MPEG encoded video sequence using the motion vectors.

14. The non-transitory computer readable storage medium according to claim 12, wherein the current scene graph state and the previous scene graph state are determined by an application engine.

15. The non-transitory computer readable storage medium according to claim 14, wherein the current scene graph state includes OpenGL function calls.

16. The non-transitory computer readable storage medium according to claim 12, the one or more programs further comprising instructions for:

creating a hash of parameters for objects that have changed between the current screen graph state and the previous screen graph state.

17. The non-transitory computer readable storage medium according to claim 16, the one or more programs further comprising instructions for:

comparing the hash to a table of hashes of parameters for objects from previous scene graph states;
retrieving MPEG encoded elements associated with the hash from the table of hashes if the hash for parameters for elements that have changed in the current scene graph state matches a hash within the table of hashes; and
providing the MPEG encoded elements to a stitcher for stitching the MPEG encoded elements associated with the matched hash with other MPEG encoded elements to form one or more MPEG encoded video frames in a series of MPEG encoded video frames.

18. The non-transitory computer readable storage medium according to claim 12, the one or more programs further comprising instructions for:

providing data representative of a two dimension screen representation of elements that have changed between the current scene graph state and the previous scene graph state to an encoder wherein the encoder compresses the data using at least spatial techniques to produce one or more MPEG elements.

19. The non-transitory computer readable storage medium according to claim 18, the one or more programs further comprising instructions for passing the encoded elements to the stitcher to form an encoded MPEG frame from the encoded MPEG elements where the encoded MPEG frame is part of an MPEG video sequence.

20. The method according to claim 1, wherein the plurality of hierarchical nodes comprises one or more nodes having one or more respective sub-nodes, wherein the one or more respective sub-nodes of the one or more nodes correspond to the one or more objects within the graphical scene and include the respective spatial information for the one or more objects.

21. The non-transitory computer readable storage medium according to claim 12, wherein the plurality of hierarchical nodes comprises one or more nodes having one or more respective sub-nodes, wherein the one or more respective sub-nodes of the one or more nodes correspond to the one or more objects within the graphical scene and include the respective spatial information for the one or more objects.

Referenced Cited
U.S. Patent Documents
3889050 June 1975 Thompson
3934079 January 20, 1976 Barnhart
3997718 December 14, 1976 Ricketts et al.
4002843 January 11, 1977 Rackman
4032972 June 28, 1977 Saylor
4077006 February 28, 1978 Nicholson
4081831 March 28, 1978 Tang et al.
4107734 August 15, 1978 Percy et al.
4107735 August 15, 1978 Frohbach
4145720 March 20, 1979 Weintraub et al.
4168400 September 18, 1979 de Couasnon et al.
4186438 January 29, 1980 Benson et al.
4222068 September 9, 1980 Thompson
4245245 January 13, 1981 Matsumoto et al.
4247106 January 27, 1981 Jeffers et al.
4253114 February 24, 1981 Tang et al.
4264924 April 28, 1981 Freeman
4264925 April 28, 1981 Freeman et al.
4290142 September 15, 1981 Schnee et al.
4302771 November 24, 1981 Gargini
4308554 December 29, 1981 Percy et al.
4350980 September 21, 1982 Ward
4367557 January 4, 1983 Stern et al.
4395780 July 1983 Gohm et al.
4408225 October 4, 1983 Ensinger et al.
4450477 May 22, 1984 Lovett
4454538 June 12, 1984 Toriumi
4466017 August 14, 1984 Banker
4471380 September 11, 1984 Mobley
4475123 October 2, 1984 Dumbauld et al.
4484217 November 20, 1984 Block et al.
4491983 January 1, 1985 Pinnow et al.
4506387 March 19, 1985 Walter
4507680 March 26, 1985 Freeman
4509073 April 2, 1985 Baran et al.
4523228 June 11, 1985 Banker
4533948 August 6, 1985 McNamara et al.
4536791 August 20, 1985 Campbell et al.
4538174 August 27, 1985 Gargini et al.
4538176 August 27, 1985 Nakajima et al.
4553161 November 12, 1985 Citta
4554581 November 19, 1985 Tentler et al.
4555561 November 26, 1985 Sugimori et al.
4562465 December 31, 1985 Glaab
4567517 January 28, 1986 Mobley
4573072 February 25, 1986 Freeman
4591906 May 27, 1986 Morales-Garza et al.
4602279 July 22, 1986 Freeman
4614970 September 30, 1986 Clupper et al.
4616263 October 7, 1986 Eichelberger
4625235 November 25, 1986 Watson
4627105 December 2, 1986 Ohashi et al.
4633462 December 30, 1986 Stifle et al.
4670904 June 2, 1987 Rumreich
4682360 July 21, 1987 Frederiksen
4695880 September 22, 1987 Johnson et al.
4706121 November 10, 1987 Young
4706285 November 10, 1987 Rumreich
4709418 November 1987 Fox et al.
4710971 December 1, 1987 Nozaki et al.
4718086 January 5, 1988 Rumreich et al.
4732764 March 22, 1988 Hemingway et al.
4734764 March 29, 1988 Pocock et al.
4748689 May 1988 Mohr
4749992 June 7, 1988 Fitzemeyer et al.
4750036 June 7, 1988 Martinez
4754426 June 28, 1988 Rast et al.
4760442 July 26, 1988 O'Connell et al.
4763317 August 9, 1988 Lehman et al.
4769833 September 6, 1988 Farleigh et al.
4769838 September 6, 1988 Hasegawa
4789863 December 6, 1988 Bush
4792849 December 20, 1988 McCalley et al.
4801190 January 31, 1989 Imoto
4805134 February 14, 1989 Calo et al.
4807031 February 21, 1989 Broughton et al.
4816905 March 28, 1989 Tweedy et al.
4821102 April 11, 1989 Ichikawa et al.
4823386 April 18, 1989 Dumbauld et al.
4827253 May 2, 1989 Maltz
4827511 May 2, 1989 Masuko
4829372 May 9, 1989 McCalley et al.
4829558 May 9, 1989 Welsh
4847698 July 11, 1989 Freeman
4847699 July 11, 1989 Freeman
4847700 July 11, 1989 Freeman
4848698 July 18, 1989 Newell et al.
4860379 August 22, 1989 Schoeneberger et al.
4864613 September 5, 1989 Van Cleave
4876592 October 24, 1989 Von Kohorn
4889369 December 26, 1989 Albrecht
4890320 December 26, 1989 Monslow et al.
4891694 January 2, 1990 Way
4901367 February 13, 1990 Nicholson
4903126 February 20, 1990 Kassatly
4905094 February 27, 1990 Pocock et al.
4912760 March 27, 1990 West, Jr. et al.
4918516 April 17, 1990 Freeman
4920566 April 24, 1990 Robbins et al.
4922532 May 1, 1990 Farmer et al.
4924303 May 8, 1990 Brandon et al.
4924498 May 8, 1990 Farmer et al.
4937821 June 26, 1990 Boulton
4941040 July 10, 1990 Pocock et al.
4947244 August 7, 1990 Fenwick et al.
4961211 October 2, 1990 Tsugane et al.
4963995 October 16, 1990 Lang
4975771 December 4, 1990 Kassatly
4989245 January 29, 1991 Bennett
4994909 February 19, 1991 Graves et al.
4995078 February 19, 1991 Monslow et al.
5003384 March 26, 1991 Durden et al.
5008934 April 16, 1991 Endoh
5014125 May 7, 1991 Pocock et al.
5027400 June 25, 1991 Baji et al.
5051720 September 24, 1991 Kittirutsunetorn
5051822 September 24, 1991 Rhoades
5057917 October 15, 1991 Shalkauser et al.
5058160 October 15, 1991 Banker et al.
5060262 October 22, 1991 Bevins, Jr et al.
5077607 December 31, 1991 Johnson et al.
5083800 January 28, 1992 Lockton
5088111 February 11, 1992 McNamara et al.
5093718 March 3, 1992 Hoarty et al.
5109414 April 28, 1992 Harvey et al.
5113496 May 12, 1992 McCalley et al.
5119188 June 2, 1992 McCalley et al.
5130792 July 14, 1992 Tindell et al.
5132992 July 21, 1992 Yurt et al.
5133009 July 21, 1992 Rumreich
5133079 July 21, 1992 Ballantyne et al.
5136411 August 4, 1992 Paik et al.
5142575 August 25, 1992 Farmer et al.
5144448 September 1, 1992 Hornbaker, III et al.
5155591 October 13, 1992 Wachob
5172413 December 15, 1992 Bradley et al.
5191410 March 2, 1993 McCalley et al.
5195092 March 16, 1993 Wilson et al.
5208665 May 4, 1993 McCalley et al.
5220420 June 15, 1993 Hoarty et al.
5230019 July 20, 1993 Yanagimichi et al.
5231494 July 27, 1993 Wachob
5236199 August 17, 1993 Thompson, Jr.
5247347 September 21, 1993 Letteral et al.
5253341 October 12, 1993 Rozmanith et al.
5262854 November 16, 1993 Ng
5262860 November 16, 1993 Fitzpatrick et al.
5303388 April 12, 1994 Kreitman et al.
5319455 June 7, 1994 Hoarty et al.
5319707 June 7, 1994 Wasilewski et al.
5321440 June 14, 1994 Yanagihara et al.
5321514 June 14, 1994 Martinez
5351129 September 27, 1994 Lai
5355162 October 11, 1994 Yazolino et al.
5359601 October 25, 1994 Wasilewski et al.
5361091 November 1, 1994 Hoarty et al.
5371532 December 6, 1994 Gelman et al.
5404393 April 4, 1995 Remillard
5408274 April 18, 1995 Chang et al.
5410343 April 25, 1995 Coddington et al.
5410344 April 25, 1995 Graves et al.
5412415 May 2, 1995 Cook et al.
5412720 May 2, 1995 Hoarty
5418559 May 23, 1995 Blahut
5422674 June 6, 1995 Hooper et al.
5422887 June 6, 1995 Diepstraten et al.
5442389 August 15, 1995 Blahut et al.
5442390 August 15, 1995 Hooper et al.
5442700 August 15, 1995 Snell et al.
5446490 August 29, 1995 Blahut et al.
5469283 November 21, 1995 Vinel et al.
5469431 November 21, 1995 Wendorf et al.
5471263 November 28, 1995 Odaka
5481542 January 2, 1996 Logston et al.
5485197 January 16, 1996 Hoarty
5487066 January 23, 1996 McNamara et al.
5493638 February 20, 1996 Hooper et al.
5495283 February 27, 1996 Cowe
5495295 February 27, 1996 Long
5497187 March 5, 1996 Banker et al.
5517250 May 14, 1996 Hoogenboom et al.
5526034 June 11, 1996 Hoarty et al.
5528281 June 18, 1996 Grady et al.
5537397 July 16, 1996 Abramson
5537404 July 16, 1996 Bentley et al.
5539449 July 23, 1996 Blahut et al.
RE35314 August 20, 1996 Logg
5548340 August 20, 1996 Bertram
5550578 August 27, 1996 Hoarty et al.
5557316 September 17, 1996 Hoarty et al.
5559549 September 24, 1996 Hendricks et al.
5561708 October 1, 1996 Remillard
5570126 October 29, 1996 Blahut et al.
5570363 October 29, 1996 Holm
5579143 November 26, 1996 Huber
5581653 December 3, 1996 Todd
5583927 December 10, 1996 Ely et al.
5587734 December 24, 1996 Lauder et al.
5589885 December 31, 1996 Ooi
5592470 January 7, 1997 Rudrapatna et al.
5594507 January 14, 1997 Hoarty
5594723 January 14, 1997 Tibi
5594938 January 14, 1997 Engel
5596693 January 21, 1997 Needle et al.
5600364 February 4, 1997 Hendricks et al.
5600573 February 4, 1997 Hendricks et al.
5608446 March 4, 1997 Carr et al.
5617145 April 1, 1997 Huang et al.
5621464 April 15, 1997 Teo et al.
5625404 April 29, 1997 Grady et al.
5630757 May 20, 1997 Gagin et al.
5631693 May 20, 1997 Wunderlich et al.
5631846 May 20, 1997 Szurkowski
5632003 May 20, 1997 Davidson et al.
5649283 July 15, 1997 Galler et al.
5668592 September 16, 1997 Spaulding, II
5668599 September 16, 1997 Cheney et al.
5708767 January 13, 1998 Yeo et al.
5710815 January 20, 1998 Ming et al.
5712906 January 27, 1998 Grady et al.
5740307 April 14, 1998 Lane
5742289 April 21, 1998 Naylor et al.
5748234 May 5, 1998 Lippincott
5754941 May 19, 1998 Sharpe et al.
5786527 July 28, 1998 Tarte
5790174 August 4, 1998 Richard, III et al.
5802283 September 1, 1998 Grady et al.
5812665 September 22, 1998 Hoarty et al.
5812786 September 22, 1998 Seazholtz et al.
5815604 September 29, 1998 Simons et al.
5818438 October 6, 1998 Howe et al.
5821945 October 13, 1998 Yeo et al.
5822537 October 13, 1998 Katseff et al.
5828371 October 27, 1998 Cline et al.
5844594 December 1, 1998 Ferguson
5845083 December 1, 1998 Hamadani et al.
5862325 January 19, 1999 Reed et al.
5864820 January 26, 1999 Case
5867208 February 2, 1999 McLaren
5883661 March 16, 1999 Hoarty et al.
5903727 May 11, 1999 Nielsen
5903816 May 11, 1999 Broadwin et al.
5905522 May 18, 1999 Lawler
5907681 May 25, 1999 Bates et al.
5917822 June 29, 1999 Lyles et al.
5946352 August 31, 1999 Rowlands et al.
5952943 September 14, 1999 Walsh et al.
5959690 September 28, 1999 Toebes et al.
5961603 October 5, 1999 Kunkel et al.
5963203 October 5, 1999 Goldberg et al.
5966163 October 12, 1999 Lin et al.
5978756 November 2, 1999 Walker et al.
5982445 November 9, 1999 Eyer et al.
5990862 November 23, 1999 Lewis
5995146 November 30, 1999 Rasmussen
5995488 November 30, 1999 Kalhunte et al.
5999970 December 7, 1999 Krisbergh et al.
6014416 January 11, 2000 Shin et al.
6021386 February 1, 2000 Davis et al.
6031989 February 29, 2000 Cordell
6034678 March 7, 2000 Hoarty
6049539 April 11, 2000 Lee et al.
6049831 April 11, 2000 Gardell et al.
6052555 April 18, 2000 Ferguson
6055314 April 25, 2000 Spies et al.
6055315 April 25, 2000 Doyle et al.
6064377 May 16, 2000 Hoarty et al.
6078328 June 20, 2000 Schumann et al.
6084908 July 4, 2000 Chiang et al.
6100883 August 8, 2000 Hoarty
6108625 August 22, 2000 Kim
6131182 October 10, 2000 Beakes et al.
6141645 October 31, 2000 Chi-Min et al.
6141693 October 31, 2000 Perlman et al.
6144698 November 7, 2000 Poon et al.
6167084 December 26, 2000 Wang et al.
6169573 January 2, 2001 Sampath-Kumar et al.
6177931 January 23, 2001 Alexander et al.
6182072 January 30, 2001 Leak et al.
6184878 February 6, 2001 Alonso et al.
6192081 February 20, 2001 Chiang et al.
6198822 March 6, 2001 Doyle et al.
6205582 March 20, 2001 Hoarty
6226041 May 1, 2001 Florencio et al.
6236730 May 22, 2001 Cowieson et al.
6243418 June 5, 2001 Kim
6253238 June 26, 2001 Lauder et al.
6256047 July 3, 2001 Isobe et al.
6259826 July 10, 2001 Pollard et al.
6266369 July 24, 2001 Wang et al.
6266684 July 24, 2001 Kraus et al.
6275496 August 14, 2001 Burns et al.
6292194 September 18, 2001 Powell, III
6305020 October 16, 2001 Hoarty et al.
6317151 November 13, 2001 Ohsuga et al.
6317885 November 13, 2001 Fries
6349284 February 19, 2002 Park et al.
6385771 May 7, 2002 Gordon
6386980 May 14, 2002 Nishino et al.
6389075 May 14, 2002 Wang et al.
6389218 May 14, 2002 Gordon et al.
6415031 July 2, 2002 Colligan et al.
6415437 July 2, 2002 Ludvig et al.
6438140 August 20, 2002 Jungers et al.
6446037 September 3, 2002 Fielder et al.
6459427 October 1, 2002 Mao et al.
6477182 November 5, 2002 Calderone
6480210 November 12, 2002 Martino et al.
6481012 November 12, 2002 Gordon et al.
6512793 January 28, 2003 Maeda
6525746 February 25, 2003 Lau et al.
6536043 March 18, 2003 Guedalia
6557041 April 29, 2003 Mallart
6560496 May 6, 2003 Michener
6564378 May 13, 2003 Satterfield et al.
6578201 June 10, 2003 LaRocca et al.
6579184 June 17, 2003 Tanskanen
6584153 June 24, 2003 Gordon et al.
6588017 July 1, 2003 Calderone
6598229 July 22, 2003 Smyth et al.
6604224 August 5, 2003 Armstrong et al.
6614442 September 2, 2003 Ouyang et al.
6621870 September 16, 2003 Gordon et al.
6625574 September 23, 2003 Taniguchi et al.
6639896 October 28, 2003 Goode et al.
6645076 November 11, 2003 Sugai
6651252 November 18, 2003 Gordon et al.
6657647 December 2, 2003 Bright
6675385 January 6, 2004 Wang
6675387 January 6, 2004 Boucher et al.
6681326 January 20, 2004 Son et al.
6681397 January 20, 2004 Tsai et al.
6684400 January 27, 2004 Goode et al.
6687663 February 3, 2004 McGrath et al.
6691208 February 10, 2004 Dandrea et al.
6697376 February 24, 2004 Son et al.
6704359 March 9, 2004 Bayrakeri et al.
6717600 April 6, 2004 Dutta et al.
6718552 April 6, 2004 Goode
6721794 April 13, 2004 Taylor et al.
6721956 April 13, 2004 Wasilewski
6727929 April 27, 2004 Bates et al.
6731605 May 4, 2004 Deshpande
6732370 May 4, 2004 Gordon et al.
6747991 June 8, 2004 Hemy et al.
6754271 June 22, 2004 Gordon et al.
6754905 June 22, 2004 Gordon et al.
6758540 July 6, 2004 Adolph et al.
6766407 July 20, 2004 Lisitsa et al.
6771704 August 3, 2004 Hannah
6785902 August 31, 2004 Zigmond et al.
6807528 October 19, 2004 Truman et al.
6810528 October 26, 2004 Chatani
6817947 November 16, 2004 Tanskanen
6886178 April 26, 2005 Mao et al.
6907574 June 14, 2005 Xu et al.
6931291 August 16, 2005 Alvarez-Tinoco et al.
6941019 September 6, 2005 Mitchell et al.
6941574 September 6, 2005 Broadwin et al.
6947509 September 20, 2005 Wong
6952221 October 4, 2005 Holtz et al.
6956899 October 18, 2005 Hall et al.
7016540 March 21, 2006 Gong et al.
7030890 April 18, 2006 Jouet et al.
7031385 April 18, 2006 Inoue et al.
7050113 May 23, 2006 Campisano et al.
7089577 August 8, 2006 Rakib et al.
7093028 August 15, 2006 Shao et al.
7095402 August 22, 2006 Kunil et al.
7114167 September 26, 2006 Slemmer et al.
7146615 December 5, 2006 Hervet et al.
7151782 December 19, 2006 Oz et al.
7158676 January 2, 2007 Rainsford
7200836 April 3, 2007 Brodersen et al.
7212573 May 1, 2007 Winger
7224731 May 29, 2007 Mehrotra
7272556 September 18, 2007 Aguilar et al.
7310619 December 18, 2007 Baar et al.
7325043 January 29, 2008 Rosenberg et al.
7346111 March 18, 2008 Winger et al.
7360230 April 15, 2008 Paz et al.
7412423 August 12, 2008 Asano
7412505 August 12, 2008 Slemmer et al.
7421082 September 2, 2008 Kamiya et al.
7444306 October 28, 2008 Varble
7444418 October 28, 2008 Chou et al.
7500235 March 3, 2009 Maynard et al.
7508941 March 24, 2009 O'Toole, Jr. et al.
7512577 March 31, 2009 Slemmer et al.
7543073 June 2, 2009 Chou et al.
7596764 September 29, 2009 Vienneau et al.
7623575 November 24, 2009 Winger
7669220 February 23, 2010 Goode
7742609 June 22, 2010 Yeakel et al.
7743400 June 22, 2010 Kurauchi
7751572 July 6, 2010 Villemoes et al.
7757157 July 13, 2010 Fukuda
7830388 November 9, 2010 Lu
7840905 November 23, 2010 Weber et al.
7936819 May 3, 2011 Craig et al.
7970263 June 28, 2011 Asch
7987489 July 26, 2011 Krzyzanowski et al.
8027353 September 27, 2011 Damola et al.
8036271 October 11, 2011 Winger et al.
8046798 October 25, 2011 Schlack et al.
8074248 December 6, 2011 Sigmon et al.
8118676 February 21, 2012 Craig et al.
8136033 March 13, 2012 Bhargava et al.
8149917 April 3, 2012 Zhang et al.
8155194 April 10, 2012 Winger et al.
8155202 April 10, 2012 Landau
8170107 May 1, 2012 Winger
8194862 June 5, 2012 Herr et al.
8243630 August 14, 2012 Luo et al.
8270439 September 18, 2012 Herr et al.
8284842 October 9, 2012 Craig et al.
8296424 October 23, 2012 Malloy et al.
8370869 February 5, 2013 Paek et al.
8411754 April 2, 2013 Zhang et al.
8442110 May 14, 2013 Pavlovskaia et al.
8473996 June 25, 2013 Gordon et al.
8619867 December 31, 2013 Craig et al.
8621500 December 31, 2013 Weaver et al.
20010008845 July 19, 2001 Kusuda et al.
20010049301 December 6, 2001 Masuda et al.
20020007491 January 17, 2002 Schiller et al.
20020013812 January 31, 2002 Krueger et al.
20020016161 February 7, 2002 Dellien et al.
20020021353 February 21, 2002 DeNies
20020026642 February 28, 2002 Augenbraun et al.
20020027567 March 7, 2002 Niamir
20020032697 March 14, 2002 French et al.
20020040482 April 4, 2002 Sextro et al.
20020047899 April 25, 2002 Son et al.
20020049975 April 25, 2002 Thomas et al.
20020054578 May 9, 2002 Zhang et al.
20020056083 May 9, 2002 Istvan
20020056107 May 9, 2002 Schlack
20020056136 May 9, 2002 Wistendahl et al.
20020059644 May 16, 2002 Andrade et al.
20020062484 May 23, 2002 De Lange et al.
20020067766 June 6, 2002 Sakamoto et al.
20020069267 June 6, 2002 Thiele
20020072408 June 13, 2002 Kumagai
20020078171 June 20, 2002 Schneider
20020078456 June 20, 2002 Hudson et al.
20020083464 June 27, 2002 Tomsen et al.
20020095689 July 18, 2002 Novak
20020105531 August 8, 2002 Niemi
20020108121 August 8, 2002 Alao et al.
20020131511 September 19, 2002 Zenoni
20020136298 September 26, 2002 Anantharamu et al.
20020152318 October 17, 2002 Menon et al.
20020171765 November 21, 2002 Waki et al.
20020175931 November 28, 2002 Holtz et al.
20020178447 November 28, 2002 Plotnick et al.
20020188628 December 12, 2002 Cooper et al.
20020191851 December 19, 2002 Keinan
20020194592 December 19, 2002 Tsuchida et al.
20020196746 December 26, 2002 Allen
20030018796 January 23, 2003 Chou et al.
20030020671 January 30, 2003 Santoro et al.
20030027517 February 6, 2003 Callway et al.
20030035486 February 20, 2003 Kato et al.
20030038893 February 27, 2003 Rajamaki et al.
20030039398 February 27, 2003 McIntyre
20030046690 March 6, 2003 Miller
20030051253 March 13, 2003 Barone, Jr.
20030058941 March 27, 2003 Chen et al.
20030061451 March 27, 2003 Beyda
20030065739 April 3, 2003 Shnier
20030071792 April 17, 2003 Safadi
20030072372 April 17, 2003 Shen et al.
20030076546 April 24, 2003 Johnson et al.
20030088328 May 8, 2003 Nishio et al.
20030088400 May 8, 2003 Nishio et al.
20030095790 May 22, 2003 Joshi
20030107443 June 12, 2003 Yamamoto
20030122836 July 3, 2003 Doyle et al.
20030123664 July 3, 2003 Pedlow, Jr. et al.
20030126608 July 3, 2003 Safadi et al.
20030126611 July 3, 2003 Chernock et al.
20030131349 July 10, 2003 Kuczynski-Brown
20030135860 July 17, 2003 Dureau
20030169373 September 11, 2003 Peters et al.
20030177199 September 18, 2003 Zenoni
20030188309 October 2, 2003 Yuen
20030189980 October 9, 2003 Dvir et al.
20030196174 October 16, 2003 Pierre Cote et al.
20030208768 November 6, 2003 Urdang et al.
20030229719 December 11, 2003 Iwata et al.
20030229900 December 11, 2003 Reisman
20030231218 December 18, 2003 Amadio
20040016000 January 22, 2004 Zhang et al.
20040034873 February 19, 2004 Zenoni
20040040035 February 26, 2004 Carlucci et al.
20040078822 April 22, 2004 Breen et al.
20040088375 May 6, 2004 Sethi et al.
20040091171 May 13, 2004 Bone
20040111526 June 10, 2004 Baldwin et al.
20040117827 June 17, 2004 Karaoguz et al.
20040128686 July 1, 2004 Boyer et al.
20040133704 July 8, 2004 Krzyzanowski et al.
20040136698 July 15, 2004 Mock
20040139158 July 15, 2004 Datta
20040157662 August 12, 2004 Tsuchiya
20040163101 August 19, 2004 Swix et al.
20040184542 September 23, 2004 Fujimoto
20040193648 September 30, 2004 Lai et al.
20040210824 October 21, 2004 Shoff et al.
20040261106 December 23, 2004 Hoffman
20040261114 December 23, 2004 Addington et al.
20040268419 December 30, 2004 Danker et al.
20050015259 January 20, 2005 Thumpudi et al.
20050015816 January 20, 2005 Christofalo et al.
20050021830 January 27, 2005 Urzaiz et al.
20050034155 February 10, 2005 Gordon et al.
20050034162 February 10, 2005 White et al.
20050044575 February 24, 2005 Der Kuyl
20050055685 March 10, 2005 Maynard et al.
20050055721 March 10, 2005 Zigmond et al.
20050071876 March 31, 2005 van Beek
20050076134 April 7, 2005 Bialik et al.
20050089091 April 28, 2005 Kim et al.
20050091690 April 28, 2005 Delpuch et al.
20050091695 April 28, 2005 Paz et al.
20050105608 May 19, 2005 Coleman et al.
20050114906 May 26, 2005 Hoarty et al.
20050132305 June 16, 2005 Guichard et al.
20050135385 June 23, 2005 Jenkins et al.
20050141613 June 30, 2005 Kelly et al.
20050149988 July 7, 2005 Grannan
20050160088 July 21, 2005 Scallan et al.
20050166257 July 28, 2005 Feinleib et al.
20050180502 August 18, 2005 Puri
20050198682 September 8, 2005 Wright
20050213586 September 29, 2005 Cyganski et al.
20050216933 September 29, 2005 Black
20050216940 September 29, 2005 Black
20050226426 October 13, 2005 Oomen et al.
20050273832 December 8, 2005 Zigmond et al.
20050283741 December 22, 2005 Balabanovic et al.
20060001737 January 5, 2006 Dawson et al.
20060020960 January 26, 2006 Relan et al.
20060020994 January 26, 2006 Crane et al.
20060031906 February 9, 2006 Kaneda
20060039481 February 23, 2006 Shen et al.
20060041910 February 23, 2006 Hatanaka et al.
20060088105 April 27, 2006 Shen et al.
20060095944 May 4, 2006 Demircin et al.
20060112338 May 25, 2006 Joung et al.
20060117340 June 1, 2006 Pavlovskaia et al.
20060143678 June 29, 2006 Chou et al.
20060161538 July 20, 2006 Kiilerich
20060173985 August 3, 2006 Moore
20060174026 August 3, 2006 Robinson et al.
20060174289 August 3, 2006 Theberge
20060195884 August 31, 2006 van Zoest et al.
20060203913 September 14, 2006 Kim et al.
20060212203 September 21, 2006 Furuno
20060218601 September 28, 2006 Michel
20060230428 October 12, 2006 Craig et al.
20060242570 October 26, 2006 Croft et al.
20060256865 November 16, 2006 Westerman
20060269086 November 30, 2006 Page et al.
20060271985 November 30, 2006 Hoffman et al.
20060285586 December 21, 2006 Westerman
20060285819 December 21, 2006 Kelly et al.
20070009035 January 11, 2007 Craig et al.
20070009036 January 11, 2007 Craig et al.
20070009042 January 11, 2007 Craig et al.
20070025639 February 1, 2007 Zhou et al.
20070033528 February 8, 2007 Merril et al.
20070033631 February 8, 2007 Gordon et al.
20070074251 March 29, 2007 Oguz et al.
20070079325 April 5, 2007 de Heer
20070115941 May 24, 2007 Patel et al.
20070124282 May 31, 2007 Wittkotter
20070124795 May 31, 2007 McKissick et al.
20070130446 June 7, 2007 Minakami
20070130592 June 7, 2007 Haeusel
20070152984 July 5, 2007 Ording et al.
20070162953 July 12, 2007 Bollinger et al.
20070172061 July 26, 2007 Pinder
20070174790 July 26, 2007 Jing et al.
20070178243 August 2, 2007 Dong et al.
20070234220 October 4, 2007 Khan et al.
20070237232 October 11, 2007 Chang et al.
20070300280 December 27, 2007 Turner et al.
20080046928 February 21, 2008 Poling et al.
20080052742 February 28, 2008 Kopf et al.
20080066135 March 13, 2008 Brodersen et al.
20080084503 April 10, 2008 Kondo
20080086688 April 10, 2008 Chandratillake et al.
20080094368 April 24, 2008 Ording et al.
20080097953 April 24, 2008 Levy et al.
20080098450 April 24, 2008 Wu et al.
20080104520 May 1, 2008 Swenson et al.
20080127255 May 29, 2008 Ress et al.
20080154583 June 26, 2008 Goto et al.
20080163059 July 3, 2008 Craner
20080163286 July 3, 2008 Rudolph et al.
20080170619 July 17, 2008 Landau
20080170622 July 17, 2008 Gordon et al.
20080178125 July 24, 2008 Elsbree et al.
20080178243 July 24, 2008 Dong et al.
20080178249 July 24, 2008 Gordon et al.
20080181221 July 31, 2008 Kampmann et al.
20080189740 August 7, 2008 Carpenter et al.
20080195573 August 14, 2008 Onoda et al.
20080201736 August 21, 2008 Gordon et al.
20080212942 September 4, 2008 Gordon et al.
20080222199 September 11, 2008 Tiu et al.
20080232452 September 25, 2008 Sullivan et al.
20080243918 October 2, 2008 Holtman
20080243998 October 2, 2008 Oh et al.
20080246759 October 9, 2008 Summers
20080253440 October 16, 2008 Srinivasan et al.
20080271080 October 30, 2008 Gossweiler et al.
20090003446 January 1, 2009 Wu et al.
20090003705 January 1, 2009 Zou et al.
20090007199 January 1, 2009 La Joie
20090025027 January 22, 2009 Craner
20090031341 January 29, 2009 Schlack et al.
20090041118 February 12, 2009 Pavlovskaia et al.
20090083781 March 26, 2009 Yang et al.
20090083813 March 26, 2009 Dolce et al.
20090083824 March 26, 2009 McCarthy et al.
20090089188 April 2, 2009 Ku et al.
20090094113 April 9, 2009 Berry et al.
20090094646 April 9, 2009 Walter et al.
20090100465 April 16, 2009 Kulakowski
20090100489 April 16, 2009 Strothmann
20090106269 April 23, 2009 Zuckerman et al.
20090106386 April 23, 2009 Zuckerman et al.
20090106392 April 23, 2009 Zuckerman et al.
20090106425 April 23, 2009 Zuckerman et al.
20090106441 April 23, 2009 Zuckerman et al.
20090106451 April 23, 2009 Zuckerman et al.
20090106511 April 23, 2009 Zuckerman et al.
20090113009 April 30, 2009 Slemmer et al.
20090132942 May 21, 2009 Santoro et al.
20090138966 May 28, 2009 Krause et al.
20090144781 June 4, 2009 Glaser et al.
20090146779 June 11, 2009 Kumar et al.
20090157868 June 18, 2009 Chaudhry
20090158369 June 18, 2009 Van Vleck et al.
20090160694 June 25, 2009 Di Flora
20090172757 July 2, 2009 Aldrey et al.
20090178098 July 9, 2009 Westbrook et al.
20090183219 July 16, 2009 Maynard et al.
20090189890 July 30, 2009 Corbett et al.
20090193452 July 30, 2009 Russ et al.
20090196346 August 6, 2009 Zhang et al.
20090204920 August 13, 2009 Beverley et al.
20090210899 August 20, 2009 Lawrence-Apfelbaum et al.
20090225790 September 10, 2009 Shay et al.
20090228620 September 10, 2009 Thomas et al.
20090228922 September 10, 2009 Haj-khalil et al.
20090233593 September 17, 2009 Ergen et al.
20090251478 October 8, 2009 Maillot et al.
20090254960 October 8, 2009 Yarom et al.
20090265617 October 22, 2009 Randall et al.
20090271512 October 29, 2009 Jorgensen
20090271818 October 29, 2009 Schlack
20090298535 December 3, 2009 Klein et al.
20090313674 December 17, 2009 Ludvig et al.
20090328109 December 31, 2009 Pavlovskaia et al.
20100033638 February 11, 2010 O'Donnell et al.
20100035682 February 11, 2010 Gentile et al.
20100058404 March 4, 2010 Rouse
20100067571 March 18, 2010 White et al.
20100077441 March 25, 2010 Thomas et al.
20100104021 April 29, 2010 Schmit
20100115573 May 6, 2010 Srinivasan et al.
20100118972 May 13, 2010 Zhang et al.
20100131996 May 27, 2010 Gauld
20100146139 June 10, 2010 Brockmann
20100158109 June 24, 2010 Dahlby et al.
20100161825 June 24, 2010 Ronca et al.
20100166071 July 1, 2010 Wu et al.
20100174776 July 8, 2010 Westberg et al.
20100175080 July 8, 2010 Yuen et al.
20100180307 July 15, 2010 Hayes et al.
20100211983 August 19, 2010 Chou
20100226428 September 9, 2010 Thevathasan et al.
20100235861 September 16, 2010 Schein et al.
20100242073 September 23, 2010 Gordon et al.
20100251167 September 30, 2010 DeLuca et al.
20100254370 October 7, 2010 Jana et al.
20100265344 October 21, 2010 Velarde et al.
20100325655 December 23, 2010 Perez
20110002376 January 6, 2011 Ahmed et al.
20110002470 January 6, 2011 Purnhagen et al.
20110023069 January 27, 2011 Dowens
20110035227 February 10, 2011 Lee et al.
20110067061 March 17, 2011 Karaoguz et al.
20110096828 April 28, 2011 Chen et al.
20110107375 May 5, 2011 Stahl et al.
20110110642 May 12, 2011 Salomons et al.
20110150421 June 23, 2011 Sasaki et al.
20110153776 June 23, 2011 Opala et al.
20110167468 July 7, 2011 Lee et al.
20110191684 August 4, 2011 Greenberg
20110231878 September 22, 2011 Hunter et al.
20110243024 October 6, 2011 Osterling et al.
20110258584 October 20, 2011 Williams et al.
20110289536 November 24, 2011 Poder et al.
20110296312 December 1, 2011 Boyer et al.
20110317982 December 29, 2011 Xu et al.
20120023126 January 26, 2012 Jin et al.
20120030212 February 2, 2012 Koopmans et al.
20120137337 May 31, 2012 Sigmon et al.
20120204217 August 9, 2012 Regis et al.
20120209815 August 16, 2012 Carson et al.
20120224641 September 6, 2012 Haberman et al.
20120257671 October 11, 2012 Brockmann et al.
20130003826 January 3, 2013 Craig et al.
20130071095 March 21, 2013 Chauvier et al.
20130086610 April 4, 2013 Brockmann
20130179787 July 11, 2013 Brockmann et al.
20130198776 August 1, 2013 Brockmann
20130254308 September 26, 2013 Rose et al.
20130272394 October 17, 2013 Brockmann et al.
20130304818 November 14, 2013 Brumleve et al.
20140033036 January 30, 2014 Gaur et al.
20140081954 March 20, 2014 Elizarov
Foreign Patent Documents
191599 April 2000 AT
198969 February 2001 AT
250313 October 2003 AT
472152 July 2010 AT
475266 August 2010 AT
550086 February 1986 AU
199060189 November 1990 AU
620735 February 1992 AU
199184838 April 1992 AU
643828 November 1993 AU
2004253127 January 2005 AU
2005278122 March 2006 AU
2010339376 August 2012 AU
2011249132 November 2012 AU
2011258972 November 2012 AU
2011315950 May 2013 AU
682776 March 1964 CA
2052477 March 1992 CA
1302554 June 1992 CA
2163500 May 1996 CA
2231391 May 1997 CA
2273365 June 1998 CA
2313133 June 1999 CA
2313161 June 1999 CA
2528499 January 2005 CA
2569407 March 2006 CA
2728797 April 2010 CA
2787913 July 2011 CA
2798541 December 2011 CA
2814070 April 2012 CA
1507751 June 2004 CN
1969555 May 2007 CN
101180109 May 2008 CN
101627424 January 2010 CN
101637023 January 2010 CN
102007773 April 2011 CN
103647980 March 2014 CN
4408355 October 1994 DE
69516139 D1 December 2000 DE
69132518 D1 September 2001 DE
69333207 D1 July 2004 DE
98961961 August 2007 DE
602008001596 August 2010 DE
602006015650 September 2010 DE
0093549 November 1983 EP
0128771 December 1984 EP
0419137 March 1991 EP
0449633 October 1991 EP
0477786 April 1992 EP
0523618 January 1993 EP
0534139 March 1993 EP
0568453 November 1993 EP
0588653 March 1994 EP
0594350 April 1994 EP
0612916 August 1994 EP
0624039 November 1994 EP
0638219 February 1995 EP
0643523 March 1995 EP
0661888 July 1995 EP
0714684 June 1996 EP
0746158 December 1996 EP
0761066 March 1997 EP
0789972 August 1997 EP
0830786 March 1998 EP
0861560 September 1998 EP
0 881 808 December 1998 EP
0933966 August 1999 EP
0933966 August 1999 EP
1026872 August 2000 EP
1038397 September 2000 EP
1038399 September 2000 EP
1038400 September 2000 EP
1038401 September 2000 EP
1051039 November 2000 EP
1055331 November 2000 EP
1120968 August 2001 EP
1345446 September 2003 EP
1422929 May 2004 EP
1428562 June 2004 EP
1521476 April 2005 EP
1645115 April 2006 EP
1725044 November 2006 EP
1767708 March 2007 EP
1771003 April 2007 EP
1772014 April 2007 EP
1877150 January 2008 EP
1887148 February 2008 EP
1900200 March 2008 EP
1902583 March 2008 EP
1908293 April 2008 EP
1911288 April 2008 EP
1918802 May 2008 EP
2100296 September 2009 EP
2105019 September 2009 EP
2106665 October 2009 EP
2116051 November 2009 EP
2124440 November 2009 EP
2248341 November 2010 EP
2269377 January 2011 EP
2271098 January 2011 EP
2304953 April 2011 EP
2364019 September 2011 EP
2384001 November 2011 EP
2409493 January 2012 EP
2477414 July 2012 EP
2487919 August 2012 EP
2520090 November 2012 EP
2567545 March 2013 EP
2577437 April 2013 EP
2628306 August 2013 EP
2632164 August 2013 EP
2632165 August 2013 EP
2695388 February 2014 EP
2207635 June 2004 ES
8211463 June 1982 FR
2529739 January 1984 FR
2891098 March 2007 FR
2207838 February 1989 GB
2248955 April 1992 GB
2290204 December 1995 GB
2365649 February 2002 GB
2378345 February 2003 GB
1134855 October 2010 HK
1116323 December 2010 HK
19913397 April 1992 IE
99586 February 1998 IL
215133 December 2011 IL
222829 December 2012 IL
222830 December 2012 IL
225525 June 2013 IL
180215 January 1998 IN
200701744 November 2007 IN
200900856 May 2009 IN
200800214 June 2009 IN
3759 March 1992 IS
60-054324 March 1985 JP
63-033988 February 1988 JP
63-263985 October 1988 JP
2001-241993 September 1989 JP
04-373286 December 1992 JP
06-054324 February 1994 JP
7015720 January 1995 JP
7-160292 June 1995 JP
8-265704 October 1996 JP
10-228437 August 1998 JP
10-510131 September 1998 JP
11-134273 May 1999 JP
H11-261966 September 1999 JP
2000-152234 May 2000 JP
2001-203995 July 2001 JP
2001-245271 September 2001 JP
2001-245291 September 2001 JP
2001-514471 September 2001 JP
2002-016920 January 2002 JP
2002-057952 February 2002 JP
2002-112220 April 2002 JP
2002-141810 May 2002 JP
2002-208027 July 2002 JP
2002-319991 October 2002 JP
2003-506763 February 2003 JP
2003-087785 March 2003 JP
2003-529234 September 2003 JP
2004-501445 January 2004 JP
2004-056777 February 2004 JP
2004-110850 April 2004 JP
2004-112441 April 2004 JP
2004-135932 May 2004 JP
2004-264812 September 2004 JP
2004-312283 November 2004 JP
2004-533736 November 2004 JP
2004-536381 December 2004 JP
2004-536681 December 2004 JP
2005-033741 February 2005 JP
2005-084987 March 2005 JP
2005-095599 March 2005 JP
8-095599 April 2005 JP
2005-156996 June 2005 JP
2005-519382 June 2005 JP
2005-523479 August 2005 JP
2005-309752 November 2005 JP
2006-067280 March 2006 JP
2006-512838 April 2006 JP
2007-522727 August 2007 JP
11-88419 September 2007 JP
2008-523880 July 2008 JP
2008-535622 September 2008 JP
04252727 April 2009 JP
2009-543386 December 2009 JP
2011-108155 June 2011 JP
2012-080593 April 2012 JP
04996603 August 2012 JP
05121711 January 2013 JP
53-004612 October 2013 JP
05331008 October 2013 JP
05405819 February 2014 JP
2005-0001362 January 2005 KR
2005-0085827 August 2005 KR
2006067924 June 2006 KR
2006-0095821 September 2006 KR
2007038111 April 2007 KR
20080001298 January 2008 KR
2008024189 March 2008 KR
2010111739 October 2010 KR
2010120187 November 2010 KR
2010127240 December 2010 KR
2011030640 March 2011 KR
2011129477 December 2011 KR
20120112683 October 2012 KR
2013061149 June 2013 KR
2013113925 October 2013 KR
1333200 November 2013 KR
2008045154 November 2013 KR
2013138263 December 2013 KR
1032594 April 2008 NL
1033929 April 2008 NL
2004670 November 2011 NL
2004780 January 2012 NL
239969 December 1994 NZ
99110 December 1993 PT
WO 8202303 July 1982 WO
WO 8908967 September 1989 WO
WO 90/13972 November 1990 WO
WO 93/22877 November 1993 WO
WO 9416534 July 1994 WO
WO 9419910 September 1994 WO
WO 9421079 September 1994 WO
WO 95/15658 June 1995 WO
WO 9532587 November 1995 WO
WO 9533342 December 1995 WO
WO 9614712 May 1996 WO
WO 9627843 September 1996 WO
WO 9631826 October 1996 WO
WO 9637074 November 1996 WO
WO 9642168 December 1996 WO
WO 9716925 May 1997 WO
WO 9733434 September 1997 WO
WO 9739583 October 1997 WO
WO 9826595 June 1998 WO
WO 99/00735 January 1999 WO
WO 99/04568 January 1999 WO
WO 9900735 January 1999 WO
WO 9930496 June 1999 WO
WO 9930497 June 1999 WO
WO 9930500 June 1999 WO
WO 9930501 June 1999 WO
WO 9935840 July 1999 WO
WO 9941911 August 1999 WO
WO 9956468 November 1999 WO
WO 99/65232 December 1999 WO
WO 99/65243 December 1999 WO
WO 9966732 December 1999 WO
WO 0002303 January 2000 WO
WO 00/07372 February 2000 WO
WO 00/08967 February 2000 WO
WO 00/19910 April 2000 WO
WO 00/38430 June 2000 WO
WO 00/41397 July 2000 WO
WO 01/39494 May 2001 WO
WO 01/41447 June 2001 WO
WO 01/82614 November 2001 WO
WO 01/92973 December 2001 WO
WO 02/089487 July 2002 WO
WO 02/076097 September 2002 WO
WO 02/076099 September 2002 WO
WO 03/026232 March 2003 WO
WO 03/026275 March 2003 WO
WO 03/047710 June 2003 WO
WO 03/065683 August 2003 WO
WO 03/071727 August 2003 WO
WO 03/091832 November 2003 WO
WO 2004/012437 February 2004 WO
WO 2004/018060 March 2004 WO
WO 2004/057609 July 2004 WO
WO 2004/073310 August 2004 WO
WO 2005/002215 January 2005 WO
WO 2005/041122 May 2005 WO
WO 2005/053301 June 2005 WO
WO 2005/076575 August 2005 WO
WO 2005/120067 December 2005 WO
WO 2006/014362 February 2006 WO
WO 2006/022881 March 2006 WO
WO 2006/053305 May 2006 WO
WO 2006/067697 June 2006 WO
WO 2006/081634 August 2006 WO
WO 2006/105480 October 2006 WO
WO 2006/110268 October 2006 WO
WO 2007/001797 January 2007 WO
WO 2007/008319 January 2007 WO
WO 2007/008355 January 2007 WO
WO 2007/008356 January 2007 WO
WO 2007/008357 January 2007 WO
WO 2007/008358 January 2007 WO
WO 2007/018722 February 2007 WO
WO 2007/018726 February 2007 WO
WO2008/044916 April 2008 WO
WO 2008/044916 April 2008 WO
WO 2008/086170 July 2008 WO
WO 2008/088741 July 2008 WO
WO 2008/088752 July 2008 WO
WO 2008/088772 July 2008 WO
WO 2008/100205 August 2008 WO
WO 2009/038596 March 2009 WO
WO 2009/099893 August 2009 WO
WO 2009/099895 August 2009 WO
WO 2009/105465 August 2009 WO
WO 2009/110897 September 2009 WO
WO 2009/114247 September 2009 WO
WO 2009/155214 December 2009 WO
WO 2010/044926 April 2010 WO
WO 2010/054136 May 2010 WO
WO 2010/107954 September 2010 WO
WO 2011/014336 September 2010 WO
WO 2011/082364 July 2011 WO
WO 2011/139155 November 2011 WO
WO 2011/149357 December 2011 WO
WO 2012/051528 April 2012 WO
WO 2012/138660 October 2012 WO
WO 2012/138660 October 2012 WO
WO 2013/106390 July 2013 WO
WO 2013/155310 July 2013 WO
WO 2013/184604 December 2013 WO
Other references
  • AC-3 digital audio compression standard, Extract, Dec. 20, 1995, 11 pgs.
  • ActiveVideo Networks BV, International Preliminary Report on Patentability, PCT/NL2011/050308, Sep. 6, 2011, 8 pgs.
  • ActiveVideo Networks BV, International Search Report and Written Opinion, PCT/NL2011/050308, Sep. 6, 2011, 8 pgs.
  • Activevideo Networks Inc., International Preliminary Report on Patentability, PCT/US2011/056355, Apr. 16, 2013, 4 pgs.
  • ActiveVideo Networks Inc., International Preliminary Report on Patentability, PCT/US2012/032010, Oct. 8, 2013, 4 pgs.
  • ActiveVideo Networks Inc., International Search Report and Written Opinion, PCT/US2011/056355, Apr. 13, 2012, 6 pgs.
  • ActiveVideo Networks Inc., International Search Report and Written Opinion, PCT/US2012/032010, Oct. 10, 2012, 6 pgs.
  • ActiveVideo Networks Inc., International Search Report and Written Opinion, PCT/US2013/020769, May 5, 2013, 9 pgs.
  • ActiveVideo Networks Inc., International Search Report and Written Opinion, PCT/US2013/036182, Jul. 29, 2013, 12 pgs.
  • ActiveVideo Networks, Inc., International Search Report and Written Opinion, PCT/US2009/032457, Jul. 22, 2009, 7 pgs.
  • AcitveVideo Networks Inc., Korean Intellectual Property Office, International Search Report; PCT/US2009/032457, Jul. 22, 2009, 7 pgs.
  • Annex C—Video buffering verifier, information technology—generic coding of moving pictures and associated audio information: video, Feb. 2000, 6 pgs.
  • Antonoff, Michael, “Interactive Television,” Popular Science, Nov. 1992, 12 pages.
  • Broadhead, Direct manipulation of MPEG compressed digital audio, Nov. 5-9, 1995, 41 pgs.
  • Cable Television Laboratories, Inc., “CableLabs Asset Distribution Interface Specification, Version 1.1”, May 5, 2006, 33 pgs.
  • CD 11172-3, Coding of moving pictures and associated audio for digital storage media at up to about 1.5 MBIT, Jan. 1, 1992, 39 pgs.
  • Craig, Notice of Allowance, U.S. Appl. No. 11/178,176, Dec. 23, 2010, 8 pgs.
  • Craig, Notice of Allowance, U.S. Appl. No. 11/178,193, Jan. 12, 2012, 7 pgs.
  • Craig, Notice of Allowance, U.S. Appl. No. 11/178,183, Jul. 19, 2012, 8 pgs.
  • Craig, Notice of Allowance, U.S. Appl. No. 11/178,189, Oct. 12, 2011, 7 pgs.
  • Craig, Notice of Allowance, U.S. Appl. No. 11/178,176, Mar. 23, 2011, 8 pgs.
  • Craig, Notice of Allowance, U.S. Appl. No. 13/609,183, Aug. 26, 2013, 8 pgs.
  • Craig, Final Office Action, U.S. Appl. No. 11/103,838, Feb. 5, 2009, 30 pgs.
  • Craig, Final Office Action, U.S. Appl. No. 11/178,181, Aug. 25, 2010, 17 pgs.
  • Craig, Final Office Action, U.S. Appl. No. 11/103,838, Jul. 6, 2010, 35 pgs.
  • Craig, Final Office Action, U.S. Appl. No. 11/178,176, Oct. 1, 2010, 8 pgs.
  • Craig, Final Office Action, U.S. Appl. No. 11/178,183, Apr. 13, 2011, 16 pgs.
  • Craig, Final Office Action, U.S. Appl. No. 11/178,177, Oct. 26, 2010, 12 pgs.
  • Craig, Final Office Action, U.S. Appl. No. 11/178,181, Jun. 20, 2011, 21 pgs.
  • Craig, Office Action, U.S. Appl. No. 11/103,838, May 12, 2009, 32 pgs.
  • Craig, Office Action, U.S. Appl. No. 11/103,838, Aug. 19, 2008, 17 pgs.
  • Craig, Office Action, U.S. Appl. No. 11/103,838, Nov. 19, 2009, 34 pgs.
  • Craig, Office Action, U.S. Appl. No. 11/178,176, May 6, 2010, 7 pgs.
  • Craig, Office-Action U.S. Appl. No. 11/178,177, Mar. 29, 2011, 15 pgs.
  • Craig, Office Action, U.S. Appl. No. 11/178,177, Aug. 3, 2011, 26 pgs.
  • Craig, Office Action, U.S. Appl. No. 11/178,177, Mar. 29, 2010, 11 pgs.
  • Craig, Office Action, U.S. Appl. No. 11/178,181, Feb. 11, 2011, 19 pgs.
  • Craig, Office Action, U.S. Appl. No. 11/178,181, Mar. 29, 2010, 10 pgs.
  • Craig, Office Action, U.S. Appl. No. 11/178,182, Feb. 23, 2010, 15 pgs.
  • Craig, Office Action, U.S. Appl. No. 11/178,183, Dec. 6, 2010, 12 pgs.
  • Craig, Office Action, U.S. Appl. No. 11/178,183, Sep. 15, 20122, 12 pgs.
  • Craig, Office Action, U.S. Appl. No. 11/178,183, Feb. 19, 2010, 17 pgs.
  • Craig, Office Action, U.S. Appl. No. 11/178,183, Jul. 23, 2009, 13 pgs.
  • Craig, Office Action, U.S. Appl. No. 11/178,189, Nov. 9, 2010, 13 pgs.
  • Craig, Office Action, U.S. Appl. No. 11/178,189, Mar. 15, 2010, 11 pgs.
  • Craig, Office Action, U.S. Appl. No. 11/178,189, Jul. 23, 2009, 10 pgs.
  • Craig, Office Action, U.S. Appl. No. 11/178,189, May 26, 2011, 14 pgs.
  • Craig, Office Action, U.S. Appl. No. 13/609,183, May 9, 2013, 7 pgs.
  • Pavlovskaia, Office Action, JP 2011-516499, Feb. 14, 2014, 19 pgs.
  • Digital Audio Compression Standard(AC-3, E-AC-3), Advanced Television Systems Committee, Jun. 14, 2005, 236 pgs.
  • European Patent Office, Extended European Search Report for International Application No. PCT/US2010/027724, dated Jul. 24, 2012, 11 pages.
  • FFMPEG, http://www.ffmpeg.org, downloaded Apr. 8, 2010, 8 pgs.
  • Herr, Notice of Allowance, U.S. Appl. No. 11/620,593, May 23, 2012, 5 pgs.
  • Herr, Notice of Allowance, U.S. Appl. No. 12/534,016, Feb. 7,2012, 5 pgs.
  • Herr, Notice of Allowance, U.S. Appl. No. 12/534,016, Sep. 28, 2011, 15 pgs.
  • Herr, Final Office Action, U.S. Appl. No. 11/620,593, Sep. 15, 2011, 104 pgs.
  • Herr, Office Action, U.S. Appl. No. 11/620,593, Mar. 19, 2010, 58 pgs.
  • Herr, Office Action, U.S. Appl. No. 11/620,593, Apr. 21, 2009, 27 pgs.
  • Herr, Office Action, U.S. Appl. No. 11/620,593, Dec. 23, 2009, 58 pgs.
  • Herr, Office Action, U.S. Appl. No. 11/620,593, Jan. 24, 2011, 96 pgs.
  • Herr, Office Action, U.S. Appl. No. 11/620,593, Aug. 27, 2010, 41 pgs.
  • Herre, Thoughts on an SAOC Architecture, Oct. 2006, 9 pgs.
  • Hoarty, The Smart Headend—A Novel Approach to Interactive Television, Montreux Int'l TV Symposium, Jun. 9, 1995, 21 pgs.
  • ICTV, Inc., International Preliminary Report on Patentability, PCT/US2006/022585, Jan. 29, 2008, 9 pgs.
  • ICTV, Inc., International Search Report / Written Opinion, PCT/US2006/022585, Oct. 12, 2007, 15 pgs.
  • ICTV, Inc., International Search Report / Written Opinion, PCT/US2008/000419, May 15, 2009, 20 pgs.
  • ICTV, Inc., International Search Report / Written Opinion; PCT/US2006/022533, Nov. 20, 2006; 8 pgs.
  • Isovic, Timing constraints of MPEG-2 decoding for high quality video: misconceptions and realistic assumptions, Jul. 2-4, 2003, 10 pgs.
  • MPEG-2 Video elementary stream supplemental information, Dec. 1999, 12 pgs.
  • Ozer, Video Compositing 101. available from http://www.emedialive.com, Jun. 2, 2004, 5pgs.
  • Porter, Compositing Digital Images, 18 Computer Graphics (No. 3), Jul. 1984, pp. 253-259.
  • RSS Advisory Board, “RSS 2.0 Specification”, published Oct. 15, 2007. Not Found.
  • SAOC use cases, draft requirements and architecture, Oct. 2006, 16 pgs.
  • Sigmon, Final Office Action, U.S. Appl. No. 11/258,602, Feb. 23, 2009, 15 pgs.
  • Sigmon, Office Action, U.S. Appl. No. 11/258,602, Sep. 2, 2008, 12 pgs.
  • TAG Networks, Inc., Communication pursuant to Article 94(3) EPC, European Patent Application, 06773714.8, May 6, 2009, 3 pgs.
  • TAG Networks Inc, Decision to Grant a Patent, JP 209-544985, Jun. 28, 2013, 1 pg.
  • TAG Networks Inc., IPRP, PCT/US2006/010080, Oct. 16, 2007, 6 pgs.
  • TAG Networks Inc., IPRP, PCT/US2006/024194, Jan. 10, 2008, 7 pgs.
  • TAG Networks Inc., IPRP, PCT/US2006/024195, Apr. 1, 2009, 11 pgs.
  • TAG Networks Inc., IPRP, PCT/US2006/024196, Jan. 10, 2008, 6 pgs.
  • TAG Networks Inc. International Search Report, PCT/US2008/050221, Jun. 12, 2008, 9 pgs.
  • TAG Networks Inc. Office Action, CN 200680017662.3, Apr. 26, 2010, 4 pgs.
  • TAG Networks Inc. Office Action, EP 06739032.8, Aug. 14, 2009, 4 pgs.
  • TAG Networks Inc. Office Action, EP 06773714.8, May 6, 2009, 3 pgs.
  • TAG Networks Inc., Office Action, EP 06773714.8, Jan. 12, 2010, 4 pgs.
  • TAG Networks Inc., Office Action, JP 2008-506474, Oct. 10, 2012, 5 pgs.
  • TAG Networks Inc., Office Action, JP 2008-506474, Aug. 8, 2011, 5 pgs.
  • TAG Networks Inc., Office Action, JP 2008-520254, Oct. 20, 2011, 2 pgs.
  • TAG Networks, IPRP, PCT/US2008/050221, Jul. 7, 2009, 6 pgs.
  • TAG Networks, International Search Report, PCT/US2010/041133, Oct. 19, 2010, 13 pgs.
  • TAG Networks, Office Action, CM 200880001325.4, Jun. 22, 2011, 4 pgs.
  • TAG Networks, Office Action, JP 2009-544985, Feb. 25, 2013, 3 pgs.
  • Talley, A general framework for continuous media transmission control, Oct. 13-16, 1997, 10 pgs.
  • Todd, AC-3: flexible perceptual coding for audio transmission and storage, Feb. 26-Mar. 1, 1994, 16 pgs.
  • Tudor, MPEG-2 Video Compression, Dec. 1995, 15 pgs.
  • TVHEAD, Inc., First Examination Report, IN 1744/MUMNP/2007, Dec. 30, 2013, 6 pgs.
  • TVHEAD, Inc., International Search Report, PCT/US2006/010080, Jun. 20, 2006, 3 pgs.
  • TVHEAD, Inc., International Search Report, PCT/US2006/024194, Dec. 15, 2006, 4 pgs.
  • TVHEAD, Inc., International Search Report, PCT/US2006/024195, Nov. 29, 2006, 9 pgs.
  • TVHEAD, Inc., International Search Report, PCT/US2006/024196, Dec. 11, 2006, 4 pgs.
  • TVHEAD, Inc., International Search Report, PCT/US2006/024197, Nov. 28, 2006, 9 pgs.
  • Vernon, Dolby digital: audio coding for digital television and storage applications, Aug. 1999, 18 pgs.
  • Wang, A beat-pattern based error concealment scheme for music delivery with burst packet loss, Aug. 22-25, 2001, 4 pgs.
  • Wang, A compressed domain beat detector using MP3 audio bitstream, Sep. 30-Oct. 5, 2001, 9 pgs.
  • Wang, A multichannel audio coding algorithm for inter-channel redundancy removal, May 12-15, 2001, 6 pgs.
  • Wang, An excitation level based psychoacoustic model for audio compression, Oct. 30-Nov. 4, 1999, 4 pgs.
  • Wang, Energy compaction property of the MDCT in comparison with other transforms, Sep. 22-25, 2000, 23 pgs.
  • Wang, Exploiting excess masking for audio compression, Sep. 2-5, 1999, 4 pgs.
  • Wang, schemes for re-compressing mp3 audio bitstreams,Nov. 30-Dec. 3, 2001, 5 pgs.
  • Wang, Selected advances in audio compression and compressed domain processing, Aug. 2001, 68 pgs.
  • Wang, The impact of the relationship between MDCT and DFT on audio compression, Dec. 13-15, 2000, 9 pgs.
  • ActiveVideo Networks Inc., International Preliminary Report on Patentability, PCT/US2013/020769, Jul. 24, 2014, 6 pgs.
  • ActiveVideo Networks Inc., International Search Report and Written Opinion, PCT/US2014/030773, Jul. 25, 2014, 8 pgs.
  • ActiveVideo Networks Inc., International Search Report and Written Opinion, PCT/US2014/041416, Aug. 27, 2014, 8 pgs.
  • ActiveVideo Networks Inc. Communication Pursuant to Rules 70(2) and 70a(2), EP10841764.3, Jun. 6, 2014, 1 pg.
  • ActiveVideo Networks Inc. Communication Pursuant to Article 94(3) EPC, EP08713106.6-1908, Jun. 25, 2014, 5 pgs.
  • ActiveVideo Networks Inc. Communication Pursuant to Article 94(3) EPC, EP08713106.6-2223, May 10, 2011, 7 pgs.
  • ActiveVideo Networks Inc. Communication Pursuant to Article 94(3) EPC, EP09713486.0, Apr. 14, 2014, 6 pgS.
  • ActiveVideo Networks Inc. Communication Pursuant to Rules 70(2) and 70a(2), EP11833486.1, Apr. 24, 2014, 1 pg.
  • ActiveVideo Networks Inc. Examination Report No. 1, AU2011258972, Apr. 4, 2013, 5 pgs.
  • ActiveVideo Networks Inc. Examination Report No. 1, AU2010339376, Apr. 30, 2014, 4 pgs.
  • ActiveVideo Networks Inc. Examination Report, App. No. EP11749946.7, Oct. 8, 2013, 6 pgs.
  • ActiveVideo Networks Inc. Summons to attend oral-proceeding, Application No. EP09820936-4, Aug. 19, 2014, 4 pgs.
  • ActiveVideo Networks Inc., International Searching Authority, International Search Report—International application No. PCT/US2010/027724, dated Oct. 28, 2010, together with the Written Opinion of the International Searching Authority, 7 pages.
  • Adams, Jerry, NTZ Nachrichtechnische Zeitschrift. vol. 40, No. 7, Jul. 1987, Berlin DE pp. 534- 36; Jerry Adams: ‘Glasfasernetz für Breitbanddienste in London’, 5 pgs. No English Translation Found.
  • Avinity Systems B.V., Communication pursuant to Article 94(3) EPC, EP 07834561.8, Jan. 31, 2014, 10 pgs.
  • Avinity Systems B.V., Communication pursuant to Article 94(3) EPC, EP 07834561.8, Apr. 89, 2010, 5 pgs.
  • Avinity Systems B.V., International Preliminary Report on Patentability, PCT/NL2007/000245, Mar. 31, 2009, 12 pgs.
  • Avinity Systems B.V., International Search Report and Written Opinion, PCT/NL2007/000245, Feb. 19, 2009, 18 pgs.
  • Avinity Systems B.V., Notice of Grounds of Rejection for Patent, JP 2009-530298, Sep. 3, 2013, 4 pgs.
  • Avinity Systems B.V., Notice of Grounds of Rejection for Patent, JP 2009-530298, Sep. 25, 2012, 6 pgs.
  • Bird et al., “Customer Access to Broadband Services,” ISSLS 86—The International Symposium on Subrscriber Loops and Services Sep. 29, 1986, Tokyo,JP 6 pgs.
  • Brockmann, Final Office Action, U.S. Appl. No. 12/443,571, Mar. 7, 2014, 21 pgs.
  • Brockmann, Final Office Action, U.S. Appl. No. 13/668,004, Jul. 16, 2014, 20 pgs.
  • Brockmann, Final Office Action, U.S. Appl. No. 13/438,617, Oct. 3, 2014, 19 pgs.
  • Brockmann, Office Action, U.S. Appl. No. 13/686,548, Mar. 10, 2014, 11 pgs.
  • Brockmann, Office Action, U.S. Appl. No. 13/668,004, Dec. 23, 2013, 9 pgs.
  • Brockmann, Office Action, U.S. Appl. No. 13/438,617, May 12, 2014, 17 pgs.
  • Brockmann, Office Action, U.S. Appl. No. 12/443,571, Jun. 5, 2013, 18 pgs.
  • Chang, Shih-Fu, et al., “Manipulation and Compositing of MC-DOT Compressed Video,” IEEE Journal on Selected Areas of Communications, Jan. 1995, vol. 13, No. 1, 11 pgs. Best Copy Available.
  • Dahlby, Office Action, U.S. Appl. No. 12/651,203, Jun. 5, 2014, 18 pgs.
  • Dahlby, Final Office Action, U.S. Appl. No. 12/651,203, Feb. 4, 2013, 18 pgs.
  • Dahlby, Office Action, U.S. Appl. No. 12/651,203, Aug. 16, 2012, 18 pgs.
  • Dukes, Stephen D., “Photonics for cable television system design, Migrating to regional hubs and passive networks,” Communications Engineering and Design, May 1992, 4 pgs.
  • Ellis, et al., “INDAX: An Operation Interactive Cabletext System”, IEEE Journal on Selected Areas in Communications, vol. sac-1, No. 2, Feb. 1983, pp. 285-294.
  • European Patent Office, Supplementary European Search Report, Application No. EP 09 70 8211, dated Jan. 5, 2011, 6 pgs.
  • Frezza, W., “The Broadband Solution-Metropolitan CATV Networks,” Proceedings of Videotex '84, Apr. 1984, 15 pgs.
  • Gordon, Notice of Allowance, U.S. Appl. No. 12/008,697, Mar. 20, 2014, 10 pgs.
  • Gordon, Final Office Action, U.S. Appl. No. 12/008,722, Mar. 30, 2012, 16 pgs.
  • Gordon, Final Office Action, U.S. Appl. No. 12/035,236, Jun. 11, 2014, 14 pgs.
  • Gordon, Final Office Action, U.S. Appl. No. 12/035,236, Jul. 22, 2013, 7 pgs.
  • Gordon, Final Office Action, U.S. Appl. No. 12/035,236, Sep. 20, 2011, 8 pgs.
  • Gordon, Final Office Action, U.S. Appl. No. 12/035,236, Sep. 21, 2012, 9 pgs.
  • Gordon, Final Office Action, U.S. Appl. No. 12/008,697, Mar. 6, 2012, 48 pgs.
  • Gordon, Office Action, U.S. Appl. No. 12/035,236, Mar. 13, 2013, 9 pgs.
  • Gordon, Office Action, U.S. Appl. No. 12/035,236, Mar. 22, 2011, 8 pgs.
  • Gordon, Office Action, U.S. Appl. No. 12/035,236, Mar. 28, 2012, 8 pgs.
  • Gordon, Office Action, U.S. Appl. No. 12/035,236, Dec. 16, 2013, 11 pgs.
  • Gordon, Office Action, U.S. Appl. No. 12/008,697, Aug. 1, 2013, 43 pgs.
  • Gordon, Office Action, U.S. Appl. No. 12/008,697, Aug. 4, 2011, 39 pgs.
  • Gordon, Office Action, U.S. Appl. No. 12/008,722, Oct. 11, 2011, 16 pgs.
  • Handley et al, “TCP Congestion Window Validation,” RFC 2861, Jun. 2000, Network Working Group, 22 pgs.
  • Henry et al. “Multidimensional Icons” ACM Transactions on Graphics, vol. 9, No. 1 Jan. 1990, 5 pgs.
  • Isensee et al., “Focus Highlight for World Wide Web Frames,” Nov. 1, 1997, IBM Technical Disclosure Bulletin, vol. 40, No. 11, pp. 89-90.
  • ICTV, Inc., International Search Report / Written Opinion, PCT/US2008/000400, Jul. 14, 2009, 10 pgs.
  • ICTV, Inc., International Search Report / Written Opinion, PCT/US2008/000450, Jan. 26, 2009, 9 pgs.
  • Kato, Y., et al., “A Coding Control algorithm for Motion Picture Coding Accomplishing Optimal Assignment of Coding Distortion to Time and Space Domains,” Electronics and Communications in Japan, Part 1, vol. 72, No. 9, 1989, 11 pgs.
  • Koenen, Rob,“MPEG-4 Overview—Overview of the MPEG-4 Standard” Internet Citation, Mar. 2001, http://mpeg.telecomitalialab.com/standards/mpeg-4/mpeg-4.htm, May 9, 2002, 74 pgs.
  • Konaka, M. et al., “Development of Sleeper Cabin Cold Storage Type Cooling System,” SAE International, The Engineering Society for Advancing Mobility Land Sea Air and Space, SAE 2000 World Congress, Detroit, Michigan, Mar. 6-9,2000, 7 pgs.
  • Le Gall, Didier, “MPEG: A Video Compression Standard for Multimedia Applications”, Communication of the ACM, vol. 34, No. 4, Apr. 1991, New York, NY, 13 pgs.
  • Langenberg, E, et al., “Integrating Entertainment and Voice on the Cable Network,” SCTE , Conference on Emerging Technologies, Jan. 6-7, 1993, New Orleans, Louisiana, 9 pgs.
  • Large, D., “Tapped Fiber vs. Fiber-Reinforced Coaxial CATV Systems”, IEEE LCS Magazine, Feb. 1990, 7 pgs. Best Copy Available.
  • “MSDL Specification Version 1.1” International Organisation for Standardisation Organisation Internationale EE Normalisation, ISO/IEC JTC1/SC29/WG11 Coding of Moving Pictures and Autdio, N1246, MPEG96/Mar. 1996, 101 pgs.
  • Noguchi, Yoshihiro, et al., “MPEG Video Compositing in the Compressed Domain,” IEEE International Symposium on Circuits and Systems, vol. 2, May 1, 1996, 4 pgs.
  • Regis, Notice of Allowance U.S. Appl. No. 13/273,803, Sep. 2, 2014, 8 pgs.
  • Regis, Notice of Allowance U.S. Appl. No. 13/273,803, May 14, 2014, 8 pgs.
  • Regis, Final Office Action U.S. Appl. No. 13/273,803, Oct. 11, 2013, 23 pgs.
  • Regis, Office Action U.S. Appl. No. 13/273,803, Mar. 27, 2013, 32 pgs.
  • Rose, K., “Design of a Switched Broad-Band Communications Network for Interactive Services,” IEEE Transactions on Communications, vol. com-23, No. 1, Jan. 1975, 7 pgs.
  • Saadawi, Tarek N., “Distributed Switching for Data Transmission over Two-Way CATV”, IEEE Journal on Selected Areas in Communications, vol. Sac-3, No. 2, Mar. 1985, 7 pgs.
  • Schrock, “Proposal for a Hub Controlled Cable Television System Using Optical Fiber,” IEEE Transactions on Cable Television, vol. CATV-4, No. 2, Apr. 1979, 8 pgs.
  • Sigmon, Notice of Allowance, U.S. Appl. No. 13/311,203, Sep. 22, 2014, 5 pgs.
  • Sigmon, Notice of Allowance, U.S. Appl. No. 13/311,203, Feb. 27, 2014, 14 pgs.
  • Sigmon, Final Office Action, U.S. Appl. No. 13/311,203, Sep. 13, 2013, 20 pgs.
  • Sigmon, Office Action, U.S. Appl. No. 13/311,203, May 10, 2013, 21 pgs.
  • Smith, Brian C., et al., “Algorithms for Manipulating Compressed Images,” IEEE Computer Graphics and Applications, vol. 13, No. 5, Sep. 1, 1993, 9 pgs.
  • Smith, J. et al., “Transcoding Internet Content for Heterogeneous Client Devices” Circuits and Systems, 1998. ISCAS '98. Proceedings of the 1998 IEEE International Symposium on Monterey, CA, USA May 31-Jun. 3, 1998, New York, NY, USA,IEEE, US, May 31, 1998, 4 pgs.
  • Stoll, G. et al., “GMF4iTV: Neue Wege zur-lnteraktivitaet Mit Bewegten Objekten Beim Digitalen Fernsehen,” Fkt Fernseh Und Kinotechnik, Fachverlag Schiele & Schon GmbH, Berlin, DE, vol. 60, No. 4, Jan. 1, 2006, ISSN: 1430-9947, 9 pgs. No English Translation Found.
  • Tamitani et al., “An Encoder/Decoder Chip Set for the MPEG Video Standard,” 1992 IEEE International Conference on Acoustics, vol. 5, Mar. 1992, San Francisco, CA, 4 pgs.
  • Terry, Jack, “Alternative Technologies and Delivery Systems for Broadband ISDN Access”, IEEE Communications Magazine, Aug. 1992, 7 pgs.
  • Tobagi, Fouad A., “Multiaccess Protocols in Packet Communication Systems,” IEEE Transactions on Communications, vol. Com-28, No. 4, Apr. 1980, 21 pgs.
  • Toms, N., “An Integrated Network Using Fiber Optics (Info) for the Distribution of Video, Data, and Telephone in Rural Areas,” IEEE Transactions on Communication, vol. Com-26, No. 7, Jul. 1978, 9 pgs.
  • Jurgen—Two-way applications for cable television systems in the '70s, IEEE Spectrum, Nov. 1971, 16 pgs.
  • va Beek, P., “Delay-Constrained Rate Adaptation for Robust Video Transmission over HomeNetworks,” Image Processing, 2005, ICIP 2005, IEEE International Conference, Sep. 2005, vol. 2, No. 11, 4 pgs.
  • Welzenbach et al., “The Application of Optical Systems for Cable TV,” AEG-Telefunken, Backnang, Federal Republic of Germany, ISSLS Sep. 15-19, 1980, Proceedings IEEE Cat. No. 80 CH1565-1, 7 pgs.
  • Yum, TS P., “Hierarchical Distribution of Video with Dynamic Port Allocation,” IEEE Transactions on Communications, vol. 39, No. 8, Aug. 1, 1991, XP000264287, 7 pgs.
  • ActiveVideo, International Search Report and Written Opinion, PCT/US2015/027803, Jun. 24, 2015, 18 pgs.
  • ActiveVideo, International Search Report and Written Opinion, PCT/US2015/027804, Jun. 25, 2015, 10 pgs.
  • ActiveVideo Networks, Inc., International Preliminary Report on Patentablity, PCT/US2013/036182, Oct. 14, 2014, 9 pgs.
  • ActiveVideo Networks, Inc., International Search Report and Written Opinion, PCT/US2014/041430, Oct. 9, 2014, 9 pgs.
  • ActiveVideo Networks Inc., Communication Pursuant to Rule 94(3), EP08713106-6, Jun. 25, 2014, 5 pgs.
  • ActiveVideo Networks Inc., Communication Pursuant to Rule 94(3), EP09713486.0, Apr. 14, 2014, 6 pgs.
  • ActiveVideo Networks Inc., Communication Pursuant to Rules 70(2) and 70a(2), EP11833486.1, Apr. 24, 2014, 1 pg.
  • ActiveVideo Networks Inc., Communication Pursuant to Rules 161(2) & 162 EPC, EP13775121.0, Jan. 20, 2015, 3 pgs.
  • ActiveVideo Networks Inc., Communication Pursuant to Article 94(3) EPC, 10754084.1, Feb. 10, 2015, 12 pgs.
  • ActiveVideo Networks Inc., Communication under Rule 71(3) EPC, Intention to Grant, EP08713106.6, Feb. 19, 2015, 12 pgs.
  • ActiveVideo Networks Inc., Decision to refuse a European patent application (Art. 97(2) EPC, EP09820936.4, Feb. 20, 2015, 4 pgs.
  • ActiveVideo Networks Inc., Examination Report No. 2, AU2011249132, May 29, 2015, 4 pgs.
  • Activevideo Networks Inc., Examination Report No. 2, AU2011315950, Jun. 25, 2015, 3 pgs.
  • ActiveVideo Networks Inc., Examination Report No. 1, AU2011258972, Jul. 21, 2014, 3 pgs.
  • ActiveVideo Networks B.V., Office Action, IL222830, Jun. 28, 2015, 7 pgs.
  • ActiveVideo Networks, Inc., Office Action, JP2013534034, Jun. 16, 2015, 6 pgs.
  • Active Video Networks, Notice of Reasons for Rejection, JP2012-547318, Sep. 26, 2014, 7 pgs.
  • ActiveVideo Networks Inc., Notice of Reasons for Rejection, JP2014-100460, Jan. 15, 2015, 6 pgs.
  • ActiveVideo Networks Inc., Notice of Reasons for Rejection, JP2013-509016, Dec. 24, 2014, 11 pgs.
  • ActiveVideo Networks Inc., Certificate of Patent JP5675765, Jan. 9, 2015, 3 pgs.
  • Avinity Systems B. V., Final Office Action, JP-2009-530298, Oct. 7, 2014, 8 pgs.
  • Avinity Systems Bv, PreTrial Reexam Report JP2009530298, Apr. 24, 2015, 6 pgs.
  • Brockmann, Notice of Allowance, U.S. Appl. No. 13/445,104, Dec. 24, 2014, 14 pgs.
  • Brockmann, Notice of Allowance, U.S. Appl. No. 13/438,617, May 22, 2015, 18 pgs.
  • Brockmann, Notice of Allowance, U.S. Appl. No. 13/445,104, Apr. 23, 2015, 8 pgs.
  • Brockmann, Notice of Allowance, U.S. Appl. No. 14/298,796, Mar. 18, 2015, 11 pgs.
  • Brockmann, Final Office Action, U.S. Appl. No. 12/443,571, Jul. 9, 2015, 28 pgs.
  • Brockmann, Final Office Action, U.S. Appl. No. 13/686,548, Sep. 24, 2014, 13 pgs.
  • Brockmann, Office Action, U.S. Appl. No. 12/443,571, Nov. 5, 2014, 26 pgs.
  • Brockmann, Office Action, U.S. Appl. No. 13/668,004, Feb. 26, 2015, 17 pgs.
  • Brockmann, Office Action, U.S. Appl. No. 13/686,548, Jan. 5, 2015, 12 pgs.
  • Brockmann, Office Action, U.S. Appl. No. 14/262,674, May 21, 2015, 7 pgs.
  • Brockmann, Office Action, U.S. Appl. No. 13/737,097, Mar. 16, 2015, 18 pgs.
  • Craig, Decision on Appeal—Reversed—, U.S. Appl. No. 11/178,177, Feb. 25, 2015, 7 pgs.
  • Craig, Notice of Allowance, U.S. Appl. No. 11/178,177, Mar. 5, 2015, 7 pgs.
  • Craig, Notice of Allowance, U.S. Appl. No. 11/178,181, Feb. 13, 2015, 8 pgs.
  • Dahlby, Office Action U.S. Appl. No. 12/651,203, Jul. 2, 2015, 25 pgs.
  • Dahlby, Office Action, U.S. Appl. No. 12/651,203, Dec. 3, 2014, 19 pgs.
  • Gecsei, J., “Adaptation in Distributed Multimedia Systems,” IEEE Multimedia, IEEE Service Center, New York, NY, vol. 4, No. 2, Apr. 1, 1997, 10 pgs.
  • Gordon, Notice of Allowance, U.S. Appl. No. 12/008,697, Apr. 1, 2015, 10 pgs.
  • Gordon, Notice of Allowance, U.S. Appl. No. 12/008,697, Dec. 8, 2014, 10 pgs.
  • Gordon, Final Office Action, U.S. Appl. No. 12/008,722, Jul. 2, 2015, 20 pgs.
  • Gordon, Office Action, U.S. Appl. No. 12/008,722, Nov. 28, 2014, 18 pgs.
  • Ohta, K., et al., “Selective Multimedia Access Protocol for Wireless Multimedia Communication,” Communications, Computers and Signal Processing, 1997, IEEE Pacific Rim Conference NCE Victoria, BC, Canada, Aug. 1997, vol. 1, 4 pgs.
  • Regis, Notice of Allowance, U.S. Appl. No. 13/273,803, Nov. 18, 2014, 9 pgs.
  • Regis, Notice of Allowance, U.S. Appl. No. 13/273,803, Mar. 2, 2015, 8 pgs.
  • Sigmon, Notice of Allowance, U.S. Appl. No. 13/311,203, Apr. 14, 2015, 5 pgs.
  • Sigmon, Notice of Allowance, U.S. Appl. No. 13/311,203, Dec. 19, 2014, 5 pgs.
  • TAG Networks Inc, Decision to Grant a Patent, JP 2008-506474, Oct. 4, 2013, 5 pgs.
  • Wei, S., “QoS Tradeoffs Using an Application-Oriented Transport Protocol (AOTP) for Multimedia Applications Over IP.” Sep. 23-26 1999, Proceedings of the Third International Conference on Computational Intelligence and Multimedia Applications, New Delhi, India, 5 pgs.
  • ActiveVideo Networks, Inc. Decision to Grant, EP08713106.6-1908, Jul. 9, 2015, 2 pgs.
  • ActiveVideo Networks, Inc. Certificate of Grant, EP08713106.-1908, Aug. 5, 2015, 1 pgs.
  • ActiveVideo Networks, Inc. Decision to Grant, JP2014100460, Jul. 24, 2015, 5 pgs.
  • ActiveVideo Networks, Inc. KIPO'S Notice of Preliminary Rejection, KR10-2010-7019512, Jul. 14, 2015, 15 pgs.
  • ActiveVideo Networks, Inc. KIPO'S Notice of Preliminary Rejection, KR10-20107021116, Jul. 13, 2015, 19 pgs.
  • ActiveVideo Networks, Inc., International Search Report and Written Opinion, PCT-US2015028072, Aug. 7, 2015, 9 pgs.
  • Brockmann, Final Office Action, U.S. Appl. No. 13/668,004, Aug. 3, 2015, 13 pgs.
  • Brockmann, Final Office Action, U.S. Appl. No. 13/686,548, Aug. 12, 2015, 13 pgs.
  • Brockmann, Final Office Action, U.S. Appl. No. 13/737,097, Aug. 14, 2015, 17 pgs.
Patent History
Patent number: 9219922
Type: Grant
Filed: Jun 6, 2013
Date of Patent: Dec 22, 2015
Patent Publication Number: 20140362086
Assignee: ACTIVEVIDEO NETWORKS, INC. (San Jose, CA)
Inventors: Ronald Alexander Brockmann (Utrecht), Anuj Dev (Amsterdam), Maarten Hoeben (Amersfoort)
Primary Examiner: Jose Couso
Application Number: 13/911,948
Classifications
Current U.S. Class: Graph Generating (345/440)
International Classification: G06K 9/36 (20060101); H04N 19/54 (20140101); H04N 19/56 (20140101);