Tagging virtualized content
Techniques for tagging virtualized content are disclosed. In some embodiments, a modeled three-dimensional scene of objects representing abstracted source content is generated and analyzed to determine a contextual characteristic of the scene that is based on a plurality of objects comprising the scene. The modeled scene is tagged with a tag specifying the determined contextual characteristic.
Latest Outward, Inc. Patents:
This application is a continuation of U.S. patent application Ser. No. 14/327,467, now U.S. Pat. No. 10,387,729, entitled TAGGING VIRTUALIZED CONTENT, filed Jul. 9, 2014, which claims priority to U.S. Provisional Patent Application No. 61/844,329, entitled RENDERING DIGITAL CONTENT, filed Jul. 9, 2013, both of which are incorporated herein by reference for all purposes.
BACKGROUND OF THE INVENTIONTagging of multi-media content has traditionally been mostly a manual process, limiting the types of tags and searchability of the multi-media content.
Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.
The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term ‘processor’ refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.
A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims, and the invention encompasses numerous alternatives, modifications, and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example, and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.
Typically, the quality of rendered content at a destination device is restricted by and does not exceed the quality of the source content despite the rendering capabilities of the destination device. However, the limitations of source content quality pose a growing challenge as rapid improvements occur in the hardware of destination devices. Poor quality source content is not suitable to serve the demands of HD (high definition) and emerging beyond-HD capable destination devices. Standard HD is quickly giving way to 2K HD, which will evolve into 4K HD. Moreover, hardware roadmaps already contemplate 8K HD, i.e., UHD (ultra high definition), which is approximately sixteen times the format size of standard HD.
Techniques for decoupling the format and quality of source content as well as the format and quality of the same content processed for storage and distribution from the rendering capabilities of a destination device are disclosed in detail herein. The disclosed multi-media technology platform facilitates flawless quality when rendering content from an arbitrary source on any of a plurality of types of destination devices by allowing performance up to the capability of the receiving system (including best-in-class display technologies) to be achieved. Thus, a high fidelity, cinematic quality viewing experience may be realized from any image or video source. The disclosed techniques not only allow users to create visually stunning image and video content anywhere, at any time, and on any device without having to be an expert in multi-media content creation tools but also allow users to re-imagine content in nearly unlimited ways. Although image or video source content is described in many of the given examples, the techniques disclosed herein may be extended to and employed with respect to any multi-media content.
The content is optionally augmented at step 208. In most cases, the content is augmented to include at least one or more elements or features that were never a part of the original content obtained at step 202 but are included to provide an enhanced viewing experience at a destination device. At step 210, the content is provided to a destination device for rendering. Since the computational space is agnostic to the destination device, a perfect match for the capabilities of any destination device can be achieved using process 200. Thus, content may be rendered at a destination device based on the native format of the destination device as well as in the best quality supported by the destination device. For example, using process 200, source content captured via a relatively low resolution mobile phone camera may be rendered at a high resolution destination device in cinematic quality. In some embodiments, the content augmentation of step 208 may be performed client-side at a destination device.
As described, process 200 facilitates a high fidelity viewing experience regardless of the format or quality of the source content and offers unprecedented variability within the actually rendered content. The disclosed platform does not comprise merely compressing, upscaling, and/or replicating the appearance of source content. Rather, information gathered from contextual clues from the source content is employed to virtualize or create a re-imagined version of the content. In most cases, information is being added to, not removed from, the content. That is, more information than what is present in the original source content is provided for rendering at an output device. In many cases, the virtualized or re-imagined version of the source content comprises very little, if any, of the data comprising the original source content. Moreover, the content may be re-imagined differently for different output devices, for example, based on the rendering capabilities of the devices and/or other criteria associated with the devices. A key objective of the disclosed approach is not to create a lossless version of the source content, although that may coincidentally be an outcome, but rather to prioritize creation of content best suited for the capabilities of a destination device and/or customized for the destination device or a user thereof.
Block 306 comprises a mapping engine configured to correlate and map the elements and/or characteristics identified by recognition engine 304 to the best matches available from an existing database 308 in which object specifications comprise prescribed computational space based representations. In some embodiments, mapping engine 306 is further configured to determine relative positions and orientations of identified objects within the model environment or scene representing the input source content. In many cases, objects in database 308 may need to be morphed or modified through available parameterization to match identified objects. In cases in which a suitably close match cannot be identified, an object may be substituted for by a best guess. Alternatively, an object and its associated properties may be created on an ad hoc basis from an analysis of source content 302 (e.g., by extracting geometry and texture) and submitted into database 308 for use during matching and mapping. In some cases, the correlation and/or mapping performed by mapping engine 306 is at least in part assisted by human input. In many cases, the environment or scene representing the input source content modeled by mapping engine 306 comprises little, if any, of the original data comprising input source content 302 since the modeled environment or scene comprises database object based representations of content. Thus, input source content 302 is effectively virtualized or re-imagined.
Block 308 comprises a database comprising an archive of available master objects and their associated properties. The depth and complexity of database 308 increases over time as it is populated as more and more data becomes available. An object entry in database 308 may comprise any appropriate and applicable parameters for specifying the object. For example, an object specification may include data comprising geometry (e.g., shape and size), surface texture, optical response to illumination, and/or any additional properties associated with the object (e.g., brand name, date and location of origin of manufacture, material properties, etc.). In some embodiments, object properties are parameterized to the extent possible to broaden the class of objects that can be represented by a given master object. Database 308 may be populated using any one or more appropriate techniques, some examples of which are depicted in
Block 316 comprises a content context modification engine configured to modify and/or customize the environment or scene modeled by mapping engine 306 according to desired performance criteria. For example, the modeled environment may be altered by modifying objects, substituting objects with alternatives, and/or removing objects altogether. Moreover, the lighting of the modeled environment (e.g., lighting type and position, luminance, chromaticity, etc.) may be modified. In addition, perceived performance errors in the modeled environment or scene may be appropriately corrected as applicable. In the case of human subjects (i.e., parameterized master objects representing people in the source content), for instance, problems in gaze, appearance, etc., may be corrected. Furthermore, alternative camera positions/perspectives may be introduced into the modeled environment, for example, in addition to or instead of a default virtual camera appropriately located in a position to recreate a perspective defined by the source content.
Block 318 represents providing or outputting the modeled environment, for example, to a (requesting) client or destination device at which the modeled environment is desired to be rendered. Instead of original source content 302 being provided, parameters defining objects comprising the model representing source content 302 are delivered. Such a model environment specification may include, for example, object identifications, relative positions, and orientations; background and lighting environment specifications; camera position; etc. In some embodiments, a subset of the object database needed to create the modeled scene is provided. Alternatively, a copy of the subset of the object database needed to create the modeled scene may already be available at the client site or destination device to which the model environment is delivered and thus need not be provided. In some embodiments, block 318 includes storing a specification of the modeled environment or scene, e.g., for later distribution.
Block 320 comprises a rendering engine configured to render the environment or scene defined by the model at an output device (e.g., a flat panel display) with the characteristics of the objects referenced to the properties provided by the object database. Best in class techniques from the computer graphics industry may be applied by rendering engine 320 when rendering a scene. The subset of the object database needed to render the scene may be accessed by rendering engine 320 through a local copy or through direct access to database 308. In many cases, the scene is rendered by rendering engine 320 with a quality that is matched to the maximum quality performance achievable by the output device, i.e., is optimally matched to the display format, color performance, frame rate, etc., of the output device. Moreover, the scene rendered by rendering engine 320 may be personalized or customized based on a user and/or profiling of a user of the output device.
The described techniques for content distribution based on context abstraction offer numerous benefits. A high quality viewing experience may be delivered regardless of the quality of the source content and even when the source content is significantly degraded, e.g., due to compression artifacts or other noise. In many cases, the source content only has to be of sufficient quality for recognition engine 304 to identify features, objects, and background types. For example, in some cases, six bits of color per pixel in the source content are sufficient rather than thirty bits. Moreover, compression based block artifacts are in most cases acceptable; thus, very high compression ratios via traditional methods may be used to store the source content.
A scene captured by a common user using a device with consumer grade sensors having limited dynamic range is not only of relatively low resolution/quality but also is typically filmed without proper lighting or environmental conditions. However, using the disclosed techniques, viewed content can still be very high quality. Once an object is recognized, local tone correction can be applied, and highlights saturating an image can be corrected naturally when the scene is relit. Similarly, any other perceived imperfections can be corrected in or removed from the scene. For example, image stabilization as well as post processing to correct for performance errors may be introduced as applicable. With respect to a human subject, for instance, gaze may be corrected, closed eyes may be opened, stray hair may be removed, etc.
In various embodiments, content may be edited and/or re-imagined with any number of alternative visual qualities (e.g., lighting, surface characteristics, camera perspective, etc.) than originally intended from the source, and new and/or different content may be introduced. For example, a scene may be re-rendered with an advertiser's content such as brands, images, products, etc. Moreover, content may be personalized or customized based on an end user. Since object models may be specified in 3D in database 308, corresponding content may be naturally rendered in 3D if desired. Conversely, content may be rendered as a cartoon (e.g., in anime style) if desired.
For many applications, the disclosed content distribution techniques significantly reduce required communication bandwidth, for example, compared to traditional video codec approaches. Ultra-low bandwidth is especially feasible with fairly static scenes since object motion is the only required data that needs to be communicated. For example, video conferencing applications (such as telepresence, video chat, mobile video calls, etc.) are particularly well-suited for the disclosed platform. Source content from a low quality webcam, for instance, may be rendered in a UHD enabled telepresence conference room in cinematic quality. In addition to providing an ultra-low bandwidth solution for many applications, the disclosed content distribution techniques facilitate cinematic quality content creation at very low cost and overhead and without requiring expertise by an end user. Existing content may be easily re-imagined and/or remastered in limitless ways. For example, any archived image or video source may be rendered beautifully with cinematic quality on a UHD display. Additional applicable fields in which the disclosed techniques may be especially relevant include applications in the gaming, entertainment, and education industries as well as in any other industry in which efficient content distribution is paramount.
The described content contextualization platform is furthermore amenable to search applications, particularly contextual based image and video searches. A search may be conducted by searching for something according to one or more object definition criteria. Alternatively, a search may be conducted by creating content that best matches the search criteria. Content may be created, for instance, by modifying something that is quite close already (e.g., by re-lighting, re-orienting, and/or color correction; object removal and/or editing; etc.). Searches may be efficiently conducted against a network of one or more databases (such as database 308) comprising content libraries of object assets.
The manner in which database objects are specified and stored in the described architecture moreover facilitates future proofing multi-media content libraries with respect to evolving display and viewing technologies and standards. Further, content generation in 2D and 3D is easily supported since many database objects are inherently specified in 3D. Additionally, multi-scale object databases may be linked, facilitating support of an “infinite zoom” feature allowing advanced zooming, possibly down to even micro-scale and nano-scale levels.
In addition to the aforementioned advantages and applications, market trends in display technology are anticipated to favor the disclosed content distribution techniques based on context abstraction. To accommodate emerging display technologies, a new paradigm for content distribution will be imperative. The disclosed techniques provide one such architecture that is not only viable and scalable but also economic and resource efficient.
At step 402, camera pose relative to a scene comprising the source content is determined. In various embodiments, a scene may be substantially static and/or may include varying degrees of motion corresponding to changing camera positions (e.g., if the camera is panned or otherwise moved to capture different angles or perspectives of the scene). From the camera motion, standard computer vision based feature tracking algorithms may be employed to compute the pose (i.e., the position of the camera) relative to the scene. In some embodiments, a sequence of camera poses is determined at step 402 for a camera cut shot, wherein a cut comprises a sequence involving a single continuous motion of the camera.
At step 404, key frames are defined. In the given context, a key frame comprises a frame in which critical features and/or objects of the scene are within the field of view of the camera. Such features and/or objects may leave or enter the field of view over time during a sequence. However, key frames may be selected to include most of the relevant features and/or objects needing tracking during a cut shot.
At step 406, objects are identified in each key frame. In some cases, object recognition is at least in part automated, e.g., for human faces, objects with regular geometries, etc. In some cases, object recognition is at least in part assisted by human intervention. In such cases, points in a frame or scene tracked by standard machine vision based algorithms are presented to a human operator for consideration. The human operator indicates or specifies a subset of tracked points that correspond to an object. Additional computer vision algorithms may be subsequently employed with respect to such a human operator identified subset of tracked points to segment a corresponding object from an image by determining the visual boundaries of the object with respect to other foreground and background content in the associated frame.
At step 408, each object identified at step 406 is mapped to a database object. In some cases, object mapping is at least in part automated. In some cases, object mapping is at least in part performed by a human operator. For example, a human operator may identify an object type or class (e.g., by traversing through a decision tree classifying different types of objects) and/or search against the database for an identified object class. In response to such a search, one or more similar objects may be presented to the operator for consideration. The operator may select one of the available database objects to represent a given object identified in the scene. Thus, an object from the source content is mostly, and in many cases completely, virtualized when matched and mapped to a corresponding database object, which is what is eventually used during rendering. In some embodiments, instead of seeking a best match, object mapping comprises purposely misinterpreting or mismatching an identified object when selecting a corresponding database object. For example, an identified cloud object in a source video may be mapped to a sun object in the database to reflect different weather.
In some embodiments, step 406 comprises identifying a partial object, and step 408 comprises mapping an identified partial object to a whole or complete database object. Objects and/or space comprising the source content may be expanded or augmented in the virtual environment or scene modeled from the source content as applicable. For example, a house corner identified at the edge of a frame in the source content may be mapped to a complete house object from the database. Thus, the virtual space comprising the modeled environment or scene may be much larger than the space encompassed by the original source content since in many cases it is desirable to render whole objects instead of partial objects, for example, to make the scene more visually appealing and amenable to interaction (e.g., via virtual user entry into the scene and change of camera position/perspective).
In some embodiments, to properly place an object in a model environment or scene representing the source content, at least a prescribed number of non-coplanar points are needed to specify the object. For example, an automated process and/or human operator may specify or select at least four non-coplanar tracked points when identifying an object at step 406. The points defining an identified object in a scene determined at step 406 are mapped to corresponding points on a selected database object to which the identified object is mapped at step 408. The relative match between the selected points in the source content and the selected points on the corresponding database object facilitates automatic rescaling and alignment of the 3D database object with the 2D view of the object in the source content. The match of orientation and position in the 2D view and the 3D object is in many cases approximate and may visually be confirmed for accuracy with the use of a semi-transparent overlay of the source image onto the camera view of the estimated 3D environment. Although placement may be completely automated, in some cases, adjustments in the placement may be made by a human operator if necessary and depending on the accuracy of the automated placement. In some cases, an object may be auto-adjusted to snap its bottom most surface to a surface it should be resting on (e.g., a table, shelf, floor, etc.) if it is not floating in the air.
Alignment performed in the aforementioned manner is helpful for confirming camera settings and camera pose for cases in which the movement of a camera in a cut shot is insufficient to estimate those parameters. Without movement, the combination of four (or some other prescribed number of) non-coplanar points on a reference object in the database and their correspondence to an object in the scene is sufficient to provide a reasonable estimate of the camera pose and settings.
An object specification in the database comprises a variety of information that defines the object, including data related to geometry and appearance. For example, a database object specification may comprise a mesh (which may or may not be parameterized) of vertices of the object and their connectivities as well as normals to the polygons comprising the surfaces defined by the vertices. Furthermore, a database object specification may comprise one or more textures that describe visual details of the object surfaces and/or images of the object surfaces. In various embodiments, such textures may comprise images of the surfaces and/or specifications of parameters that define manners in which the textures may be procedurally generated, i.e., created automatically by a mathematical model rather than by direct application of an image. Moreover, a database object specification may comprise a bump map that describes the roughness of the object surfaces. In various embodiments, such data may be in the form of vector orientations relative to the normal directions of the surfaces of the object and/or may be procedurally generated rather than being tabulated. A database object specification may further comprise a model of the optical behavior of the object at various points on its surfaces and/or information that describes the desired interaction between its surfaces and incident light rays. In various embodiments, such an optical behavior model may range from simple global behavior such as specular reflection (mirror-like response to light), Lambertian diffusion (equally diffuse reflection of light in all directions like paper), a combination of both, or more complex behaviors such as modeling skin or translucent materials where some of the light passes through, bounces around inside, and emerges from another location on the surface and/or object.
Once an object in a scene is initially identified at step 406 and is mapped to a corresponding database object at step 408, features of the object in the source video may be tracked during motion of the object, and the correspondence with the database object as well as camera position/pose may be maintained for reasonable motions, subject to constraints defined by the physics of the scene. For example, the orientation and position of the database object may be estimated through the motion of the tracked points or features of the object in the scene, with the orientation and position being adjusted based on physical constraints such as the object staying connected to a surface it is resting on.
As described, the disclosed platform facilitates representing or reconstructing source content in three dimensions (3D) regardless of whether the source content itself is in 2D or 3D. That is, objects and environments comprising source content may be mapped to 3D versions of similar reference objects and environments from a database. When abstracting into 3D, relative spacings, sizes, and orientations of objects and environments in the modeled scene need to be estimated or determined. For example, object locations and orientations may be estimated such that when they are modeled in the 3D scene and viewed from a perspective corresponding to the camera perspective of the source content, the projected view from the modeled 3D environment onto that 2D perspective are very similar. Given the availability of 3D data with respect to a modeled scene or environment, analysis of such 3D data may be used to determine or infer contextual properties or characteristics of the scene as further described below.
An abstracted or virtualized scene in primitive form is specified by object definitions and their relative positions and/or orientations in the scene. More complex metrics may be computed from such data to better understand the modeled environment or scene.
Analyzing data defining an abstracted scene provides metrics that assist in obtaining an improved understanding of what is transpiring in the scene. Analytics may be performed on a frame of a scene in isolation or over a plurality of frames of the scene, such as frames comprising different perspectives or camera angles or a time sequence of frames. Based on such analytics, at least a partial semantic understanding of the scene may be extracted for a given frame of the scene or a sequence of frames of the scene or the scene as a whole. That is, interpretations of statistics and/or metrics associated with a scene may be employed to infer specific actions or behaviors associated with the scene.
As described, the disclosed content abstraction platform facilitates transforming source content into an abstracted or virtualized 3D modeled environment or scene. In various embodiments, mapping source content to objects of a database, statistical analysis of the objects comprising a scene, and interpretations of statistical data and mapping to associated actions or behaviors may at least in part or completely be automated. Moreover, such information associated with a modeled scene and/or frames and objects thereof may be persisted via metadata or tags. For example, each object comprising a scene may be tagged with tags specifying one or more attributes or characteristics of the object such as type/class, size, shape, composition, material, brand/manufacturer, etc., as well as scale, position, and/or orientation in the modeled environment. Furthermore, tags may be associated with a particular frame or a sequence of frames derived from a scene. Likewise, tags may be associated with a modeled scene as a whole, and such tags may be inherited by all frames or sequences of frames derived from the modeled scene. In some cases, an object may similarly inherit tags from an associated frame and/or scene. At least some of the tags may describe the relationships of objects to other objects in the frame/scene, such as statistical data and/or metrics determined for the frame/scene as well as classifications of the metrics into actions or behaviors. For example, a tag may specify or be based on a number or density of objects, types or classes of objects, the spatial distributions of objects, correlations of objects, camera perspectives associated with a frame/scene, etc., as well as various inferred actions or behaviors that may be based on, for instance, relative motions of objects, time variances of objects, etc. The various levels or hierarchy of tags (e.g., at the object, frame, and scene levels) may be automatically generated and are searchable, providing unprecedented search options for multi-media content. Videos, for example, may be searched and/or organized according to such detailed tags. For instance, a search may be conducted to identify scenes in which a prescribed set of objects (e.g., a phone, a table, a lamp, and a couch) appear simultaneously in view of the camera. Moreover, videos may be searched according to the spatial relationships of the objects comprising a scene. For instance, a search may be conducted to identify scenes in which a phone is laying on top of a table while next to a lamp that is turned on. In addition to tagging, context characteristics and/or inferences may be used to determine the type of advertising that should be served to a customer. For instance, if the estimated movement from a tennis ball over a sequence of frames indicates that the scene is showing people playing tennis, advertising related to products relevant to tennis may be provided.
Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.
Claims
1. A method, comprising:
- generating a modeled scene to represent source content by mapping elements identified in the source content to existing three-dimensional database objects, wherein the modeled scene comprises a modified and virtualized three-dimensional version of the source content;
- determining a metric associated with the modeled scene based on at least a subset of the three-dimensional objects comprising the modeled scene and their positions in the modeled scene;
- inferring at least a partial semantic understanding of the modeled scene based on interpreting the determined metric; and
- tagging the modeled scene with a tag specifying the at least partial semantic understanding of the modeled scene inferred from the determined metric.
2. The method of claim 1, wherein the method is completely automated.
3. The method of claim 1, further comprising providing a specification of the modeled scene instead of the source content in response to a request for the source content.
4. The method of claim 1, wherein determining the metric associated with the modeled scene comprises statistically analyzing data defining the modeled scene.
5. The method of claim 1, wherein the at least partial semantic understanding of the modeled scene comprises a contextual characteristic associated with the modeled scene.
6. The method of claim 1, wherein the at least partial semantic understanding of the modeled scene comprises an action or behavior associated with the modeled scene.
7. The method of claim 1, wherein the at least partial semantic understanding of the modeled scene is based on a number or density of objects comprising the modeled scene.
8. The method of claim 1, wherein the at least partial semantic understanding of the modeled scene is based on types or classes of objects comprising the modeled scene.
9. The method of claim 1, wherein the at least partial semantic understanding of the modeled scene is based on spatial relationships of objects comprising the modeled scene.
10. The method of claim 1, wherein the at least partial semantic understanding of the modeled scene is based on correlations of objects comprising the modeled scene.
11. The method of claim 1, wherein the at least partial semantic understanding of the modeled scene is based on motions of objects comprising the modeled scene.
12. The method of claim 1, wherein the at least partial semantic understanding of the modeled scene is based on time evolutions of objects comprising the modeled scene.
13. The method of claim 1, wherein the at least partial semantic understanding of the modeled scene is based on inter-class or intra-class spatial correlations of objects comprising the modeled scene.
14. The method of claim 1, wherein the tag is associated with an object included in the modeled scene or a frame comprising the modeled scene.
15. The method of claim 1, wherein the tag is inherited by a frame derived from the modeled scene.
16. The method of claim 1, further comprising determining an advertisement to be provided with respect to the modeled scene based on the at least partial semantic understanding of the modeled scene.
17. The method of claim 1, wherein the tagged modeled scene is searchable via the tag.
18. The method of claim 1, wherein the source content comprises two-dimensional content.
19. A system, comprising:
- a processor configured to: generate a modeled scene to represent source content by mapping elements identified in the source content to existing three-dimensional database objects, wherein the modeled scene comprises a modified and virtualized three-dimensional version of the source content; determine a metric associated with the modeled scene based on at least a subset of the three-dimensional objects comprising the modeled scene and their positions in the modeled scene; infer at least a partial semantic understanding of the modeled scene based on interpreting the determined metric; and tag the modeled scene with a tag specifying the at least partial semantic understanding of the modeled scene inferred from the determined metric; and
- a memory coupled to the processor and configured to provide the processor with instructions.
20. The system of claim 19, wherein the processor is further configured to provide a specification of the modeled scene instead of the source content in response to a request for the source content.
21. The system of claim 19, wherein to determine the metric associated with the modeled scene comprises to statistically analyze data defining the modeled scene.
22. The system of claim 19, wherein the at least partial semantic understanding of the modeled scene comprises a contextual characteristic associated with the modeled scene.
23. The system of claim 19, wherein the at least partial semantic understanding of the modeled scene comprises an action or behavior associated with the modeled scene.
24. The system of claim 19, wherein the at least partial semantic understanding of the modeled scene is based on a number or density of objects comprising the modeled scene.
25. The system of claim 19, wherein the at least partial semantic understanding of the modeled scene is based on types or classes of objects comprising the modeled scene.
26. The system of claim 19, wherein the at least partial semantic understanding of the modeled scene is based on spatial relationships of objects comprising the modeled scene.
27. The system of claim 19, wherein the at least partial semantic understanding of the modeled scene is based on correlations of objects comprising the modeled scene.
28. The system of claim 19, wherein the at least partial semantic understanding of the modeled scene is based on motions of objects comprising the modeled scene.
29. The system of claim 19, wherein the at least partial semantic understanding of the modeled scene is based on time evolutions of objects comprising the modeled scene.
30. The system of claim 19, wherein the at least partial semantic understanding of the modeled scene is based on inter-class or intra-class spatial correlations of objects comprising the modeled scene.
31. The system of claim 19, wherein the tag is associated with an object included in the modeled scene or a frame comprising the modeled scene.
32. The system of claim 19, wherein the tag is inherited by a frame derived from the modeled scene.
33. The system of claim 19, wherein the processor is further configured to determine an advertisement to be provided with respect to the modeled scene based on the at least partial semantic understanding of the modeled scene.
34. The system of claim 19, wherein the tagged modeled scene is searchable via the tag.
35. The system of claim 19, wherein the source content comprises two-dimensional content.
36. A computer program product stored on a non-transitory computer readable storage medium and comprising computer instructions which when executed cause a computer to:
- generate a modeled scene to represent source content by mapping elements identified in the source content to existing three-dimensional database objects, wherein the modeled scene comprises a modified and virtualized three-dimensional version of the source content;
- determine a metric associated with the modeled scene based on at least a subset of the three-dimensional objects comprising the modeled scene and their positions in the modeled scene;
- infer at least a partial semantic understanding of the modeled scene based on interpreting the determined metric; and
- tag the modeled scene with a tag specifying the at least partial semantic understanding of the modeled scene inferred from the determined metric.
37. The computer program product of claim 20, further comprising computer instructions which when executed cause a computer to provide a specification of the modeled scene instead of the source content in response to a request for the source content.
38. The computer program product of claim 20, wherein to determine the metric associated with the modeled scene comprises to statistically analyze data defining the modeled scene.
39. The computer program product of claim 20, wherein the at least partial semantic understanding of the modeled scene comprises a contextual characteristic associated with the modeled scene.
40. The computer program product of claim 20, wherein the at least partial semantic understanding of the modeled scene comprises an action or behavior associated with the modeled scene.
41. The computer program product of claim 20, wherein the at least partial semantic understanding of the modeled scene is based on a number or density of objects comprising the modeled scene.
42. The computer program product of claim 20, wherein the at least partial semantic understanding of the modeled scene is based on types or classes of objects comprising the modeled scene.
43. The computer program product of claim 20, wherein the at least partial semantic understanding of the modeled scene is based on spatial relationships of objects comprising the modeled scene.
44. The computer program product of claim 20, wherein the at least partial semantic understanding of the modeled scene is based on correlations of objects comprising the modeled scene.
45. The computer program product of claim 20, wherein the at least partial semantic understanding of the modeled scene is based on motions of objects comprising the modeled scene.
46. The computer program product of claim 20, wherein the at least partial semantic understanding of the modeled scene is based on time evolutions of objects comprising the modeled scene.
47. The computer program product of claim 20, wherein the at least partial semantic understanding of the modeled scene is based on inter-class or intra-class spatial correlations of objects comprising the modeled scene.
48. The computer program product of claim 20, wherein the tag is associated with an object included in the modeled scene or a frame comprising the modeled scene.
49. The computer program product of claim 20, wherein the tag is inherited by a frame derived from the modeled scene.
50. The computer program product of claim 20, further comprising computer instructions which when executed cause a computer to determine an advertisement to be provided with respect to the modeled scene based on the at least partial semantic understanding of the modeled scene.
51. The computer program product of claim 20, wherein the tagged modeled scene is searchable via the tag.
52. The computer program product of claim 20, wherein the source content comprises two-dimensional content.
| 5930783 | July 27, 1999 | Li |
| 6084590 | July 4, 2000 | Robotham |
| 6124864 | September 26, 2000 | Madden |
| 6516090 | February 4, 2003 | Lennon |
| 7065250 | June 20, 2006 | Lennon |
| 7124149 | October 17, 2006 | Smith |
| 7143434 | November 28, 2006 | Paek |
| 7185049 | February 27, 2007 | Benitez |
| 7383504 | June 3, 2008 | Divakaran |
| 7836437 | November 16, 2010 | Kacmarcik |
| 8370869 | February 5, 2013 | Paek |
| 8737721 | May 27, 2014 | Arcas |
| 8798965 | August 5, 2014 | Quan |
| 8817021 | August 26, 2014 | Hickman |
| 9330296 | May 3, 2016 | Chakraborty |
| 20060285755 | December 21, 2006 | Hager |
| 20080193010 | August 14, 2008 | Eaton |
| 20100214284 | August 26, 2010 | Rieffel |
| 20100245344 | September 30, 2010 | Chen |
| 20110025689 | February 3, 2011 | Perez |
| 20110072047 | March 24, 2011 | Wang |
| 20120075296 | March 29, 2012 | Wegbreit |
| 20120219223 | August 30, 2012 | Hutter |
| 20120314935 | December 13, 2012 | Cheng |
| 20130124311 | May 16, 2013 | Sivanandan |
| 20130129307 | May 23, 2013 | Choe |
| 20130177235 | July 11, 2013 | Meier |
| 20130282747 | October 24, 2013 | Cheng |
| 20140270363 | September 18, 2014 | Chakraborty |
| 20140354693 | December 4, 2014 | Isola |
Type: Grant
Filed: Jun 7, 2019
Date of Patent: Jun 9, 2020
Patent Publication Number: 20190362152
Assignee: Outward, Inc. (San Jose, CA)
Inventor: Clarence Chui (Los Altos Hills, CA)
Primary Examiner: Eric Rush
Application Number: 16/435,376
International Classification: G06K 9/72 (20060101); G06K 9/00 (20060101); G06T 19/20 (20110101); G06T 19/00 (20110101);
