Abstract: Disclosed are systems and methods for generating a walkable 360-degree video or virtual reality (VR) environment. 360-degree video data is obtained for a real-world environment and comprises a plurality of chronologically ordered frames captured by traversing a first path through the real-world environment. One or more processing operations are applied to generate a processed 360-degree video, which can be displayed to a user of an omnidirectional treadmill. Locomotion information is received from one or more sensors of the omnidirectional treadmill, wherein the locomotion information is generated based on a physical movement on or within the omnidirectional treadmill. Using the received locomotion information, one or more playback commands for controlling playback of the processed 360-degree video are generated. One or more selected frames of the processed 360-degree video are rendered for presentation and display to the user, based on the one or more playback commands.

Abstract: Digital image dynamic shadow generation is described as implemented by a dynamic shadow system using one or more computing devices. The dynamic shadow system is configured to generate shadow objects based on one or more source objects included in a digital image (e.g., a two-dimensional digital image), automatically and without user intervention. The shadow object is based on a shape of the source object that is to “cast” the shadow and thus promotes realism. The shadow object is also generated by the dynamic shadow system to address an environment, in which, the shadow object is disposed within the digital image.

Abstract: Techniques to improve operation of a processor or device utilizing colorspace conversions and operations are provided. A system includes a programmable logic array that can perform, a colorspace conversion, which can in turn be by a distinct processor device. The colorspace conversion can be performed by the programmable logic in response to the change of a colorspace associated with a target, e.g. a real or virtual environment, and the colorspace conversion is intended to optimize the colorspace of an object, entity, or other environment in the changed real or virtual environment, and where the optimized object, entity or other environment can be used by the processor device as part of a computing application as suitable for that colorspace conversion, which can include but is not limited to augmented reality, printing of images, detection of images, global positioning, and detection of real and virtual objects in real or virtual environments.

Abstract: A sensor simulation system may generate sensor data for use in simulations by rendering two-dimensional views of a three-dimensional simulated environment. In various examples, the sensor simulation system uses sensor dependency data to determine specific views to be re-rendered at different times during the simulation. The sensor simulation system also may generate unified views with multi-sensor data at each region (e.g., pixel) of the two-dimensional view for consumption by different sensor types. A hybrid technique may be used in some implementations in which rasterization is used to generate a view, after which ray tracing is used to align the view with a particular sensor. Spatial and temporal upsampling techniques also may be used, including depth-aware and velocity-aware analyses for simulated objects, to improve view resolution and reduce the frequency of re-rendering views.

Abstract: Methods, apparatuses, mediums, and devices for generating multi-angle free-respective image data are provided. The method for generating multi-angle free-perspective image data includes: acquiring multiple synchronized images, where the multiple images have different shooting angles; determining the depth data of each image based on the multiple images; and for each of the images, storing pixel data of the image in a first field and storing depth data in a second field associated with the first field. Technical solutions in the example embodiments of the present invention may improve the user experience.

Abstract: In one embodiment, a computing system accesses a three-dimensional (3D) model of an environment, the 3D model comprising a virtual representation of an object in the environment. The computing system accesses an image of the object captured by a camera from a camera pose. The computing system accesses light source parameters associated with a virtual representation of a light source in the environment. The computing system renders, using the 3D model, pixels associated with the virtual representation of the object based on the light source parameters, the pixels being rendered from a virtual perspective corresponding to the camera pose. The computing system determines updated light source parameters based on a comparison of the rendered pixels to corresponding pixels located in the image of the object.

Abstract: Techniques are disclosed for improving the throughput of ray intersection or visibility queries performed by a ray tracing hardware accelerator. Throughput is improved, for example, by releasing allocated resources before ray visibility query results are reported by the hardware accelerator. The allocated resources are released when the ray visibility query results can be stored in a compressed format outside of the allocated resources. When reporting the ray visibility query results, the results are reconstructed based on the results stored in the compressed format. The compressed format storage can be used for ray visibility queries that return no intersections or terminate on any hit ray visibility query. One or more individual components of allocated resources can also be independently deallocated based on the type of data to be returned and/or results of the ray visibility query.

Abstract: Disclosed are systems and methods for generating large data sets for training deep learning networks (DLNs) for 3D measurements extraction from 2D images taken using a mobile device camera. The method includes the steps of receiving a 3D model of a 3D object; extracting spatial features from the 3D model; generating a first type of augmentation data for the 3D model, such as but not limited to skin color, face contour, hair style, virtual clothing, and/or lighting conditions; augmenting the 3D model with the first type of augmentation data to generate an augmented 3D model; generating at least one 2D image from the augmented 3D model by performing a projection of the augmented 3D model onto at least one plane; and generating a training data set to train the deep learning network (DLN) for spatial feature extraction by aggregating the spatial features and the at least one 2D image.

Abstract: The present disclosure relates to information processing apparatus and method that makes it possible to suppress a reduction in encoding efficiency. Information relating to quantization of a three-dimensional position of an encoding target is generated. For example, the information relating to the quantization includes information relating to a coordinate system to be subjected to the quantization, information relating to a bounding box for normalization of position information of the encoding target, or information relating to a voxel for quantization of position information of the encoding target. In addition, three-dimensional information of the encoding target is restored from a signal string on the basis of the information relating to the quantization of the three-dimensional position of the encoding target.

Abstract: Apparatus and method for acceleration data structure refit. For example, one embodiment of an apparatus comprises: a ray generator to generate a plurality of rays in a first graphics scene; a hierarchical acceleration data structure generator to construct an acceleration data structure comprising a plurality of hierarchically arranged nodes including inner nodes and leaf nodes stored in a memory in a depth-first search (DFS) order; traversal hardware logic to traverse one or more of the rays through the acceleration data structure; intersection hardware logic to determine intersections between the one or more rays and one or more primitives within the hierarchical acceleration data structure; a node refit unit comprising circuitry and/or logic to read consecutively through at least the inner nodes in the memory in reverse DFS order to perform a bottom-up refit operation on the hierarchical acceleration data structure.

Abstract: A graphics processing unit (GPU) includes one or more processor cores adapted to execute a software-implemented shader program, and one or more hardware-implemented ray tracing units (RTU) adapted to traverse an acceleration structure to calculate intersections of rays with bounding volumes and graphics primitives asynchronously with shader operation. The RTU implements traversal logic to traverse the acceleration structure including transformation of rays as needed to account for variations in coordinate space between levels, stack management, and other tasks to relieve burden on the shader, communicating intersections to the shader which then calculates whether the intersection hit a transparent or opaque portion of the object intersected. Thus, one or more processing cores within the GPU perform accelerated ray tracing by offloading aspects of processing to the RTU, which traverses the acceleration structure within which the 3D environment is represented.

Abstract: Described herein is a technique for performing ray tracing. According to this technique, instead of executing intersection and/or any hit shaders during traversal of an acceleration structure to determine the closest hit for a ray, an acceleration structure is fully traversed in an invocation of a shader program, and the closest intersection with a triangle is recorded in a data structure associated with the material of the triangle. Later, a scheduler launches waves by grouping together multiple data items associated with the same material. The rays processed by that wave are processed with a continuation ray, rather than the full original ray. A continuation ray starts from the previous point of intersection and extends in the direction of the original ray. These steps help counter divergence that would occur if a single shader program that inlined the intersection and any hit shaders were executed.

Abstract: Embodiments described herein provide a process and method running on a computer for creating an augmented image. According to an embodiment, a graphical user interface gathers data that is programmatically analyzed to obtain photographic properties from a first image. Photographic properties are provided to a user for obtaining a second image containing a fiducial mark. The second image is programmatically analyzed to obtain photographic properties. The first image and the second image are programmatically analyzed and processed to produce an augmented image.

Abstract: The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned geometry model can predict one or more adjusted depths for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.

Abstract: To provide an information processing device, an information processing method, and a program that can give a user a stronger impression that a real world is enhanced by using an AR technique. The information processing device includes a recognition unit that recognizes an object included in a real space so as to distinguish the object from a background on the basis of three-dimensional data of the real space in order to generate a virtual object image obtained by changing a state of the object.

Abstract: Techniques are described for operating an optical system. In some embodiments, light associated with a world object is received at the optical system. Virtual image light is projected onto an eyepiece of the optical system. A portion of a system field of view of the optical system to be at least partially dimmed is determined based on information detected by the optical system. A plurality of spatially-resolved dimming values for the portion of the system field of view may be determined based on the detected information. The detected information may include light information, gaze information, and/or image information. A dimmer of the optical system may be adjusted to reduce an intensity of light associated with the world object in the portion of the system field of view according to the plurality of dimming values.

Abstract: In a ray tracer, to prevent any long-running query from hanging the graphics processing unit, a traversal coprocessor provides a preemption mechanism that will allow rays to stop processing or time out early. The example non-limiting implementations described herein provide such a preemption mechanism, including a forward progress guarantee, and additional programmable timeout options that can be time or cycle based. Those programmable options provide a means for quality of service timing guarantees for applications such as virtual reality (VR) that have strict timing requirements.

Abstract: Embodiments are generally directed to methods and apparatuses for encoding based on shading rates. An embodiment of a computing system comprises: a memory; a graphics processing unit (GPU) coupled to the memory, the GPU to render a scene of a graphics application into a color buffer within a frame buffer of the memory; and an encoder coupled to the memory, the encoder to encode the content of the color buffer into a video bitstream based on a plurality of shading rates each corresponding to a pixel in the color buffer.

Abstract: One embodiment of a method for computing a texture color includes tracing a ray cone through a graphics scene, determining a curvature of a first surface within the graphics scene at a point where the ray cone hits the first surface based on differential barycentric coordinates associated with the point, determining, based on the curvature of the first surface, a width of the ray cone at a subsequent point where the ray cone hits a second surface within the graphics scene, and computing a texture color based on the width of the ray cone.

Abstract: A method, apparatus and computer program product are provided to localize data from at least one of two or more data sets a based upon the registration of synthetic images representative of the two or more data sets. In the context of a method, first and second synthetic images are created from first and second data sets, respectively. In creating the first and second synthetic images, representations of one or more features from the first and second data sets are rasterized. The method also determines a transformation based upon a phase correlation between the first and second synthetic images. The transformation provides for improved localization of the data from at least one of the first or second data sets. The method also generates a report that defines the transformation.

Abstract: A method is performed at a moveable scanner with one or more optical sensors. The method includes scanning, using the moveable scanner, an object having a surface. The scanning generates color data from a plurality of orientations of the moveable scanner with respect to the object. The method further includes generating, using at least the color data, a pixel map of the surface of the object, the pixel map including, for each respective pixel of a plurality of pixels: a color value of a corresponding point on the surface of the object; and a value for a non-color property of the corresponding point on the surface of the object.

Abstract: A medical processing apparatus comprises processing circuitry configured to: receive an image data set for rendering; for each of a plurality of a pixels or voxels in the image data set: set a region of interest around the pixel or voxel; determine a maximum data value and a minimum data value for pixels or voxels in the region of interest; and designate the pixel or voxel as visible or as non-visible based on the maximum data value and the minimum data value for the region of interest; and perform a rendering process using the pixels or voxels of the image data set that designated as visible.

Abstract: Disclosed techniques relate to primitive testing associated with ray intersection processing for ray tracing. In some embodiments, shader circuitry executes a first SIMD group that includes a ray intersect instruction for a set of rays. Ray intersect circuitry traverses, in response to the ray intersect instruction, multiple nodes in a spatially organized acceleration data structure (ADS). In response to reaching a node of the ADS that indicates one or more primitives, the apparatus forms a second SIMD group that executes one or more instructions to determine whether a set of rays that have reached the node intersect the one or more primitives. The shader circuitry may execute the first SIMD group to shade one or more primitives that are indicated as intersected based on results of execution of the second SIMD group. Thus, disclosed techniques may use both dedicated ray intersect circuitry and dynamically formed SIMD groups executed by shader processors to detect ray intersection.

Abstract: A system and method for generating a set of samples stratified across two-dimensional elementary intervals of a two-dimensional space is disclosed within the application. A computer-implemented technique for generating the set of samples includes selecting an elementary interval associated with a stratification of the two-dimensional space, initializing at least one data structure that indicates valid regions within the elementary interface based on other samples previously placed within the two-dimensional space, and generating a sample in a valid region of the elementary interval utilizing the at least one data structure to identify the valid region prior to generating the sample. In some embodiments, the data structures comprise a pair of binary trees. The process can be repeated for each elementary interval of a selected stratification to generate the set of stratified two-dimensional samples.

Abstract: A method renders photorealistic images in a web browser. The method is performed at a computing device having a general purpose processor and a graphics processing unit (GPU). The method includes obtaining an environment map and images of an input scene. The method also includes computing textures for the input scene including by encoding an acceleration structure of the input scene. The method further includes transmitting the textures to shaders executing on a GPU. The method includes generating samples of the input scene, by performing at least one path tracing algorithm on the GPU, according to the textures. The method also includes lighting or illuminating a sample of the input scene using the environment map, to obtain a lighted scene, and tone mapping the lighted scene. The method includes drawing output on a canvas, in the web browser, based on the tone-mapped scene to render the input scene.

Abstract: Methods, systems, and non-transitory computer readable storage media are disclosed for generating enriched light sources by utilizing surface-centric representations of three-dimensional surfaces. Specifically, the disclosed system utilizes a surface-centric re-parameterization that combines geometric and algebraic components of a sphere to model different light source types in a continuous range of lighting configurations. The disclosed systems utilize a set of intuitive parameters to determine a shape and emission parameters for generating an enriched light source. Additionally, the disclosed system provides a set of interactive light source controls to modify a position, orientation, shape, emittance, and lighting attenuation over distance of a light source within a three-dimensional environment. The disclosed system determines the light source controls based on sets of three-dimensional interaction primitives to control one or more parameters of the light source.

Abstract: A system for the viewing and selection of hair products includes a processor configured to obtain customer identification information characterizing one or more characteristics of a customer and to obtain customer measurement information characterizing one or more physical measurements of the customer. The processor also obtains selected product information characterizing a selected hair product of the customer and sends finalized order information regarding the selected hair product.

Abstract: A medical image processing apparatus comprises processing circuitry configured to: acquire a data volume to be compressed; acquire a function relating color value to each data value of the data volume; and change a compression rate per region of the data volume based on color values in each region.

Abstract: A method and controller for selecting media content based on a lighting scene, the method comprising: selecting the lighting scene, the lighting scene having properties comprising one or more lighting properties; determining one or more of the properties of the selected lighting scene; selecting media content based on the determined one or more of the properties, wherein the media content comprises audio content; controlling a media device to output the selected media content; and adjusting one or more of the lighting properties based on the audio content of the selected media content.

Abstract: In one embodiment, a method includes accessing one or more surfaces of an artificial reality scene. The one or more surfaces are generated based on one or more images of the artificial reality scene rendered at a first rate and based on a first viewing position. The method includes generating subframes at a second rate higher than the first rate. Each of the subframes is generated by determining a second viewing position, determining a visibility of the one or more surfaces of the artificial reality scene from the second viewing position, generating, based on the determined visibility of the one or more surfaces, color values of the subframe corresponding to output positions of a display, and providing the color values of the subframe for display.

Abstract: A hierarchical acceleration structure is generated for intersection testing in a ray tracing system. Nodes of the hierarchical acceleration structure are determined, wherein each of the nodes represents a region in a scene, and wherein the nodes are linked to form the hierarchical acceleration structure. Data is stored representing the hierarchical acceleration structure. The stored data defines the regions represented by a plurality of the nodes of the hierarchical acceleration structure. At least one node is an implicitly represented node, wherein data defining a region represented by an implicitly represented node is not explicitly included as part of the stored data but can be inferred from the stored data.

Abstract: A video analysis system includes: a video data acquiring means that acquires video data; a moving object detecting means that detects a moving object from video data acquired by the video data acquiring means, by using a moving object detection parameter, which is a parameter for detecting a moving object; an environment information collecting means that collects environment information representing an external environment of a place where the video data acquiring means is installed; and a parameter changing means that changes the moving object detection parameter used when the moving object detecting means detects a moving object, on the basis of the environment information collected by the environment information collecting means.

Abstract: The invention relates to a holographic display device for representing a two-dimensional and/or three-dimensional scene. The holographic display device comprises at least one spatial light modulator device and an optical component. The at least one spatial light modulator device is provided in order to reconstruct the scene and in order to generate at least one virtual visibility region in an observer plane. The optical component is configured with at least two regions that have a different transparency to one another, the value of the transparency respectively lying between 0 and 1. Furthermore, the optical component is arranged in the display device in such a way that it provides filtering, to be carried out at least partially, of a diffraction order spot in at least one diffraction order inside the virtual visibility region.

Abstract: A hair rendering method, a device, an electronic apparatus, and a storage medium, pertaining to the technical field of computer graphics. The method comprises: acquiring multiple initial sections used as cross sections for simulating hair growth (201); determining hair distribution points on each of the multiple initial sections according to a rule of successively decreasing hair distribution densities, and obtaining multiple hair sections (202); sequentially rendering, by means of a pre-determined illumination model, the multiple hair sections on a three-dimensional model requiring hair rendering, and obtaining a target three-dimensional model, the pre-determined illumination model being used to simulate a hair illumination effect (203).

Abstract: A field cooperation system according to an embodiment includes a terminal device and a management device. The terminal device includes: a position measurement unit; and a display unit. The position measurement unit measures a position of the terminal device. A display device displays a display image. The management device includes: a database; and a display control unit. The database stores three-dimensional data representing a layout of a structure. The display control unit generates a display image to illustrate the structure and an icon representing the position of the terminal device on the display unit in correspondence with each other based on the three-dimensional data and the measured position of the terminal device.

Abstract: Techniques for performing ray tracing for a ray are provided. The techniques include, based on first traversal of a bounding volume hierarchy, identifying a first memory page that is classified as resident, obtaining a first portion of the bounding volume hierarchy associated with the first memory page, traversing the first portion of the bounding volume hierarchy according to a ray intersection test, based on second traversal of the bounding volume hierarchy, identifying a second memory page that is classified as valid and non-resident, and in response to the second memory page being classified as valid and non-resident, determining that a miss occurs for each node of the bounding volume hierarchy within the second memory page.

Abstract: Illustrative methods and systems described herein use lightmaps to present 3D graphics. For example, a method includes identifying a viewpoint within a 3D scene and receiving a dynamic lightmap for an object represented by a 3D model in the 3D scene. The identified viewpoint is selected by a user of a media player device presenting the 3D scene to the user and the dynamic lightmap is generated or updated at a first level of detail determined based on the identified viewpoint. The method further includes rendering an image for presentation to the user from the identified viewpoint. This rendering of the image includes rendering the 3D model at a second level of detail that is lower than the first level of detail, and applying the dynamic lightmap at the first level of detail to the 3D model rendered at the second level of detail. Corresponding systems are also disclosed.

Abstract: Disclosed are herein a three-dimensional (3D) object acquisition method and an object acquisition apparatus using artificial light photographs. A 3D object acquisition method according to an embodiment of the present invention includes receiving a plurality of images of a 3D object photographed by a camera, reconstructing spatially-varying bidirectional reflectance distribution functions for the 3D object based on the plurality of images received, estimating shading normals for the 3D object based on the reconstructed spatially-varying bidirectional reflectance distribution functions, and acquiring 3D geometry for the 3D object based on the estimated shading normals.

Abstract: Enhanced techniques applicable to a ray tracing hardware accelerator for traversing a hierarchical acceleration structure are disclosed. For example, traversal efficiency is improved by combining programmable traversals based on ray operations with per-node static configurations that modify traversal behavior. The per-node static configurations enable creators of acceleration data structures to optimize for potential traversals without necessarily requiring detailed information about ray characteristics and ray operations used when traversing the acceleration structure. Moreover, by providing for selective exclusion of certain nodes using per-node static configurations, less memory is needed to express an acceleration structure that includes, for example, different geometric levels of details corresponding to a single object.

Abstract: Introduced here are techniques for relighting an image by automatically segmenting a human object in an image. The segmented image is input to an encoder that transforms it into a feature space. The feature space is concatenated with coefficients of a target illumination for the image and input to an albedo decoder and a light transport detector to predict an albedo map and a light transport matrix, respectively. In addition, the output of the encoder is concatenated with outputs of residual parts of each decoder and fed to a light coefficients block, which predicts coefficients of the illumination for the image. The light transport matrix and predicted illumination coefficients are multiplied to obtain a shading map that can sharpen details of the image. Scaling the resulting image by the albedo map to produce the relight image. The relight image can be refined to denoise the relight image.

Abstract: Systems and methods for simulation of ambient light based on generated imagery are disclosed herein. Such a system can include a simulation sled, a simulation display that can display generated imagery viewable from the simulation sled, an ambient light simulator that can selectively illuminate portions of the simulation sled, and a processor. The simulation sled can include a plurality of user controls. The processor can: control the simulation display to generate imagery; identify an effect of the generated imagery on the simulation sled; and control the ambient light simulator to selectively illuminate at least portions of the simulation sled according to the identified effect of the simulated light source.

Abstract: A volumetric data structure models a particular volume representing the particular volume at a plurality of levels of detail. A first entry in the volumetric data structure includes a first set of bits representing voxels at a first level of detail, the first level of detail includes the lowest level of detail in the volumetric data structure, values of the first set of bits indicate whether a corresponding one of the voxels is at least partially occupied by respective geometry, where the volumetric data structure further includes a number of second entries representing voxels at a second level of detail higher than the first level of detail, the voxels at the second level of detail represent subvolumes of volumes represented by voxels at the first level of detail, and the number of second entries corresponds to a number of bits in the first set of bits with values indicating that a corresponding voxel volume is occupied.

Abstract: Disclosed techniques relate to ray intersection processing for ray tracing. In some embodiments, ray intersection circuitry traverses a spatially organized acceleration data structure and includes bounding region circuitry configured to test, in parallel, whether a ray intersects multiple different bounding regions indicated by a node of the data structure. Shader circuitry may execute a ray intersect instruction to invoke traversal by the ray intersect circuitry and the traversal may generate intersection results. The shader circuitry may shade intersected primitives based on the intersection results. Disclosed techniques that share processing between intersection circuitry and shader processors may improve performance, reduce power consumption, or both, relative to traditional techniques.

Abstract: In various examples, the actual spatial properties of a virtual environment are used to produce, for a pixel, an anisotropic filter kernel for a filter having dimensions and weights that accurately reflect the spatial characteristics of the virtual environment. Geometry of the virtual environment may be computed based at least in part on a projection of a light source onto a surface through an occluder, in order to determine a footprint that reflects a contribution of the light source to lighting conditions of the pixel associated with a point on the surface. The footprint may define a size, orientation, and/or shape of the anisotropic filter kernel and corresponding filter weights. The anisotropic filter kernel may be applied to the pixel to produce a graphically-rendered image of the virtual environment.

Abstract: Systems and methods for displaying a cursor and a focus indicator associated with real or virtual objects in a virtual, augmented, or mixed reality environment by a wearable display device are disclosed. The system can determine a spatial relationship between a user-movable cursor and a target object within the environment. The system may render a focus indicator (e.g., a halo, shading, or highlighting) around or adjacent objects that are near the cursor. The focus indicator may be emphasized in directions closer to the cursor and deemphasized in directions farther from the cursor. When the cursor overlaps with a target object, the system can render the object in front of the cursor (or not render the cursor at all), so the object is not occluded by the cursor. The cursor and focus indicator can provide the user with positional feedback and help the user navigate among objects in the environment.

Abstract: Various implementations disclosed herein include devices, systems, and methods that enable presenting environments comprising visual representations of multiple applications. In one implementation, a method includes presenting a view of an environment at an electronic device on a display of the electronic device. The environment comprising visual representations corresponding to a plurality of applications. A first application among the plurality of applications is designated as an elevated application. The elevated application is provided with access to a control parameter configured to modify an ambience of the environment. Other applications of the plurality of applications are restricted from accessing the control parameter while the first application is designated as the elevated application.

Abstract: Described is a method for processing image data to determine if a portion of the imaged environment is exposed to high illumination, such as sunlight. In some implementations, image data from multiple different imaging devices may be processed to produce for each imaging device a respective illumination mask that identifies pixels that represent a portion of the environment that is exposed to high illumination. Overlapping portions of those illumination masks may then be combined to produce a unified illumination map of an area of the environment. The unified illumination map identifies, for different portions of the environment, a probability that the portion is actually exposed to high illumination.

Abstract: Methods for capturing and generating information about objects in a 3D environment that can be used to support augmented reality or virtual reality playback operations in a data efficient manner are described. In various embodiments one or more frames including foreground objects are generated and transmitted with corresponding information that can be used to determine the location where the foreground objects are to be positioned relative to a background for one or more frame times are described. Data efficiency is achieved by specifying different locations for a foreground object for different frame times avoiding in some embodiments the need to transmit an image and depth information defining the same of the foreground for each frame time. The frames can be encoded using a video encoder even though some of the information communicated are not pixel values but alpha blending values, object position information, mesh distortion information, etc.

Abstract: In one implementation, a non-transitory computer-readable storage medium stores program instructions computer-executable on a computer to perform operations. The operations include obtaining first content representing a physical environment in which an electronic device is located using an image sensor of the electronic device. A physical feature corresponding to a physical object in the physical environment is detected using the first content. A feature descriptor corresponding to a physical parameter of the physical feature is determined using the first content. Second content representing a computer generated reality (CGR) environment is generated based on the feature descriptor and presented on a display of the electronic device.

Abstract: A probabilistic model is provided based on an output of a matching procedure that matches a particular object to representations of objects, where the probabilistic model relates a probability of an object being present to a number of matching features. The probabilistic model is used for detecting whether a particular object is present in received visual data.