Abstract: A physician performing a physical examination can miss subtle abnormalities, such as alterations of respiratory rate or dangerous skin lesion. A surgeon performing surgery can miss small areas of bleeding or tumor implants. This invention comprises head display unit comprising sensors including low light level TV (L3TV) cameras that gather data and in real time analyzes the data for potential hazards and alerts the user of a potential hazardous scenario.

Abstract: Disclosed are various embodiments for selecting augmenting reality (AR) objects based on contextual cues associated with an image captured by a camera associated with electronic device. Contextual cues are obtained at an electronic device and AR objects are identified from a memory associated with the electronic device. The electronic device implements a processor employing image segmentation techniques to combine the identified AR objects with the captured image and render the combined image for display at a display associated with the electronic device.

Abstract: The present invention is a file generation apparatus that generates a file for generating a virtual viewpoint image. The image file generation apparatus in one aspect of the present invention has a material information accumulation unit 520 configured to acquire and accumulate material information used for generation of a virtual viewpoint image, a virtual advertisement information acquisition unit 530 configured to acquire advertisement information that is displayed on a virtual viewpoint image, and an image file generation unit 540 configured to generate an image file including material information and advertisement information.

Abstract: A system and a method are provided for selecting a site for displaying information to a user in a geographical area including a plurality of sites at which information may be displayed. Position, orientation and optionally gaze direction data are received from headsets worn by a plurality of users in the geographical area. The data is analyzed to determine for each site a probability of the site being viewed by users in said geographical area. A request is received for a selection of a site at which to display information to a particular user wearing a headset. The request identifies the headset and includes position of said particular user. In response to a request, a site is selected based on the analysis and signal identifying the selected site is output.

Abstract: Techniques are described for providing stabilized display components in a wearable heads-up display. A hinge is provided that includes at least two portions rotatable relative to each other, such that each portion may be directly coupled to an optical combiner or a light engine. A bias element may be provided to bias arms of a WHUD towards an unfolded configuration. The bias element and surrounding structure may inhibit movement of arms of the WHUD between the unfolded configuration and a folded configuration.

Abstract: A method for providing an immersive VR experience comprises defining in the computer memory, a model representing a three-dimensional model. The method further comprises producing field data based upon a simulation of the three-dimensional model. Additionally, the method comprises storing the field data within a data structure. The method also comprises extracting, for display, a surface of the three-dimensional model from a simulation model. The method additionally comprises creating a surface texture for the surface of the three-dimensional model from the field data. Further, the method comprises creating a query optimized grid from the calculated field data. Further still, the method comprises displaying a visualization of the calculated field data by means of the surface and the query optimized grid.

Abstract: In one embodiment, a method includes receiving image data corresponding to an external environment of a user. The image data is captured at a first time and comprises a body part of the user. The method also includes receiving a first tracking data generated based on measurements made at the first time by at least one motion sensor associated with the body part; generating, based at least on the image data, a model representation associated with the body part; receiving a second tracking data generated based on measurements made at a second time by the at least one motion sensor associated with the body part; determining a deformation of the model representation associated with the body part based on the first tracking data and the second tracking data; and displaying the deformation of the model representation associated with the body part of the user.

Abstract: Described is an intelligent real-time multiplayer content management and data analytics system for AR-based social platforms. The data management system described herein manages AR content and multiple user interactions with the AR content. Additionally, a new multiplayer multiple-stage information augmentation design based on real-time data analysis and live AR interaction is described. In this design, AR content design is very flexible and may be organized into one or more stages containing pre-defined content, on-line searched content, user-generated content, other user-generated content, real-time user interactively generated content, or some combination. The flexible content structure allows for a highly customizable AR social experience to maximize the system performance and user experience.

Abstract: A system and method for visualizing multiple datasets in a virtual 3-dimensional interactive environment. Multiple datasets may be related and virtually cast as 3-dimensional type structures. User interfaces, such as game controllers or headsets, may be used to present the dataset from differing perspectives including the appearance of moving through the data. Certain embodiments provide for mirror image views that allow for presentation of higher order datasets. Other embodiments provide for animation or motion indicia to show how the data is changing and the results on the display. The datasets may represent physical areas or virtual areas as well as demographic, sensors and financial information.

Abstract: An augmented reality system includes a light source to generate a virtual light beam, the virtual light beam carrying information for a virtual object. The system also includes a light guiding optical element, the light guiding optical element allowing a first portion of a first real-world light beam to pass therethrough, where the virtual light beam enters the light guiding optical element, propagates through the light guiding optical element by substantially total internal reflection (TIR), and exits the light guiding optical element. The system further includes a lens disposed adjacent and exterior to a surface of the light guiding optical element, the lens comprising a light modulating mechanism to absorb a second portion of the real-world light beam and to allow the first portion of the real-world light to pass through the lens.

Abstract: A non-transitory computer-readable medium including an image processing program is provided. To control synthesis of a real space image and a virtual object image, the image processing program causes a server to: obtain positional information of a real space image and a user terminal with a photographing device from the user terminal; refer to a storing unit storing information defined in advance as light source related information and positional information so as to be associated with each other, to specify the light source related information corresponding to positional information of the user terminal; set a virtual light source for a virtual object on a basis of the specified light source related information; generate a virtual object image on a basis of the set virtual light source; and output a synthetic image obtained by synthesizing the generated the virtual object image and the real space image.

Abstract: Methods and systems are provided for a reprojection engine for augmented-reality devices. The augmented-reality device projects virtual content within a real-world environment. The augmented-reality device tracks a six degrees of freedom headpose of the augmented-reality device, depth information of the virtual content, motion vectors that correspond to movement of the virtual content, and a color buffer for a reprojection engine. The reprojection engine generates a reprojection of the virtual content defined by an extrapolation of a first frame using the headpose, the depth information, motion vectors, and the color surface data structure. The reprojected virtual content continues to appear as if positioned with the real-world environment regardless of changes in the headpose of the augmented-reality device or motion of the virtual content.

Abstract: A method for remotely controlling a computing device comprises repeatedly capturing an image from a video frame, detecting a human face in the captured image, matching the detected human face to a previously detected human face, extracting facial landmarks from the matched detected human face, estimating a 3D head pose of the matched detected human face based on the extracted facial landmarks, the 3D head pose being represented in an egocentric coordinate system by a 3D pose vector which is directed from the human face, the 3D pose vector being free to rotate around x-, y- and z-axes of the egocentric coordinate system using respective rotation matrices and free to translate along these x-, y- and z-axes using a translation vector, and controlling a user interface on a display screen of the computing device according to the estimated 3D head pose.

Abstract: A removable lens for glasses includes an electrochromic layer adapted to transition between at least two states of a transparency level dependent on an electrical condition applied to the electrochromatic layer and an electrical connection element physically adapted to make an electrical connection with the electrochromic layer, wherein the electrical connection element is positioned to align with a variable-power electrode mounted on the glasses such that, when the removable lens is mounted on the glasses, the variable-power electrode and the electrical connection element make the electrical connection between the variable-power electrode and an electrically conductive layer of the electrochromic layer.

Abstract: Techniques are disclosed for facilitating electronic commerce in an augmented reality environment. In some embodiments, a method comprises detecting, by a mobile device, presence of the physical product or the real life service; and presenting, on the mobile device, information to conduct the transaction of a physical product or a real life service via the augmented reality environment. In some embodiments, a method comprises detecting one or more targets in the augmented reality platform using a select area in a perspective of a user, the perspective being captured by a mobile device; and prompting the user to choose an object of interest from the one or more detected targets.

Abstract: Techniques described herein enable a user to interact with an automated assistant and obtain relevant output from the automated assistant without requiring arduous typed input to be provided by the user and/or without requiring the user to provide spoken input that could cause privacy concerns (e.g., if other individuals are nearby). The assistant application can operate in multiple different image conversation modes in which the assistant application is responsive to various objects in a field of view of the camera. The image conversation modes can be suggested to the user when a particular object is detected in the field of view of the camera. When the user selects an image conversation mode, the assistant application can thereafter provide output, for presentation, that is based on the selected image conversation mode and that is based on object(s) captured by image(s) of the camera.

Abstract: A method for rendering computer graphics based on saccade detection is provided. One embodiment of the method includes rendering a computer simulated scene for display to a user, detecting an onset of a saccade that causes saccadic masking in an eye movement of the user viewing the computer simulated scene, and reducing a computing resource used for rendering frames of the computer simulated scene during at least a portion of a duration of the saccade. Systems perform similar steps, and non-transitory computer readable storage mediums each storing one or more computer programs are also provided.

Abstract: An augmented reality-based lighting design method includes displaying, by an augmented reality device, a real-time image of a target physical area on a display screen. The method further includes displaying, by the augmented reality device, a lighting fixture 3-D model on the display screen in response to a user input, where the lighting fixture 3-D model is overlaid on the real-time image of the target physical area. The method also includes displaying, by the augmented reality device, a lighting pattern on the display screen overlaid on the real-time image of the target physical area, wherein the lighting pattern is generated based on at least photometric data associated with the lighting fixture 3-D model.

Abstract: Data characterizing a relative arrangement of vehicle occupants with respect to one another are continuously transmitted to display devices worn on the heads of the vehicle occupants. Virtual environments are displayed as a function of these data.

Abstract: There is provided an information processing apparatus that presents a virtual object useful for a user while suppressing a reduction in visibility of a real object. The information processing apparatus includes: an acquisition unit that acquires position information of a user and position information of a destination; and a display control unit that adds a first virtual object in association with a position of the destination within a visual field of the user on the basis of the position information of the user and the position information of the destination, in which the display control unit controls transparency of a predetermined virtual object within the visual field on the basis of the position information of the user.

Abstract: Methods and systems for integrating functionality of XR devices and non-XR devices are disclosed herein. An XR device can identify and obtain one or more content items being displayed on non-XR devices and create virtual objects associated with the content item in an artificial reality environment of the XR device. A user of the XR device can interact with the virtual objects, such as by moving the virtual objects to new locations within the artificial reality environment or interact with the virtual objects to view the content items. The user of the XR device can also provide one or more gestures or commands to send content items or identifiers of content items associated with the virtual objects to non-XR devices for display at the non-XR devices.

Abstract: Embodiments of the present disclosure provide a method and device for labelling a point of interest, a computer device, and a storage medium. The method includes the following. Image data to be labelled is obtained. The image data includes an image to be labelled and a collection location of the image to be labelled. Feature extraction is performed on the image to be labelled to obtain a first image feature of the image to be labelled. A first reference image corresponding to the image to be labelled is determined based on a similarity between the first image feature and a second image feature corresponding to each reference image in an image library. The point of interest of the image to be labelled is labelled based on a category of the first reference image and the collection location of the image to be labelled.

Abstract: Systems and methods for generating AR content to virtually alter a real-world item are provided. The real-world item may be a deformable item including one or more flexible surfaces. According to an embodiment, a method includes determining the relative positions of a plurality of points on the real-world item. Based on the relative positions of the plurality of points, a 3D model of a virtual item is distorted to produce a distorted 3D model of the virtual item. The distorted 3D model may have substantially the same shape as the real-world item. AR content is then generated based on the distorted three-dimensional model.

Abstract: A method, computer program product, and computing system for generating a three-dimensional model of at least a portion of a three-dimensional space incorporating an ACI system via a video recording subsystem of an ACI calibration platform; and generating one or more audio calibration signals for receipt by an audio recording system included within the ACI system via an audio generation subsystem of the ACI calibration platform.

Abstract: Apparatus, methods and systems of providing AR content are disclosed. Embodiments of the inventive subject matter can obtain an initial map of an area, derive views of interest, obtain AR content objects associated with the views of interest, establish experience clusters and generate a tile map tessellated based on the experience clusters. A user device could be configured to obtain and instantiate at least some of the AR content objects based on at least one of a location and a recognition.

Abstract: Systems and methods are described for extended reality environment interaction. An extended reality environment including an object is generated for display, and an eye motion is detected. Based on the detecting, it is determined whether the object is in a field of view for at least a predetermined period of time, and in response to determining that the object is in the field of view for at least the predetermined period of time, one or more items related to the object are generated for display in the extended virtual reality environment.

Abstract: Methods for observing a virtual reality environment of a virtual reality player is are provided. One method includes establishing, by a handheld device of an observer, a connection with a computer executing the virtual reality environment being presented to the virtual reality player using a head mounted display (HMD). Capturing, by a first camera of the handheld device, a current position of the virtual reality player in a real world space. Capturing, by a second camera of the handheld device, a face position of the observer. The face position being monitored to determine a viewing direction to a screen of the handheld device. The screen of the handheld device is configured to present the virtual reality environment and at least part of the virtual reality player interacting in the virtual reality environment.

Abstract: Systems, methods, apparatuses and non-transitory, computer-readable storage mediums are disclosed for generating AR self-portraits or “AR selfies.” In an embodiment, a method comprises: capturing, by a first camera of a mobile device, live image data, the live image data including an image of a subject in a physical, real-world environment; receiving, by a depth sensor of the mobile device, depth data indicating a distance of the subject from the camera in the physical, real-world environment; receiving, by one or more motion sensors of the mobile device, motion data indicating at least an orientation of the first camera in the physical, real-world environment; generating a virtual camera transform based on the motion data, the camera transform for determining an orientation of a virtual camera in a virtual environment; and generating a composite image data, using the image data, a matte and virtual background content selected based on the virtual camera orientation.

Abstract: A request is received to replicate an object in a scene description. A first data structure associating the object with object instances is generated to produce the object instances in the scene description, with each instance corresponding to a position that an instance of the object is to appear in the scene. The first data structure includes a first set of characteristics of the object instances that includes the position. A selection of an object instance and an instruction to customize a characteristic of the object instance is received. A customized characteristic of the object instance is produced by generation of a second data structure associating the customized characteristic with the first data structure. The object instances are rendered such that the customized characteristic of the second data structure supersedes the characteristic of the object in the first data structure.

Abstract: Systems, apparatuses and methods to provide image data, augmented with related data, to be displayed on a mobile computing device are disclosed. An example mobile device includes a camera to provide images of a scene from different angles to a server, at least one sensor to sense a position and an orientation of the camera, and a screen to present augmented reality data over the scene based on the position and the orientation of the camera and on a three-dimensional representation of the scene based on the images.

Abstract: There is provided an information processing apparatus, an information processing method, and a program capable of further diversifying a method of implementing a virtual object using the AR technology, the information processing apparatus including: a display control unit configured to perform a first process of displaying a virtual object superimposed on a real space at a position which is based on a real object detected in the real space or a second process of displaying the virtual object superimposed on the real space at a position which is not based on the real object detected in the real space.

Abstract: Disclosed are methods, systems, and articles of manufacture for managing and displaying web pages and web resources in a virtual three-dimensional (3D) space with an extended reality system. These techniques receive an input for 3D transform for a web page or a web page panel therefor. In response to the input, a browser engine coupled to a processor of an extended reality system determines 3D transform data for the web page or the web page panel based at least in part upon the 3D transform of the web page or the web page panel, wherein the 3D transform comprises a change in 3D position, rotation, or scale of the web page or the web page panel therefor in a virtual 3D space. A universe browser engine may present contents of the web page in a virtual 3D space based at least in part upon the 3D transform data.

Abstract: An information processing system includes a processing unit that detects a pointing action for a first user in a real space to specify a first object and an evaluation action on the first object by the first user based on a signal output from a sensor that senses actions in the real space and updates evaluation information of the first object specified by a detection result of the pointing action based on a detection result of the evaluation action.

Abstract: A display device includes: a display panel having a central part and a peripheral part surrounding the central part, the display panel comprising a plurality of pixels; and a lens array on the display panel, wherein: the lens array comprises a first lens on the central part and a second lens on the peripheral part; a first focal length of the first lens is smaller than a second focal length of the second lens; and each of the first lens and the second lens overlaps at least two pixels among the plurality of pixels.

Abstract: A system providing connectivity management is provided. The system comprises: a content management server configured to manage connectivity for a network; one or more central controllers configured to collect connectivity information for at least a portion of the network for use by the content management server; and at least one outlet having one or more ports for receiving one or more plugs, wherein connectivity information is communicated between the outlet and the central controller through one or more wireless communication interface.

Abstract: A device for displaying AR markings comprising a top and a base, with the top rotatably attached to the base, and the base configured to be held by a hand or placed on a fixed surface. The AR markings are positioned on the top such that when the top rotates with respect to the base, so do the AR markings. When the AR markings are scanned by an appropriate scanning and display device, such as a smart phone, a 3d image associated with the AR markings will be displayed on the display device as an augmented reality projection. When the top rotates with respect to the base, so too does the augmented reality projection.

Abstract: Provided are a system and method for displaying a virtual image through a head mounted display (HMD) device. A method by which a device displays a virtual image through an HMD device includes: receiving, from the HMD device, an object image of a real space captured through a camera included in the HMD device; determining a virtual image to be displayed at a particular position around the captured object; determining an operation of the virtual image corresponding to the object and the particular position; and providing the virtual image and operation information about the determined operation to the HMD device, wherein the provided virtual image is displayed through the HMD device.

Abstract: Systems, methods, and computer readable media directed to multi-user visual experiences such as interactive gaming experiences and artistic media experiences. A viewing electronic device includes a camera configured to capture images, a display, and a processor coupled to the camera and the display. The processor is configured to capture, with the camera, images of a rotating marker where the rotating marker is presented on a monitor of a remote device and to present, on the display, a visual experience where the visual experience has an adjustable feature. The processor detects a parameter of the rotating marker from the captured images that corresponds to the adjustable feature and updates the visual experience responsive to the detected parameter. The parameter may be one or more of an angle, speed of rotation, direction of rotation, color, or pattern of the rotating marker.

Abstract: A method of operating an augmented reality (AR) system includes capturing images of a first real-world workspace using a camera of a first head mounted AR device of the AR system being worn by a first user, processing the images using a first processor of the AR system to identify physical objects in the first real-world workspace and detect 3D positions of the identified physical objects in a 3D space corresponding to the first real-world workspace, rendering virtual objects representing the identified physical objects on the display of the first head mounted AR device at the respective 3D positions for the identified physical objects, manipulating a first one of the virtual objects using at least one hand-held controller of the AR system in a manner that mimics a performance of a first procedural task using the physical object associated with the first one of the virtual objects, recording the manipulation of the first one of the virtual objects that mimics the performance of the first procedural task as fi

Abstract: In a method for displaying a three dimensional interface on a device, a scene is displayed on a display of the device and a three dimensional user interface control with three dimensional effects is displayed on the display of the device, the three dimensional effects based on a virtual light source, a virtual camera, and a virtual depth of a three dimensional object relative to the scene. A change in the position of the device relative to the virtual light source and the virtual camera is detected. The three dimensional effects are dynamically changed based on the change in position of the device relative to the virtual light source and the virtual camera. Orientation of the virtual camera is dynamically changed to change the display of the scene and the display of the three dimensional user interface control to a new perspective based on the change in position of the device relative to the virtual light source and the virtual camera.

Abstract: This disclosure describes techniques that enable an augmentation controller to generate a real-time augmented multimedia stream. More specifically, the augmentation controller may receive a real-time multimedia stream from a client device, along with a representation of a related, virtual object. The real-time multimedia stream may be analyzed to identify a real-world object (i.e. a vehicle) along with a set of associated feature descriptors (i.e. vehicle wheels). Similarly, the augmentation controller may identify feature descriptors of the virtual object (i.e. wheelrim) and in doing so, overlay a representation of the virtual object on the real-world object based on an alignment of their respective feature descriptors. In this way, the augmentation controller may transmit to the client device, an augmented multimedia stream of the real-time multimedia stream with the virtual object overlaid onto the real-world object.

Abstract: A system and method for rendering perspective adjusted views of a virtual object in a real-world environment is provided. A registration code is associated with a mobile device and includes an identification of the mobile device. The registration code is displayed on the mobile device and a copy of the registration code captured by at least one other device is received. Positions of the device and the other device are synchronized based on a size and proportion of the registration code captured by the other device. A virtual object is accessed and provided for display on the device and the other device. The display of the virtual object is adjusted on the device and the other device based on the synchronization of the device and the other device.

Abstract: Provided is an information processing device that includes a recognition unit that recognizes, based on a result of sensing by a sensor unit corresponding to a viewpoint position of a user, a hand gesture performed by the user to move, closer to a first object, a second object so that the positional relation between the first object and the second object becomes a first positional relation. The information processing device further includes a correction unit that corrects the user's interocular parameter related to virtual object display on a display unit based on a result of recognition of the hand gesture when it is determined that a predetermined end condition is satisfied. At least one of the first object and the second object is a virtual object.

Abstract: A contact determination section determines whether or not there is any contact between a body part of a user and an object in a real world. An AR region determination section determines, as an augmented reality region, a region that corresponds to a partial space including a portion of the object in the real world, the portion having been contacted by the body part, and that is viewed from a viewpoint of the user. An AR generation section generates an augmented reality image in the augmented reality region in a shot image of the real world.

Abstract: To improve the social interaction and interaction with external objects for a user wearing an AR head-mounted display (HMD), cameras on the HMD detect that other users (“social targets”) are looking at and/or speaking to the user. The region around the social target in the AR user's view is determined and in response an optical occlusion system in the HMD is changed to allow more outside light to reveal the HMD user's eyes to the other users, The HMD optionally decreases the brightness of the virtual images presented in the HMD to allow the HMD user to improve his real world social and scene interaction.

Abstract: A method and an electronic device providing multi-views with an augmented reality (AR) device are provided. The method includes establishing a connection with at least one browsing device, receiving content source information from the at least one browsing device, transmitting the content source information to a second server to fetch AR content from the second server, and displaying, based on the content source information, the AR content on an AR plane which is separately viewable from a webpage displayed on a display of the at least one browsing device.

Abstract: A server machine modifies an augmented reality (AR) object in response to fulfillment of a condition. The machine provides, to a user device, object data that defines the AR object. The object data specifies a physical geolocation of the AR object, a presentation attribute of the AR object, a conditional modification program, and a trigger condition for execution of the conditional modification program. The object data causes the user device to present the AR object with a first appearance, located at the physical geolocation. The machine detects fulfillment of the trigger condition, and in response, the machine executes the conditional modification program. This modifies the object data by modifying the presentation attribute. The machine provides, to the user device, the modified object data, which causes the user device to present the AR object with a second appearance based on the modified presentation attribute.

Abstract: In displaying content superposed in a visual field of a vehicle occupant by a display device worn on the head of the vehicle occupant, vehicle regions are specified on which content must not be superposed. Alternatively, only content that is redundant with respect to content currently displayed within the vehicle regions may be superposed on the restricted vehicle regions.

Abstract: A rear-facing camera captures a live-action video image while a front-facing camera captures an image of a distributor. The avatar controller controls an avatar based on the image of the distributor captured by the front-facing camera, and move the position of the avatar in the real space coordinate system according to the position of the video synthesis device in the real space coordinate system. A synthesizer arranges the avatar in a predetermined position of a real space coordinate system and synthesizes the avatar with the live-action video image.

Abstract: Provided is an information processing device provided with an obtaining unit (211) that obtains first physical information of a first user present in a first space, and second physical information of a second user present in a second space, a virtual space generation unit (213) that generates a virtual space on the basis of the first physical information or the second physical information, as operation mode control unit (211) that switches an operation mode in a case where a trigger caused by at least any one of the first user or the second user occurs, and a control unit (210) that interlocks switch of the operation mode with at least one device present is the first space or at least one device present in the second space.