Abstract: Systems and methods herein describe a system for context triggered augmented reality. The proposed systems and methods receive first user input indicative of a selection of a user interface element corresponding to a recipient user, generate an augmented reality content item based on second user input from the first computing device, generate a contextual trigger for the generated augmented reality content item, the contextual trigger defining a set of conditions for presenting the generated augmented reality content item on a second computing device, generate a multi-media message comprising audio data recorded at the first computing device, detect at least one condition of the set of conditions being satisfied, and in response to detecting at least one of the set of conditions being satisfied, causing presentation of the augmented reality content item and the multi-media message at the second computing device.

Abstract: A travel control device carries out a travel control for a vehicle based on detected lane boundary lines. A first prescribed position, an absolute vehicle position and an absolute vehicle azimuth angle are stored while changing from a state where the lane boundary lines can be detected to a state where the lane boundary lines cannot be detected. A second prescribed position is stored while changing from a state in which the lane boundary lines cannot be detected to a state in which the lane boundary lines can be detected. The host vehicle is controlled to travel along a travel path connecting the first prescribed position and the second prescribed position where a current absolute position and a current absolute azimuth angle of the host vehicle do not deviate by a prescribed value or more from the stored absolute position and the stored absolute azimuth angle.

Abstract: A system and method for detecting a pose of an object is described. An augmented reality display device accesses first sensor data from an image sensor and a depth sensor of the augmented reality display device. The first sensor data includes a first plurality of images of an object and corresponding depth data relative to the augmented reality display device and the object. The augmented reality display device detects first features corresponding to the object by applying a convolutional neural network to the first sensor data, forms a plurality of training clusters based on the first features, and stores the plurality of training clusters in a training database.

Abstract: A method for positioning a virtual object in an image of a scene including a real object is disclosed. The method comprises receiving a signal from a short-range wireless transmitter arranged at a predetermined position relative to the real object and determining a location and orientation of the portable device relative to the short-range wireless transmitter based on the received signal. The image of the scene is displayed with the virtual object, wherein the position of the virtual object in the image is based on the determined location and orientation of the portable device relative to the short-range wireless transmitter and on the predetermined position of the short-range wireless transmitter relative to the real object. A system, a portable device and a computer program product is also disclosed.

Abstract: The subject technology selects a set of augmented reality content generators from a plurality of augmented reality content generators. The subject technology causes display, at a client device, of a graphical interface comprising a plurality of selectable graphical items. The subject technology receives, at the client device, a selection of a first selectable graphical item from the plurality of selectable graphical items, the first selectable graphical item comprising a first augmented reality content generator corresponding to a particular geolocation. The subject technology causes display, at the client device, at least one augmented content reality item generated by the first augmented reality content generator.

Abstract: A method implemented on an augmented reality (AR) device includes receiving an image of a document on the AR device. The image of the document includes one or more areas of obfuscated text. A marker on the document is identified. The marker is associated with an area of obfuscated text on the document. The marker is scanned using the AR device. When the user of the AR device is authenticated, a non-obfuscated image of the text associated with the marker is displayed on the AR device.

Abstract: A related information output apparatus 1 includes an acquisition unit 10 that acquires a captured image, a recognition unit 12 that recognizes one or more objects included in the captured image acquired by the acquisition unit 10, and an output unit 13 that outputs related information related to the objects recognized by the recognition unit 12. The output unit 13 may output the related information based on a combination of a plurality of the objects recognized by the recognition unit 12.

Abstract: A system includes a processor and a display. The processor is configured to: (a) receive, from a camera inserted into an eye of a patient, at least an optical image of at least a region-of-interest (ROI) of the eye, (b) receive, from a position tracking system (PTS), a position signal indicative of a position of a medical instrument treating the eye, (c) register the optical image and the PTS in a common coordinate system, and (d) estimate the position of the medical instrument in the optical image. The display is configured to visualize at least the ROI and the medical instrument.

Abstract: The method and system disclosed herein presents a method and system for capturing, by a camera device on a mobile robot moving in an environment, an image frame at a first location within a portion of the environment. In accordance with a determination that the image frame contains a first visual marker: the method includes determining a camera orientation angle from the camera device to a first anchor associated with the first visual marker based on the image frame; detecting, by a detector on the mobile robot, a first beacon signal emitted by the first anchor. The method includes determining a distance between the mobile robot and the first anchor; and determining an orientation angle of the mobile robot based on the camera orientation angle, and the distance between the mobile robot and the first anchor.

Abstract: The application provides an AR-based information displaying method and an AR apparatus, an electronic device and a storage medium, applicable to the technical field of computers. The method comprises: acquiring voice information and a user image of a user; identifying the voice information and extracting user characteristics; and when the user image matches the user characteristics, displaying, by an AR displaying device, target information associated with the user at a display position corresponding to the user image, wherein the target information comprises at least one of user information and voice associated information.

Abstract: Techniques for formulating queries and retrieving relevant results for 3D components in a virtual or augmented reality system. In an aspect, a user works with a 3D component using a workflow, and views the 3D component from one or more selected perspective views. Data associated with the workflow and the selected perspective views are transmitted to an online engine. The online engine may include a query formulation module for automatically forming a query based on the received workflow data and selected perspective views. The formulated query may be supplied to a search engine to retrieve online results based on relevance to the formulated queries. One or more most relevant online results may be seamlessly served to the user as part of the workflow.

Abstract: Disclosed herein are system, method, and computer program product embodiments for mobile device tracking system within a VR simulation. The method includes determining a position and orientation of a mobile device tracking module (e.g., case) attached to a mobile device, calculating a position and orientation of the of the mobile device based at least partially on a position and orientation of a tracking module, simulating a real world environment, generating a virtual visualization of the mobile device and rendering a VR simulation based at least partially on the position and orientation of the display screen of the mobile device. The position and orientation of the display screen provides a virtual position and orientation of the display screen relative to a virtual origin within the VR simulation.

Abstract: Methods and systems for a virtual experience where a user moves physically to move his or her ‘agent’, a position in the virtual space that determines which virtual content the user may interact with. If the virtual space contains obstacles, and the user moves such that the agent would intersect with one, either (a) move the correlation vector between the virtual space and real space to counteract the user's physical movement, preventing the user from penetrating the obstacle, or (b) allow the user to proceed, but become separated from his or her agent. Also described are applying these methods if the real space is too small to fit the virtual space, if the agent has a speed limitation that the user violates, how to direct the agent when it is separated from the user, and how to handle three-dimensional obstacles, real-world obstacles, and multiple users.

Abstract: Aspects of the present disclosure relate to haptic feedback systems and methods for use in head-worn computing systems. A head worn computer includes a frame adapted to hold a computer display in front of a user's eye, a processor adapted to present digital content in the computer display and to produce a haptic signal in coordination with the digital content display, and a haptic system including a plurality of haptic segments, wherein each of the haptic segments is individually controlled in coordination with the haptic signal.

Abstract: Disclosed herein is an improved AR technology for aligning virtual content with a real-world environment. In one aspect, the disclosed process may involve a first computing device receiving an indication that a second computing device detected a given QR pattern on a given strip of QR tape installed in a real-world environment. In response to receiving the indication, the first computing device may obtain installation information regarding a layout of the given strip of QR tape. Based at least on an identifier of the given QR pattern and the information regarding the layout of the given strip of QR tape, the first computing device may then determine a position and orientation of the second computing device, align virtual content on the real-world environment, and instruct the second computing device to present a view of the real-world environment that has the aligned virtual content superimposed onto the real-world environment.

Abstract: An information display system, an information display method, and a processing device are disclosed. The system includes a plurality of light-transmissive displays and a plurality of processing devices connected and communicating with each other through gateways. A first processing device is selected from the processing devices according to position information of a user, and determines sight line information of the user according to the position information and posture information of the user. A second processing device different from the first processing device calculates a target coordinate of a target. The first processing device selects a third processing device from the processing devices according to the sight line information of the user. The third processing device determines display position information of a virtual object according to a user coordinate and the object coordinate, and controls one of the displays to display the virtual object according to the display position information.

Abstract: A system comprises a frame, a mirror coupled to the frame, an electronic display device coupled to the frame, a camera coupled to the frame, one or more processors, and one or more memory devices. The one or more processors are configured to execute instructions that are stored in the one or more memory devices. The one or more processors executing the instructions causes the system to continuously display, on the electronic display device, a real-time video feed of a user within a field of view of the camera; identify and object within the field of view of the camera; and, subsequent to the identification, modify the continuously displayed real-time video feed based on at least the identified object.

Abstract: Systems, computer program products, and methods are described herein for authorizing a database model using distributed ledger technology. The present invention is configured to receive, from a computing device of a user, a physical database model; determine one or more data sources; initiate a second machine learning model on the one or more data sources and the physical database model; determine at least one data source that is likely to be triggered when populating a database created using the physical database model; generate a new transaction object for the physical database model; broadcast the new transaction object for the physical database model to the at least one data source in a distributed ledger for validation; receive a validation of the physical database model; and authorize a generation of a database using the physical database model based on at least receiving the validation.

Abstract: A vehicle control device includes a peripheral recognition unit configured to recognize a peripheral status of a vehicle including a position of a traffic participant present in a periphery of the vehicle on the basis of an output of an in-vehicle device, an estimation unit configured to estimate a peripheral attention ability of the traffic participant on the basis of an output of the in-vehicle device, and a risk area setting unit configured to set a risk area of the traffic participant on the basis of a result of the estimation performed by the estimation unit.

Abstract: A head-mounted device may have a display that displays computer-generated content for a user. The head-mounted device may have an optical system that directs the computer-generated image towards eye boxes for viewing by a user. The optical system may be a see-through optical system that allows the user to view a real-world object through the optical system while receiving the computer-generated image or the optical system may include a non-removable lens and a removable vision correction lens through which an opaque display is viewable. The optical system may include a removable lens. The removable lens may serve as a custom vision correction lens to correct for a user's vision defects. The optical system may have a projection bias lens that places computer-generated content at one or more desired virtual image distances and a corresponding compensation bias lens.

Abstract: A system and method of performing a payment transaction in an augmented user experience displayed to a user of an augmented reality device. The system includes receiving a request for user authentication from a user, authenticating the user for performing payment transactions via a merchant application executed on the augmented reality device, receiving a request for payment to a merchant for an item offered for sale to the user in an augmented environment presented by the merchant application, processing payment in the payment amount between a payment account of the user and a merchant account, wherein the request for payment is performed without the user exiting the merchant application.

Abstract: Disclosed is an electronically-controlled automatic light-shading device, comprising a first glass substrate, a light-shading coating, a polarizing element and a second glass substrate. An image module and a photosensitive element adjacent thereto are embedded in the first glass substrate. The first glass substrate has a first surface on the opposite side to an external light source. The light-shielding coating is applied on the first surface. The polarizing element is disposed on the light-shielding coating. The second glass substrate has a second surface facing the first surface. A plurality of spacers in contact with the polarizing element are disposed on the second surface, and an optical fiber element is disposed in each spacer.

Abstract: An apparatus for performing image searches including a camera, storage devices storing a set of instructions, and a processor coupled to the at least one storage device and the camera. The instructions configure the at least one processor to perform operations including identifying attributes of the captured image using a classification model; identifying first results based on the identified attributes; selecting a subset of first results based on corresponding probability scores, generating a first graphical user interface including interactive icons corresponding to first results in the subset, an input icon, and a first button. The operations may also include receiving a selection of the first button, performing a search to identify second results, and generating a second graphical user interface displaying the second results.

Abstract: The present invention provides an eye tracking system including: a pupil sensing unit sensing a pupil of a user by at least one sensor embedded in smart glasses; a display unit including smart lenses to which a plurality of target objects is floated; an eye tracking unit tracking an eye through at least one sensor and acquiring eye state information of the user for the plurality of target objects; and an input unit performing an input after selecting a specific object in which the eye of the user stays for a predetermined time or more among the plurality of target objects as an input object based on the eye state information, in which at least one of a size, a color, and a movement direction of the input object is changed based on the eye state information of the user.

Abstract: Systems, methods, and non-transitory computer readable media including instructions for performing duty cycle control operations for a wearable extended reality appliance are disclosed. Performing the duty cycle control operations includes receiving data representing virtual content in an extended reality environment associated with a wearable extended reality appliance; identifying in the extended reality environment a first display region and a second display region separated from the first display region; determining a first duty cycle configuration for the first display region; determining a second duty cycle configuration for the second display region, wherein the second duty cycle configuration differs from the first duty cycle configuration; and causing the wearable extended reality appliance to display the virtual content in accordance with the determined first duty cycle configuration for the first display region and the determined second duty cycle configuration for the second display region.

Abstract: An augmented reality-based content authoring tool is presented. A content author arranges machine-recognizable markers in a physical environment. A computing device operating as the authoring tool recognizes the markers and their arrangement based on a captured digital representation of the physical environment. Once recognized, augmented reality primitives corresponding to the markers can be bound together via their primitive interfaces to give rise to a content set. The individual primitives and content set are instantiated based on the nature of the marker's arrangement.

Abstract: Systems, methods, and non-transitory computer readable media configured for enabling content sharing between users of wearable extended reality appliances are provided. In one implementation, the computer readable medium may be configured to contain instructions to cause at least one processor to establish a link between a first wearable extended reality appliance and a second wearable extended reality appliance. The first wearable extended reality appliance may display first virtual content. The second wearable extended reality appliance may obtain a command to display first virtual content via the second wearable extended reality appliance, and in response, this content may be transmitted and displayed via the second extended reality appliance. Additionally, the first wearable extended reality appliance may receive second virtual content from the second wearable extended reality appliance, and display said second virtual content via the first wearable extended reality appliance.

Abstract: A processor may receive user command generated by a user. A processor may analyze user information for one or more official activities. The one or more official activities are based, at least in part, on a security policy. A processor may identify the user information is associated with the one or more official activities. A processor may prevent, responsive to determining the user command is associated with the one or more official activities, the one or more personal devices from collecting the user command.

Abstract: One implementation forms a composited stream of computer-generated reality (CGR) content using multiple data streams related to a CGR experience to facilitate recording or streaming. A media compositor obtains a first data stream of rendered frames and a second data stream of additional data. The rendered frame content (e.g., 3D models) represents real and virtual content rendered during a CGR experience at a plurality of instants in time. The additional data of the second data stream relates to the CGR experience, for example, relating to audio, audio sources, metadata identifying detected attributes of the CGR experience, image data, data from other devices involved in the CGR experience, etc. The media compositor forms a composited stream that aligns the rendered frame content with the additional data for the plurality of instants in time, for example, by forming time-stamped, n-dimensional datasets (e.g., images) corresponding to individual instants in time.

Abstract: Provided is a personal authentication device based on an auditory brainstem response signal. The personal authentication device includes a signal generator, a signal acquirer, and a signal processor. The signal generator may output an auditory stimulus to an ear of a user, using a sound generator. The signal acquirer may acquire a first reference potential corresponding to a first interval, from a first electrode in close contact with the other ear of the user, and acquire a first evoked potential corresponding to the first interval from a second electrode in close contact with the ear, based on the first reference potential. The signal processor may generate a first auditory brainstem response signal, based on the first reference potential and the first evoked potential, generate authentication data for the user from the first auditory brainstem response signal, and compare the authentication data with registration data to authenticate the user.

Abstract: Systems and methods are provided for displaying supplemental content for print media using augmented reality. A user profile for a user of an augmented reality device is determined. Content of the print media is searched to identify a first portion of the print media that matches the user profile and a second portion of the print media that does not match the user profile. Supplemental content is obtained based on content of the first portion of the print media. A display of the supplemental content is positioned over the second portion of the print media.

Abstract: Eye and hand tracking systems in head-mounted display (HMD) devices are arranged with lensless camera systems using optical masks as encoding elements that apply convolutions to optical images of body parts (e.g., eyes or hands) of HMD device users. The convolved body images are scrambled or coded representations that are captured by a sensor in the system, but are not human-recognizable. A machine learning system such as a neural network is configured to extract body features directly from the coded representation without performance of deconvolutions conventionally utilized to reconstruct the original body images in human-recognizable form. The extracted body features are utilized by the respective eye or hand tracking systems to output relevant tracking data for the user's eyes or hands which may be utilized by the HMD device to support various applications and user experiences. The lensless camera and machine learning system are jointly optimizable on an end-to-end basis.

Abstract: Devices and methods for performing an interventional vascular procedure with visual guidance using one or more optical head mounted displays are disclosed. Devices and methods for compensating the display of an optical head mounted display for cardiac and/or respiratory motion are disclosed.

Abstract: An augmented reality customer interaction system includes a transparent panel having a first side and a second side that is opposite to the first side, and a camera device configured to capture visual data from an area adjacent to the second side of the transparent panel. The visual data includes identifying features of a customer located in the area with respect to the second side of the transparent panel. The system further includes a projection system configured to project information on the first side of the transparent panel. The information projected on the first side of the transparent panel may include customer interaction data retrieved from a data store based on the identifying features of the customer.

Abstract: A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).

Abstract: A method and system for generating attention pointers, including: displaying, in a display of a mobile device, an object within and outside a field of view (FOV) of an user wherein the object outside the FOV are real objects; monitoring, by a processor of the mobile device, for a change in the object within and outside the FOV; in response to a change, generating by the processor one or more attention pointers within the FOV of the user for directing user attention to the change in the object which is either inside or outside the FOV; and displaying, by the processor, on a virtual screen within the FOV to the user, the one or more attention pointers wherein the one or more attention pointers are dynamically configured to interact with the user in response to detections based on a movement of the user or the object within or outside the FOV of the user.

Abstract: A head-mounted display includes a display unit, a head support, and cameras. The display unit includes a display for outputting graphical content. The head support is coupled to the display unit for engaging a head of a user to support the display unit with the display in front of an eye of the user. The cameras are each coupled to one of the display unit or the head support. The cameras have camera fields of view that overlap horizontally to cooperatively provide the head-mounted display with a head-mounted display field of view of 360 degrees horizontal.

Abstract: Systems, apparatus, and methods of context-based management of tokens are described. In an example, a geographical location of a device is used as one possible context. This location can correspond to a physical location associated with an AR virtual object container. This container can be associated with a set of virtual object container information applicable to the context, such as to the device's location. Based on separately maintained virtual object information, virtual objects to be shown as being available from the container are determined. Each of such virtual objects can be associated with a set of tokens. In an AR session, the container and the virtual objects are presented. An interaction with the container or a virtual object can result in associating a relevant set of tokens with a user account by recording information about the container, the virtual object, the user account, and/or the context(s).

Abstract: In one example, a method performed by a processing system including at least one processor includes identifying an environment surrounding a user of an augmented reality display, identifying a relative location of the user within the environment, determining a field of view of the augmented reality display, identifying an individual within the field of view, querying a data source for information related to the individual, and modifying the augmented reality display to present the information related to the individual.

Abstract: A display apparatus and an operating method thereof obtain at least one first sensor data of a first point in time, receive a first image frame including a first augmented reality (AR) object corresponding to the first point in time, the first AR object generated by a first processor, based on the first sensor data, obtain second sensor data of a second point in time subsequent to the first point in time, correct the first AR object to a second AR object matched to the second point in time, based on the second sensor data, and display a second image frame including the second AR object.

Abstract: A method for spatial messaging includes the steps of setting, as a space for messaging, a location designated by a user as a virtual space assigned a unique coordinate value corresponding to a location in the real world; and collecting, as a message for the space, at least one content authored by another user who shares the corresponding space through a coordinate value of the space.

Abstract: A system receives, from an augmented reality device, a first image of a web application, where the first image shows a first element of the web application. The system receives eye tracking information that indicates eye movements of a user as the user is looking at different elements of the web application. The system determines that the user is looking at a first location coordinate on the first image of the web application. The system determines that the first element is located at the first location coordinate. The system identifies first element attributes associated with the first element. The system generates an augmented reality message comprising the first element attributes. The system generates an augmented reality display in which the augmented reality message is presented as a virtual object. The system transmits the augmented reality display to the augmented reality device.

Abstract: Augmented reality systems and methods for creating, saving and rendering designs comprising multiple items of virtual content in a three-dimensional (3D) environment of a user. The designs may be saved as a scene, which is built by a user from pre-built sub-components, built components, and/or previously saved scenes. Location information, expressed as a saved scene anchor and position relative to the saved scene anchor for each item of virtual content, may also be saved. Upon opening the scene, the saved scene anchor node may be correlated to a location within the mixed reality environment of the user for whom the scene is opened. The virtual items of the scene may be positioned with the same relationship to that location as they have to the saved scene anchor node. That location may be selected automatically and/or by user input.

Abstract: A mobile device is fitted with an extended reality (XR) software application program executing on a processor within an XR system, and optionally a camera. Via the XR software application program, various techniques are performed for interacting with a physical object via the XR environment. In particular, the XR software application program generates and displays visual representations of real-time metric data received from a data intake and query system along with auxiliary data received from an asset management system. In addition, the XR software application program detects user interactions with the XR environment. In response, the XR software application generates messages directed to the asset management system. The messages include commands to update the auxiliary data associated with the physical object.

Abstract: An information display method, including: obtaining target information and an actual image when an information display signal is received, the actual image being an image obtained by photographing a real environment by using a camera; generating an augmented reality (AR) image according to the target information and the actual image; and displaying the AR image in a user interface, an AR element in the AR image being used for displaying the target information. In this application, an AR image is generated according to the obtained target information and actual image, and the AR image is displayed on a user interface, where the AR image includes an AR element configured to display the target information.

Abstract: A method for providing augmented reality content includes identifying a customer at a customer interaction location, retrieving customer-specific interaction data, determining locations of a customer service representative in an area adjacent to a first side of a transparent panel and the customer in an area adjacent to a second side of the transparent panel, and projecting an augmented reality element to the first side of a transparent panel, the augmented reality element displaying the customer-specific interaction data on the first side of the transparent panel.

Abstract: A method for creating and storing a captured image and associated spatial data and augmented reality (AR) data in a file that allows subsequent manipulation and processing of AR objects is disclosed. In embodiments, one or more frames are extracted from a video stream, along with spatial information about the camera capturing the video stream. The one or more frames are analyzed in conjunction with the spatial information to calculate a point cloud of depth data. The one or more frames are stored in a file in a first layer, and the point cloud is stored in the file in a second layer. In some embodiments, one or more AR objects are stored in a third layer.

Abstract: In one implementation, a method of including a person in a CGR experience or excluding the person from the CGR experience is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes, while presenting a CGR experience, capturing an image of scene; detecting, in the image of the scene, a person; and determining an identity of the person. The method includes determining, based on the identity of the person, whether to include the person in the CGR experience or exclude the person from the CGR experience. The method includes presenting the CGR experience based on the determination.

Abstract: A method for generating the at least one highway exit indicator, the method may include receiving video information and location information obtained during driving sessions of the plurality of vehicles; determining, based on the location information, multiple suspected highway exit events; selecting video information segments, wherein each selected video information segment is acquired before a suspected highway exit event and in timing proximity to the suspected highway exit event; and applying a machine learning process on at least some of the selected video information segments to find the at least one highway exit indicator.

Abstract: An AR system to assist with installation of wire harnesses identifies landmarks in the installation environment and continuously orients 3D models of the wire harness components with reference to the landmarks in 3D space. The 3D models are rendered in an AR device to correspond to the placement in 3D space with respect to the landmarks. The installation process is broken down into steps and sub-steps. An installer may install the wire harness according to predefined steps and be presented with higher granularity instructions upon request. The higher granularity sub-steps may include additional 3D models.