Abstract: A method is performed at an electronic device including one or more processors, a non-transitory memory, and a first input device. The method includes detecting, via the first input device, an input directed to a first location within a physical environment. The first location is identified by an extremity tracking function based on the input. The method includes determining an interaction event based on a function of a semantic identifier that is associated with a portion of the first location. The method includes presenting computer-generated content that is a function of the interaction event.

Abstract: A removable lens for glasses includes an electrochromic layer adapted to transition between at least two states of a transparency level dependent on an electrical condition applied to the electrochromatic layer and an electrical connection element physically adapted to make an electrical connection with the electrochromic layer, wherein the electrical connection element is positioned to align with a variable-power electrode mounted on the glasses such that, when the removable lens is mounted on the glasses, the variable-power electrode and the electrical connection element make the electrical connection between the variable-power electrode and an electrically conductive layer of the electrochromic layer.

Abstract: A system includes a vehicle cockpit and a processor. The vehicle cockpit includes imperceptible cockpit reference fiducials and a head wearable display (HWD) device. The imperceptible cockpit reference fiducials are imperceptible to a naked eye of a user in the vehicle cockpit. The HWD device includes a display and an optical sensor configured to: capture images of the imperceptible cockpit reference fiducials; and output optical sensor image data. The processor is configured to: receive the optical sensor image data; and based at least on the optical sensor image data, determine a position and orientation of the head wearable display device. The display is configured to display the images aligned with the field of view of the user based at least on the determined position and the determined orientation of the head wearable display device.

Abstract: Embodiments of the present invention provide computer-implemented methods, computer program products and computer systems. Embodiments of the present invention can, in response to receiving a request, identify a core component from source material based on topic analysis. Embodiments of the present invention can then generate three-dimensional representations of physical core components associated with the request. Finally, embodiments of the present invention then render the generated three-dimensional representations of the physical core components over the physical core components.

Abstract: A system for displaying customized mobile content to users at a venue via a user device, includes a server system in conjunction with a plurality of machine-readable codes mounted within the venue. The server system performs the following steps: providing mobile content in a database; collecting user data associated with the user; receiving a request from one of the user devices, the request being generated by scanning one of the machine-readable codes with the user device; determining the user data associated with the user who scanned the machine-readable code; selecting, based upon the user data, customized mobile content from the mobile content in the database; and providing the customized mobile content to the user device.

Abstract: An apparatus for controlling an augmented reality includes: a communicator configured to communicate with an external device; and a processor configured to identify, via the communicator, a sensing device which is an external device, generate virtual image data based on sensing data received from the identified sensing device, and provide the virtual image data to an image display device.

Abstract: There are provided systems and methods for tagging objects in augmented reality to track object data. A user may capture visual data utilizing a device at a location, where the visual data includes one or more objects that the user may wish to associate with a purchase or sell offer. A device may perform object recognition to identify the object and an identifying characteristic of the object. A service provider may then receive offer details and may generate a graphical object or other virtual data that can be displayed with the object in an augmented reality experience. Later, when the object is identified in another augmented reality experience on a different device, the virtual graphic may be displayed, which may show the offer for the object. Further, the augmented reality may be used to process a transaction for the object and track the object.

Abstract: A system and method and for identifying a template for a document includes receiving a request to identify the template from among a plurality of available templates, the plurality of templates being templates that are available for use in an application. After receiving the request, the content and structure of the document are encoded into one or more embedding representations via a trained document encoder and the embedding representations are compared to a plurality of template representations, each of the plurality of template representations being a representation of content and structure of one of the plurality of templates to identify one of the plurality of the templates as corresponding to the document. The identified template is then provided for display as a recommended template.

Abstract: A system and method for visualizing multiple datasets in a virtual 3-dimensional interactive environment. Multiple datasets may be related and virtually cast as 3-dimensional type structures. User interfaces, such as game controllers or headsets, may be used to present the dataset from differing perspectives including the appearance of moving through the data. Certain embodiments provide for mirror image views that allow for presentation of higher order datasets. Other embodiments provide for animation or motion indicia to show how the data is changing and the results on the display. The datasets may represent physical areas or virtual areas as well as demographic, sensors and financial information.

Abstract: A method for an image capturing device includes: acquiring a first image of a target object; extracting coordinate information of a first 2D key point corresponding to the target object; acquiring a second image of the target object; acquiring coordinate information of a second 2D key point in the second image according to the coordinate information of the first 2D key point, the first 2D key point and the second 2D key point being the same key point in the target object; acquiring coordinate information of a 3D key point corresponding to the target object, the 3D key point being a key point obtained after a triangulation operation; and determining camera posture information of the second image according to the coordinate information of the first 2D key point, the coordinate information of the second 2D key point, the coordinate information of the 3D key point, and a 3D-2D projection.

Abstract: In an approach for dynamically adjusting parallel reality (PR) displays, a processor configures a viewing event. A processor receives data from data collecting devices located throughout a location of the viewing event. A processor classifies a crowd of the viewing event into at least two partitions using a learning-based neural network that ingests the data. A processor selects content to be displayed to each of the at least two partitions. A processor enables a PR display to simultaneously display the content to each of the at least two partitions.

Abstract: Disclosed herein are systems and methods for presenting and annotating virtual content. According to an example method, a virtual object is presented to a first user at a first position via a transmissive display of a wearable device. A first input is received from the first user. In response to receiving the first input, a virtual annotation is presented at a first displacement from the first position. A first data is transmitted to a second user, the first data associated with the virtual annotation and the first displacement. A second input is received from the second user. In response to receiving the second input, the virtual annotation is presented to the first user at a second displacement from the first position. Second data is transmitted to a remote server, the second data associated with the virtual object, the virtual annotation, the second displacement, and the first position.

Abstract: A method for quasi-random placement of a virtual item in an XR space includes: accessing a previously generated spatial mapping mesh (SMM) of the XR space; compiling a record from the SMM of open spaces between surfaces of physical elements in the XR space, with corresponding positions and dimensions; selecting from the open spaces: a spawn position for a virtual character, and a random set of other positions, filtering the random set to form a subset. The method then performs a collision analysis to assign a score to each position in the subset partly based on accessibility to that position for the virtual character beginning from the spawn position; and places the virtual item at a position in the subset having a score as high or higher than all other positions in the subset. The method is carried out before user interaction with any virtual element in the XR space.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for imaging scenes of three-dimensional (3-D) environments using a virtual reality system. A request to display a target three-dimensional environment in a virtual reality system is received. An image of a scene of the target three-dimensional environment is generated. The image of the scene includes features distributed within the scene of the target three-dimensional environment according to a distance from a viewing point of the scene of the target three-dimensional environment. A request to display a modified image of the scene of the target three-dimensional environment is received. The modified image of the scene of the target three-dimensional environment is generated by adding auxiliary data associated with the plurality of features relative to the distance from the viewing point of the scene of the target three-dimensional environment to each of the plurality of features.

Abstract: To present a pickup location in a transportation service, a server device receives a request for ridesharing services from a rider's current location to a drop-off location. The server device broadcasts the request to drivers and receives an indication of acceptance of the request from one of the drivers. Then the server device provides a pickup location to the rider and the driver which is presented in a map display of the rider and driver's client devices. When the rider or driver's current location is within a threshold distance of the pickup location, the corresponding client device switches to an augmented reality mode and presents a camera view from a camera of the client device depicting real-world imagery. The server device provides information to the corresponding client device to present in the augmented reality mode overlaying the real-world imagery to assist the rider or driver in finding the pickup location.

Abstract: The present disclosure relates to techniques for providing tangibility visualization of virtual objects within a computer-generated reality (CGR) environment, such as a CGR environment based on virtual reality and/or a CGR environment based on mixed reality. A visual feedback indicating tangibility is provided for a virtual object within a CGR environment that does not correspond to a real, tangible object in the real environment. A visual feedback indicating tangibility is not provided for a virtual representation of a real object within a CGR environment that corresponds to a real, tangible object in the real environment.

Abstract: An image processing method and apparatus for augmented reality, an electronic device and a storage medium, including: acquiring a target image in response to an image acquiring instruction triggered by a user, where the target image includes a target object; acquiring an augmented reality model of the target object, and outputting the augmented reality model in combination with the target object; acquiring target audio data selected by the user, and determining an audio feature with temporal regularity according to the target audio data; and driving the augmented reality model according to the audio feature and a playing progress of the target audio data when outputting the target audio data.

Abstract: A head-mounted device may have a head-mounted housing. The head-mounted housing may have rear-facing displays that display images for a user. The images are viewable from eye boxes while the head-mounted device is being worn by the user. A publicly viewable front-facing display may be mounted on a front face of the head-mounted housing. A display cover layer may overlap a pixel array associated with the front-facing display. An inactive border area of the display cover layer may overlap optical components. Shutter systems may be provided between the display cover layer and the optical components to selectively block light. Actuators may move shutter members or other movable members to cover and uncover the optical components. A removable cover may be used to protect the display cover layer and may have portions in different areas that have different optical properties.

Abstract: A user may be directed to the nail application (e.g., a mobile application that provides the various augmented reality and booking features discussed herein) via interactions with a social media site. For example, a user may browse Facebook and see a friend's post or advertisement regarding a nail design, or more generally, some artwork that the user feels would look nice as a nail design. The user may select the artwork and be redirected to the nail application to begin an augmented reality testing of one or more nail artwork designs and/or schedule treatments. In some embodiments, the social media website utilizes an API provided by the data center (e.g. the entity that coordinates the treatment booking process and provides augmented reality software to the users) so that the user can “Touch to try-on” a particular design team in social media by clicking on the design.

Abstract: The described positional awareness techniques employing sensory data gathering and analysis hardware with reference to specific example implementations implement improvements in the use of sensors, techniques and hardware design that can enable specific embodiments to provide positional awareness to machines with improved speed and accuracy. The sensory data are gathered from an operational camera and one or more auxiliary sensors.

Abstract: A system and method for generating a virtual content to a physical object is described. A processor includes an augmented reality application. The augmented reality application creates virtual content at the head mounted device, and associates the virtual content with predefined conditions based on data from sensors embedded in the head mounted device at a time of creation of the virtual content. The virtual content is displayed in a display of the head mounted device in response to sensor data satisfying the predefined conditions.

Abstract: A data processing apparatus includes avatar monitoring circuitry to receive gaze data for a first user associated with a first avatar in a virtual reality environment, the gaze data for the first user indicative of a gaze point for the first user with respect to the virtual reality environment, in which the avatar monitoring circuitry is configured to select one or more objects in the virtual reality environment in dependence upon the gaze data for the first user and to store first avatar information for the first avatar indicative of one or more of the selected objects, input circuitry to receive gaze data for a second user indicative of a gaze point for the second user with respect to the virtual reality environment, and processing circuitry to generate one or more images for the virtual reality environment for display to the second user, in which the processing circuitry is configured to: select the first avatar in dependence upon the gaze point for the second user with respect to the first avatar; and gen

Abstract: Feature information is received from a client device and describes extended reality scene(s) in an extended reality (ER) environment at the client device. An ER description is formed in an ER description format corresponding to the feature information and is stored. Some of the stored ER description format is provided in a representational format upon request of the client device or other client devices viewing the one or more ER scenes and assisting the positioning of corresponding client devices in the ER environment. A client device captures environmental visual data and generates feature information, describing ER scene(s) in an extended reality environment at the client device, from the environmental visual data, and sends the generated feature information toward a server. A client device can localize itself in a 3D environment of an ER description, and generate ER anchors and objects, link them, and send them to a server.

Abstract: A method or apparatus provides binaural sound that originates to a person at a location inside a field of view (FOV) provided to the person by an electronic device. The electronic device determines when the location of the binaural sound moves outside the FOV and assists the person in determining the location of the binaural sound. A display of the electronic device displays a visual indication that shows a direction to the location of the binaural sound outside the FOV.

Abstract: A head-mountable device can include multiple microphones for directional audio detection. The head-mountable device can also include a speaker for audio output and/or a display for visual output. The head-mountable device can be configured to provide visual outputs based on audio inputs by displaying an indicator on a display based on a location of a source of a sound. The head-mountable device can be configured to audio outputs based on audio inputs by modifying an audio output of the speaker based on a detected sound and a target characteristic. Such characteristics can be based on a direction of a gaze of the user, as detected by an eye sensor.

Abstract: A display frame position/orientation adjustment is made to a computer display data set before it is displayed on a display screen (for example, a display screen of a user's smartphone). The offsets the display from the display frame defined by the edges of the display screen. The user tends to counter these offsets by moving from a relatively poor body posture and into a good body posture.

Abstract: Among other things, techniques are described for expressive vehicle systems. These techniques may include obtaining, with at least one processor, data associated with an environment, the environment comprising a vehicle and at least one object; determining an expressive maneuver including a deceleration of the vehicle such that the vehicle stops at least a first distance away from the at least one object and the vehicle reaches a peak deceleration when the vehicle is a second distance away from the at least one object; generating data associated with control of the vehicle based on the deceleration associated with the expressive maneuver; and transmitting the data associated with the control of the vehicle to cause the vehicle to decelerate based on the deceleration associated with the expressive maneuver.

Abstract: Aspects of an embodiment of the present invention disclose a method, computer program product, and computing system that receives, at a processor, a review record comprising an image of an artifact and reference links associated with the artifact, wherein the image is recorded with a smart lens. The present invention also assigns the review record to a reviewer based on a qualification associated with the reviewer and the review record and receives a comment from the reviewer. The method performed in embodiments of the present invention also includes relaying the comment to the smart lens.

Abstract: The disclosed computer-implemented method may include establishing a virtual boundary for a virtual-world environment in reference to a real-world environment, and determining whether the virtual boundary requires a correction. The method may include providing, in response to determining that the virtual boundary requires the correction, and alert and, in response to the alert, receiving a request from a user to modify the virtual boundary. The method may further include monitoring, in response to the request from the user, an orientation of a direction indicator to generate orientation data, and modifying the virtual boundary based on the orientation data. Various other methods, systems, and computer-readable media are also disclosed.

Abstract: Systems and methods for displaying a cursor and a focus indicator associated with real or virtual objects in a virtual, augmented, or mixed reality environment by a wearable display device are disclosed. The system can determine a spatial relationship between a user-movable cursor and a target object within the environment. The system may render a focus indicator (e.g., a halo, shading, or highlighting) around or adjacent objects that are near the cursor. The focus indicator may be emphasized in directions closer to the cursor and deemphasized in directions farther from the cursor. When the cursor overlaps with a target object, the system can render the object in front of the cursor (or not render the cursor at all), so the object is not occluded by the cursor. The cursor and focus indicator can provide the user with positional feedback and help the user navigate among objects in the environment.

Abstract: An augmented reality surgical system includes a head mounted display (HMD) with a see-through display screen, a motion sensor, a camera, and computer equipment. The motion sensor outputs a head motion signal indicating measured movement of the HMD. The computer equipment computes the relative location and orientation of reference markers connected to the HMD and to the patient based on processing a video signal from the camera. The computer equipment generates a three dimensional anatomical model using patient data created by medical imaging equipment, and rotates and scales at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers, and further rotate at least a portion of the three dimensional anatomical model based on the head motion signal to track measured movement of the HMD. The rotated and scaled three dimensional anatomical model is displayed on the display screen.

Abstract: A system for generating an augmented reality image using an initial image comprising a reference element, the system comprising: an image analyzer for identifying the reference element within the initial image; an image generating unit for: generating a simulation image of a scene of a virtual environment based on the identified reference element, an outline of the simulation image corresponding to the reference outline; overlaying the initial image with the simulation image to obtain the augmented reality image; and providing the augmented reality image for display.

Abstract: An active heads up display system includes a driver monitoring system, a gateway circuit, and a heads up display. The driver monitoring system tracks a gaze direction of a user, calculates track values in response to a field of view of the user based on the gaze direction, and transmits the track values on a first bus. The gateway circuit receives the track values from the first bus, translates the track values to command values, and transmits the command values on a second bus. The heads up display receives the command values from the second bus, generates a graphical presentation viewable by the user, projects the graphical presentation at a current position in front of the user, and adjusts the current position to an updated position centered within the field of view of the user in response to the command values.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and method for including a video feed in a message thread. The program and method provide for determining that a message thread is being concurrently displayed on a first device associated with a first user and on a second device associated with a second user, the first user and the second user corresponding to contacts within a messaging application; and transmitting, in response to the determining, image data captured on the first device to the second device, for display within the message thread on the second device.

Abstract: A method for displaying information to the occupants of a vehicle includes setting a display-hosting region on a window of a vehicle and obtaining a number of driving parameters of the vehicle. Points of interest (POI) information, such as landmarks, and commercial and other desired locations, which satisfy a number of preset conditions according to the number of the driving parameters are obtained. Facial images and face parameters of occupants are obtained, and POI and associated information can be displayed according to the number of facial parameters. The POI information is displayed to one or more display regions.

Abstract: Systems and methods are described for improved color rendering. In some embodiments, video is captured of a real-world scene, and a region of screen content is detected in the captured video. A processor selectively applies a screen-content color transformation on the region of screen content to generate processed video, and the processed video is displayed substantially in real time, for example on a video-see-through head-mounted display. The screen-content color processing may be different for different types of external displays. The screen-content color processing may also be determined based at least in part on illumination conditions. In some embodiments, the color of displayed virtual objects is adjusted based on visual parameters of the screen content.

Abstract: According to various implementations, a method is performed at a first electronic device with a non-transitory memory and one or more processors. The method includes determining a reference location in a three-dimensional space based on a feature. The feature is generated by a second electronic device. The method includes obtaining, for the reference location, first reference coordinates in an augmented reality coordinate system of the first electronic device and second reference coordinates in an augmented reality coordinate system of the second electronic device. The method includes determining a coordinate transformation based on a function of the first reference coordinates and the second reference coordinates. The method includes synchronizing an augmented reality coordinate system of the first electronic device with an augmented reality coordinate system of the second electronic device using the coordinate transformation.

Abstract: Low-power eye tracking for detecting position and movements of a user's eyes in a head-mounted device (HMD). An eye tracking system for an HMD may include eye tracking cameras and eye odometers that may reduce power consumption of the system. The eye odometers may be used as a low-power component to track relative movement of the user's eyes with respect to the HMD between frames captured by the eye tracking cameras, which may allow the frame rate of the eye tracking cameras to be reduced.

Abstract: One or more custom actions are stored in memory. Each of the custom actions is associated with a pattern of physical movement and a threshold. Physical movement data captured by a user device of the user is received and mapped to an identified one of the custom actions. When the physical movement data matches the associated pattern of physical movement within the associated threshold, it is determined that the user is authorized to access the identified custom action.

Abstract: Computer-implemented methods, apparatuses, and computer program products are disclosed for proximate financial transactions. An example method includes receiving a request for participation in a proximate financial transaction by a first user device associated with a first user and a first user profile where the proximate financial transaction is associated with a transfer of physical currency notes. The method further includes determining at least a second user device associated with a second user and second user profile proximate the first user for participation in the proximate financial transaction. The method also includes causing presentation of identification data of the second user via the first user device and detecting the transfer of physical currency notes between the first user and the second user. The method further includes effectuating an electronic financial transaction between a first user account of the first user and a second user account of the second user.

Abstract: The present disclosure provides systems and methods for calibration-free instant motion tracking useful, for example, for rending virtual content in augmented reality settings. In particular, a computing system can iteratively augment image frames that depict a scene to insert virtual content at an anchor region within the scene, including situations in which the anchor region moves relative to the scene. To do so, the computing system can estimate, for each of a number of sequential image frames: a rotation of an image capture system that captures the image frames; and a translation of the anchor region relative to an image capture system, thereby providing sufficient information to determine where and at what orientation to render the virtual content within the image frame.

Abstract: A technology for reducing a processing delay when a plurality of mixed reality spaces are shared between terminals is provided. An information processing system includes a terminal configured to display an image representing a mixed reality space obtained by superimposing a virtual space in a real space, and a server apparatus configured to communicate with the terminal. The server apparatus manages, for each of a plurality of real spaces, information for identifying the real space and anchor information for defining a superimposition position of the virtual space in the real space and, in a case where an acquisition request for anchor information corresponding to a first real space is received from the terminal, transmits response information including anchor information corresponding to the first real space and anchor information corresponding to a second real space adjacent to the first real space to the terminal.

Abstract: A head-mounted display may include a display system and an optical system that are supported by a housing. The optical system may be a catadioptric optical system having one or more lens elements. In one example, the optical system includes a single lens element and a retarder that is coated on a curved surface of the lens element. The retarder may be coated on an aspheric concave surface of the lens element. In another example the retarder may be coated on an aspheric convex surface of the lens element. One or more components of the optical system may be formed using a direct printing technique. This may allow for one or more adhesive layers and one or more hard coatings to be omitted from the optical system. A lens element may be directly printed on the display system to improve alignment between the optical system and the display system.

Abstract: An augmented reality system to generate and cause display of a presentation of a space at a first client device, receive one or more selections of points within the presentation of the space at the first client device, and render graphical elements at the one or more points within the presentation of the space at the first client device. The augmented reality system is further configured to receive a display request to display the space at a second client device, and in response, may render a second presentation of the space at the second client device, wherein the second presentation of the space includes the graphical elements at the one or more points.

Abstract: An electronic device includes a communication interface and a processor, wherein the processor receives a first image predistorted and rendered at a first time, from an external device through the communication interface, calculates a pixel shift for each pixel of the received first image at a second time that is different from the first time, generates a second image by reprojecting the first image based on the calculated pixel shift, and transmits the generated second image to the external device through the communication interface.

Abstract: In some embodiments, an electronic device navigates between user interfaces based at least on detecting a gaze of the user. In some embodiments, an electronic device enhances interactions with control elements of user interfaces. In some embodiments, an electronic device scrolls representations of categories and subcategories in a coordinated manner. In some embodiments, an electronic device navigates back from user interfaces having different levels of immersion in different ways.

Abstract: A display control method includes performing plane detection on a physical environment within a front field of view (FOV) of an augmented reality (AR) device, determining a first object in the physical environment, determining a display position of a second object based at least on the first object and a current FOV of the AR device, and displaying the second object at the display position. The display position includes a display height and a display distance.

Abstract: Disclosed herein is an image generation device including: a space construction section that constructs a virtual world to be displayed on a head-mounted display; an image generation section that generates, based on a position and a posture of a head of a player wearing the head-mounted display, a display image representing the virtual world in a field of view corresponding to a point of view of the player, and causes the head-mounted display to display the generated display image; and a visitor information acquisition section that acquires information regarding presence of a visitor without a head-mounted display in a space where the player is able to move. While the visitor is present, the space construction section displays an object indicating the presence of the visitor at a corresponding position in the virtual world.

Abstract: Described are system, method, and computer-program product embodiments for developing an object detection model. The object detection model may detect a physical object in an image of a real world environment. A system can automatically generate a plurality of synthetic images. The synthetic images can be generated by randomly selecting parameters of the environmental features, camera intrinsics, and a target object. The system may automatically annotate the synthetic images to identify the target object. In some embodiments, the annotations can include information about the target object determined at the time the synthetic images are generated. The object detection model can be trained to detect the physical object using the annotated synthetic images. The trained object detection model can be validated and tested using at least one image of a real world environment. The image(s) of the real world environment may or may not include the physical object.

Abstract: Methods and systems are disclosed for collecting and releasing virtual objects between disparate augmented reality environments. One method comprises receiving user selection to collect an object in a first environment displayed on a user device at a first time. A search request may then be transmitted to a content server from which a virtual object corresponding to the collected object is received in response. The virtual object may be stored in a user library associated with a user identifier of the user. When a second environment is displayed on the user device, the virtual object may be added to the second environment in response to receiving user selection to add the virtual object and determining that the user is associated with the user identifier.