Abstract: A map processing device obtains images from a camera transported with a mobile device, and detects 2D features in the images. At least some of the 2D features detected in the images are densified to generate a densified set of 2D features. Locations of the densified set of 2D features are determined in 3D space using an image-based localization algorithm. A structure-based map comprising depth information from a depth sensor transported by another mobile device is accessed, wherein the depth information includes a set of data points indicating locations in 3D space corresponding to features in the real-world sensed by the depth sensor. The camera transported with the mobile device is localized to the real-world based on matching the locations of the densified set of 2D features in 3D space to the depth information accessed in the structure-based map.

Abstract: A mobile electronic device is provided for use with a headset. A camera outputs digital pictures of a portion of the headset. A display device displays information for viewing by a user wearing the headset. A processor retrieves calibration parameters that characterize at least a pose of the camera relative to the display device, and processes a digital picture from the camera to identify a pose of an optically identifiable feature within the digital picture. A pose of the mobile electronic device is identified relative to the holder based on the identified pose of the optically identifiable feature within the digital picture and based on at least the pose of the camera relative to the display device as characterized by the calibration parameters. The processor controls where graphical objects are rendered on the display device based on the identified pose of the mobile electronic device relative to the holder.

Abstract: A method for determining a transformation between a first coordinate system of an ultrasonic haptic device and a second coordinate system of a visual sensor device. The method includes: triggering generation of an ultrasonic focal point; obtaining images depicting a body part of a user while the ultrasonic focal point is active; determining a position of the ultrasonic focal point in the second coordinate system when the body part of the user is in the ultrasonic focal point; repeating the triggering, obtaining and determining; and calculating a first transformation between the first coordinate system and the second coordinate system based on the positions of the ultrasonic focal points in the first coordinate system and the second coordinate system to thereby calibrate a relationship between the ultrasonic haptic device and the visual sensor device.

Abstract: It is provided a method for enabling predicting performance of a localization-related device in relation to localization and/or mapping, the localization-related device forming part of a device network. The method is performed in a predictor device and comprises the steps of: detecting that a first localization-related device joins the device network; obtaining a first set of properties of the first localization-related device; determining, based on the first set of properties, a predictive model for the first localization-related device; and deploying the predictive model for the first localization-related device.

Abstract: A method (100) of controlling a portable device comprising a first camera and a second camera facing in the same direction. The method comprises: selecting (110) one of the first camera and the second camera as a visualization camera; initializing (120) a localization algorithm having as an input image data representing images captured by one of the first camera and the second camera; determining (130) a respective focus score for at least one of the first camera and the second camera, said focus score indicating a focus quality of features identified from images captured by one of the respective camera; selecting (140,) one of the first camera and the second camera as an enabled camera based on the at least one focus score; and generating a control signal configured to cause the selected camera to be enabled such that the image data representing images captured by the enabled camera are provided as the input to the localization algorithm.

Abstract: A mobile electronic device is provided for use with a headset. A camera outputs digital pictures of a portion of the headset. A display device displays information for viewing by a user wearing the headset. A processor retrieves calibration parameters that characterize at least a pose of the camera relative to the display device, and processes a digital picture from the camera to identify a pose of an optically identifiable feature within the digital picture. A pose of the mobile electronic device is identified relative to the holder based on the identified pose of the optically identifiable feature within the digital picture and based on at least the pose of the camera relative to the display device as characterized by the calibration parameters. The processor controls where graphical objects are rendered on the display device based on the identified pose of the mobile electronic device relative to the holder.

Abstract: A method for aligning representations of a 3D space is provided, including: obtaining a first set of geometric restrictions for a first representation corresponding to capturing restrictions of a first capturing device, the first representation forming part of a map for positioning of capturing devices; obtaining a second set of geometric restrictions corresponding to capturing restrictions of a second capturing device for providing a second representation of a 3D physical space; determining that a difference between the first set of geometric restrictions and the second set of geometric restrictions exceeds a threshold; determining which one of the first set of geometric restrictions and the second set of geometric restrictions is most restrictive and which one is least restrictive; and triggering a restrictive determination of the representation that is based on the least restrictive set of geometric restrictions, the determination being based on the most restrictive set of geometric restrictions.

Abstract: A device is disclosed that is configured to perform localization using one or both of a monocular camera and a depth sensor that are transportable with the device. The device includes at least one processor operationally connected to the monocular camera and the depth sensor. The device also includes at least one memory storing program code that is executed by the at least one processor to perform operations to receive image data from the monocular camera. The operations determine a benefit level of activating the depth sensor for localization, based on the image data, and activate the depth sensor for localization based on a determination that the benefit level of activating the depth sensor satisfies an activation rule. Related methods and computer program products are also disclosed.

Abstract: A device is disclosed that is configured for performing localization using one or both of a depth sensor and a monocular camera that are transportable with the device. The device is adapted to receive depth data from the depth sensor, determine a benefit level of activating the monocular camera for localization, based on the depth data, and activate the monocular camera for localization based on a determination that the benefit level of activating the monocular camera satisfies an activation rule. Related methods and computer program products are also disclosed.

Abstract: A method by a device performing localization using a set of sensors that are transported with the device is disclosed. A confidence score is determined for each of the sensors among the set of sensors. A subset of the sensors is defined from among the set of sensors that are to remain active based on their respective confidence scores satisfying a defined threshold. The method deactivates the sensors within the set of sensors having confidence scores that do not satisfy the defined threshold. The deactivation includes controlling power consumption by deactivated ones of the sensors.

Abstract: A method performed by a device for occlusion handling in augmented reality is provided. The device can generate at least one pixel classification image in a frame including an occluding object and having foreground, background, and unknown pixels. Generation of the at least one pixel classification image can include (1) calculating an initial foreground pixel probability image, and an initial background pixel probability image, and (2) calculating a normalized depth image based on depth information of the occluding object. The device can obtain an alpha mask to blend a virtual object and the foreground of the at least one pixel classification image based on determining a color of the unknown pixels. The device can render a final composition of an augmented reality image containing the virtual object occluded by the occluding object based on applying the alpha mask to pixels in the at least one pixel classification image.

Abstract: It is provided a method for performing localisation of a first user device comprising an environment sensor. The method is performed in a localisation determiner and comprising the steps of: determining a dynamicity parameter indicating an extent of environment dynamicity for the first user device; determining when the dynamicity parameter indicates that the first user device is in a dynamic environment; triggering localisation to occur using localisation procedures of a cellular network to which the first user device is connected, when the first user device is determined to be in a dynamic environment; and triggering localisation to occur using at least one environment sensor of the first user device when the first user device is determined to not be in a dynamic environment.

Abstract: A device is configured for performing localization using a set of sensors that are transportable with the device. The device includes at least one processor operationally connected to the set of sensors, and at least one memory that stores program code. The program code configures the at least one processor to determine a first set of device poses where a first sensor satisfies a localization performance rule, and to determine a second set of device poses where a second sensor satisfies the localization performance rule. The at least one processor is further configured to activate the second sensor while the first sensor is active based on a pose of the device transitioning from not being within to being within the second set of device poses.

Abstract: A method of controlling a portable device including a camera, the method including: performing localization of the portable device using a localization algorithm having as an input image data representing images captured by the camera; identifying a current focus area of the camera; determining a desired focus area for the camera; determining that a change of focus is required when the desired focus area is different to the current focus area; and generating a control signal configured to cause the focus of the camera to change in respect of a new focus area.

Abstract: A method (100) of controlling a portable device comprising a first camera and a second camera facing in the same direction. The method comprises: selecting (110) one of the first camera and the second camera as a visualization camera; initializing (120) a localization algorithm having as an input image data representing images captured by one of the first camera and the second camera; determining (130) a respective focus score for at least one of the first camera and the second camera, said focus score indicating a focus quality of features identified from images captured by one of the respective camera; selecting (140,) one of the first camera and the second camera as an enabled camera based on the at least one focus score; and generating a control signal configured to cause the selected camera to be enabled such that the image data representing images captured by the enabled camera are provided as the input to the localization algorithm.

Abstract: A mobile electronic device is provided for use with a headset. A camera outputs digital pictures of a portion of the headset. A display device displays information for viewing by a user wearing the headset. A processor retrieves calibration parameters that characterize at least a pose of the camera relative to the display device, and processes a digital picture from the camera to identify a pose of an optically identifiable feature within the digital picture. A pose of the mobile electronic device is identified relative to the holder based on the identified pose of the optically identifiable feature within the digital picture and based on at least the pose of the camera relative to the display device as characterized by the calibration parameters. The processor controls where graphical objects are rendered on the display device based on the identified pose of the mobile electronic device relative to the holder.

Abstract: A portable electronic device is used with a mixed reality (MR) headset. The MR headset includes a lens and a holder that retains the portable electronic device in a defined orientation relative to the lens. The portable electronic device includes a camera, a display device, and a processor. The camera outputs video frames and is arranged to view the lens. The display device is arranged to display information that is projected on the lens for reflection directly or indirectly toward the user's eyes and the camera. The processor performs operations that include processing the video frames from the camera to identify locations of at least one real-world feature, displaying information on the display device, and controlling at least one of the processing of the video frames and the displaying of the information on the display device to at least partially reduce occurrence in the video frames of the displayed information.

Abstract: A method for determining a transformation between a first coordinate system of an ultrasonic haptic device and a second coordinate system of a visual sensor device. The method includes: triggering generation of an ultrasonic focal point; obtaining images depicting a body part of a user while the ultrasonic focal point is active; determining a position of the ultrasonic focal point in the second coordinate system when the body part of the user is in the ultrasonic focal point; repeating the triggering, obtaining and determining; and calculating a first transformation between the first coordinate system and the second coordinate system based on the positions of the ultrasonic focal points in the first coordinate system and the second coordinate system to thereby calibrate a relationship between the ultrasonic haptic device and the visual sensor device.

Abstract: A mobile electronic device is provided for use with a headset. A camera outputs digital pictures of a portion of the headset. A display device displays information for viewing by a user wearing the headset. A processor retrieves calibration parameters that characterize at least a pose of the camera relative to the display device, and processes a digital picture from the camera to identify a pose of an optically identifiable feature within the digital picture. A pose of the mobile electronic device is identified relative to the holder based on the identified pose of the optically identifiable feature within the digital picture and based on at least the pose of the camera relative to the display device as characterized by the calibration parameters. The processor controls where graphical objects are rendered on the display device based on the identified pose of the mobile electronic device relative to the holder.

Abstract: A portable electronic device is used with a mixed reality (MR) headset. The MR headset includes a lens and a holder that retains the portable electronic device in a defined orientation relative to the lens. The portable electronic device includes a camera, a display device, and a processor. The camera outputs video frames and is arranged to view the lens. The display device is arranged to display information that is projected on the lens for reflection directly or indirectly toward the user's eyes and the camera. The processor performs operations that include processing the video frames from the camera to identify locations of at least one real-world feature, displaying information on the display device, and controlling at least one of the processing of the video frames and the displaying of the information on the display device to at least partially reduce occurrence in the video frames of the displayed information.