Abstract: A system having sensor modules and a computing device. Each sensor module has an inertial measurement unit attached to a portion of a user to generate motion data identifying a sequence of orientations of the portion. The computing device provides the sequences of orientations measured by the sensor modules as input to an artificial neural network, obtains as output from the artificial neural network a predicted orientation measurement of a part of the user, and controls an application by setting an orientation of a rigid part of a skeleton model of the user according to the predicted orientation measurement. The artificial neural network can be trained to predict orientations measured using an optical tracking system based on orientations measured using inertial measurement units and/or to prediction orientation measurements of some rigid parts in a kinematic chain based on orientation measurements of other rigid parts in the kinematic chain.

Abstract: A system to track orientations of parts of a user based on both images and inertial measurement units (IMUs). For example, the system receives images showing a portion of the user wearing sensor modules. The system receives a first set of orientation measurements generated by the sensor modules attached to some parts of the user. The system determines the second set of orientation measurements of one or more features of the portion of the user from the images. The system provides the first set of orientation measurements and the second set of orientation measurements as input to an artificial neural network that is configured to predict orientation measurements of the one or more other parts of the user that would be measured by additional sensor modules if the additional sensor modules were to be attached to the other parts of the user.

Abstract: A system having sensor modules and a computing device. Each sensor module has an inertial measurement unit configured to track its orientation. In a kinematic chain, multiple rigid parts of a user are connected via joints. At least one rigid part is not independently tracked using sensor modules. The computing device computes the estimates of the orientation of the rigid part, separately using an artificial neural network model or using assumed orientation relations. During a sequence of actions performed by the user, orientation estimates produced by one technique can be more accurate than another technique at some time instances, but less accurate at other time instances. An artificial neural network is trained to classify the accuracy the estimates and/or to combine the estimates to provide improved orientation estimates for the duration of the sequence of actions.

Abstract: A system including: a plurality of sensor modules having inertial measurement units and attached to different parts of a user (e.g., head, hands, arms) to measure their orientations; a plurality of optical marks attached to the user; a camera attached to the user; and a computing device configured to correct an accumulated error by detecting the optical marks in an image generated by the camera and identifying mismatches in directions of the optical marks (or the sensors, or parts of the users) as measured and/or calculated based on the image and the corresponding directions of the optical marks (or the sensors, or parts of the users) as measured and/or calculated from the orientation measurements generated by the sensor modules.

Abstract: Automatically switching between different modes of using user motions to control applications running in a computing device. For example, the computing device communicates with devices attached to portions of a user (e.g., arm, hand) respectively to receive motion-based measurement data. One of the devices is attached to a portion of the user (e.g., hand) and capable of tracking six independent motions of the portion of the user. The computing device can generate input controls for one or more applications running therein using measurements based on the six independent motions in a first mode, and using measurements based on three of the six independent motions in a second mode. Based on an indication derived from inputs, or the lack of inputs, from some of the devices attached to the portions of the user, the computing device can automatically switch between the first and second modes.

Abstract: A system including: a plurality of sensor modules having inertial measurement units and attached to different parts of a user (e.g., head, hands, arms) to measure their orientations; a plurality of optical marks attached to the user; a camera attached to the user; and a computing device configured to correct an accumulated error by detecting the optical marks in an image generated by the camera and identifying mismatches in directions of the optical marks (or the sensors, or parts of the users) as measured and/or calculated based on the image and the corresponding directions of the optical marks (or the sensors, or parts of the users) as measured and/or calculated from the orientation measurements generated by the sensor modules.

Abstract: A system including a plurality of sensor modules, each module having an inertial measurement unit (IMU) and being attached to a respective body portion of a user (e.g., upper arm, hand, and/or head) to measure the current orientation of the corresponding portion of the user. A computing device coupled to the sensor modules is configured to identify that a user is at a predefined pose, which the predefined pose of the user is representative of the forearms and the upper arms of the user lying in a horizontal plane. A head mount display (HMD) attached to a head of the user using a camera to generate camera data, the HMD calculating the first length using the camera data, and the first length is calculated as corresponds to a distance between hands of the user and shoulders of the user; the computing device determining one or more lengths of one or more bones of the user based on the first length and the plurality of orientations of arm bones of the user at the predefined pose.

Abstract: A system having a plurality of sensor modules and a stereo camera and a computing device. Each sensor module has an inertial measurement unit (IMU) measuring its orientation relative to a reference orientation. Different IMUs may have different reference orientations. To calibrate the IMUs with respect to a common reference (e.g., defined based on a standardized pose of a user), the stereo camera captures a stereo image of a respective sensor module attached to a respective portion of the user; the inertial measurement unit of the respective sensor module generates an orientation measurement at a time of capturing the stereo image; and the computing device calculates, based on the stereo image, at least one orientation and uses the orientation and the orientation measurement in determining a rotation that calibrates measurements of the inertial measurement unit relative to the common reference.

Abstract: Automatically switching between different modes of using user motions to control applications running in a computing device. For example, the computing device communicates with devices attached to portions of a user (e.g., arm, hand) respectively to receive motion-based measurement data. One of the devices is attached to a portion of the user (e.g., hand) and capable of tracking six independent motions of the portion of the user. The computing device can generate input controls for one or more applications running therein using measurements based on the six independent motions in a first mode, and using measurements based on three of the six independent motions in a second mode. Based on an indication derived from inputs, or the lack of inputs, from some of the devices attached to the portions of the user, the computing device can automatically switch between the first and second modes.

Abstract: A method to calibrate orientation measurements of an inertial measurement unit of a sensor device based on an image of a portion of a user to which the sensor device is attached. For example, the sensor device can be configured to be attached to the middle phalange of the index finger and configured with a touch pad. In response to the determination that the thumb of the user is placed on the touch pad of the sensor device, the camera of the system can capture the image showing that the hand of the user. A convolutional neural network is configured to determine, from the image, orientations of predefined features of the hand of the user. A further artificial neural network is configured to determine the orientation of the sensor device based on the orientations of the predefined features to calibrate the orientation measurements of the inertial measurement unit.

Abstract: A system to track orientations of parts of a user based on both images and inertial measurement units (IMUs). For example, the system receives images showing a portion of the user wearing sensor modules. The system receives a first set of orientation measurements generated by the sensor modules attached to some parts of the user. The system determines the second set of orientation measurements of one or more features of the portion of the user from the images. The system provides the first set of orientation measurements and the second set of orientation measurements as input to an artificial neural network that is configured to predict orientation measurements of the one or more other parts of the user that would be measured by additional sensor modules if the additional sensor modules were to be attached to the other parts of the user.

Abstract: A method to calibrate orientation measurements of an inertial measurement unit of a sensor device based on an image of a portion of a user to which the sensor device is attached. For example, the sensor device can be configured to be attached to the middle phalange of the index finger and configured with a touch pad. In response to the determination that the thumb of the user is placed on the touch pad of the sensor device, the camera of the system can capture the image showing that the hand of the user. A convolutional neural network is configured to determine, from the image, orientations of predefined features of the hand of the user. A further artificial neural network is configured to determine the orientation of the sensor device based on the orientations of the predefined features to calibrate the orientation measurements of the inertial measurement unit.

Abstract: A system including a plurality of sensor modules, each module having an inertial measurement unit (IMU) and being attached to a respective body portion of a user (e.g., upper arm, hand, and/or head) to measure the current orientation of the corresponding portion of the user. A computing device coupled to the sensor modules is configured to identify that a user is at a predefined pose, which the predefined pose of the user is representative of the forearms and the upper arms of the user lying in a horizontal plane. A head mount display (HMD) attached to a head of the user using a camera to generate camera data, the HMD calculating the first length using the camera data, and the first length is calculated as corresponds to a distance between hands of the user and shoulders of the user; the computing device determining one or more lengths of one or more bones of the user based on the first length and the plurality of orientations of arm bones of the user at the predefined pose.

Abstract: A system having sensor modules and a computing device. Each sensor module has an inertial measurement unit attached to a portion of a user to generate motion data identifying a sequence of orientations of the portion. The sensor modules include a first subset and a second subset that share a common sensor module. The computing device provides orientation measurements generated by the first subset as input to a first artificial neural network to obtain at least one first orientation measurement of the common module, provides orientation measurements generated by the second subset as input to a second artificial neural network to obtain at least one second orientation measurement of the common module, and generates, a predicted orientation measurement of the common module by combining the at least one first orientation measurement of the common module and the at least one second orientation measurement of the common module.

Abstract: A system including a computing device receiving a first indication, the first indication indicating that a plurality of sensor modules are positioned in placeholders in a container device, wherein the placeholders are configured to hold the plurality of sensor modules at pre-determined positions and orientations in the container device. In response to the first indication, the computing device calibrates orientation measurements of the plurality or sensor modules relative to a first common reference system based on the pre-determined positions and orientations in the container device.

Abstract: A system having sensor modules and a computing device. Each sensor module has an inertial measurement unit configured to track its orientation. In a kinematic chain, multiple rigid parts of a user are connected via joints. At least one rigid part is not independently tracked using sensor modules. The computing device computes the estimates of the orientation of the rigid part, separately using an artificial neural network model or using assumed orientation relations. During a sequence of actions performed by the user, orientation estimates produced by one technique can be more accurate than another technique at some time instances, but less accurate at other time instances. An artificial neural network is trained to classify the accuracy the estimates and/or to combine the estimates to provide improved orientation estimates for the duration of the sequence of actions.

Abstract: A handheld controller having at least one input device, a left capacitive sensor electrode mounted in a left portion of the handheld device; a right capacitive sensor electrode mounted in a right portion of the handheld device; and a microcontroller. The input device is configured to receive user inputs provided via a finger of the hand holding the handheld controller. The handheld controller is symmetric from left to right. The microcontroller is configured to determine whether the hand is a left hand or a right hand based on measurements made via the left capacitive sensor electrode and the right capacitive sensor electrode and dynamically configures the handheld controller has left-handed or right-handed based on the measurements.

Abstract: A system having sensor modules and a computing device. Each sensor module has an inertial measurement unit attached to a portion of a user to generate motion data identifying a sequence of orientations of the portion. The sensor modules include a first subset and a second subset that share a common sensor module. The computing device provides orientation measurements generated by the first subset as input to a first artificial neural network to obtain at least one first orientation measurement of the common module, provides orientation measurements generated by the second subset as input to a second artificial neural network to obtain at least one second orientation measurement of the common module, and generates, a predicted orientation measurement of the common module by combining the at least one first orientation measurement of the common module and the at least one second orientation measurement of the common module.

Abstract: A handheld controller having at least one input device, a left capacitive sensor electrode mounted in a left portion of the handheld device; a right capacitive sensor electrode mounted in a right portion of the handheld device; and a microcontroller. The input device is configured to receive user inputs provided via a finger of the hand holding the handheld controller. The handheld controller is symmetric from left to right. The microcontroller is configured to determine whether the hand is a left hand or a right hand based on measurements made via the left capacitive sensor electrode and the right capacitive sensor electrode and dynamically configures the handheld controller has left-handed or right-handed based on the measurements.

Abstract: A system having sensor modules and a computing device. Each sensor module has an inertial measurement unit attached to a portion of a user to generate motion data identifying a sequence of orientations of the portion. The sensor modules include a first subset and a second subset that share a common sensor module. The computing device provides orientation measurements generated by the first subset as input to a first artificial neural network to obtain at least one first orientation measurement of the common module, provides orientation measurements generated by the second subset as input to a second artificial neural network to obtain at least one second orientation measurement of the common module, and generates, a predicted orientation measurement of the common module by combining the at least one first orientation measurement of the common module and the at least one second orientation measurement of the common module.