Abstract: Systems and methods for using neuromuscular information to improve speech recognition. The system includes a plurality of neuromuscular sensors, arranged on one or more wearable devices, wherein the plurality of neuromuscular sensors is configured to continuously record a plurality of neuromuscular signals from a user, at least one storage device configured to store one or more trained statistical models, and at least one computer processor programmed to provide as an input to the one or more trained statistical models, the plurality of neuromuscular signals or signals derived from the plurality of neuromuscular signals, determine based, at least in part, on an output of the one or more trained statistical models, at least one instruction for modifying an operation of a speech recognizer, and provide the at least one instruction to the speech recognizer.

Abstract: Implementations of the present disclosure include actions of receiving image data of an image capturing a scene, receiving data describing one or more entities determined from the scene, the one or more entities being determined from the scene, determining one or more actions based on the one or more entities, each action being provided at least partly based on search results from searching the one or more entities, and providing instructions to display an action interface comprising one or more action elements, each action element being to induce execution of a respective action, the action interface being displayed in a viewfinder.

Abstract: Computerized systems, methods, kits, and computer-readable media storing code for implementing the methods are provided for interacting with a physical object in an augmented reality (AR) environment generated by an AR system. One such system includes: a plurality of neuromuscular sensors able to sense a plurality of neuromuscular signals from a user, and at least one computer processor. The neuromuscular sensors are arranged on one or more wearable devices worn by the user to sense the neuromuscular signals. The at least one computer processor is or are programmed to: determine, based at least in part, on the neuromuscular signals sensed by the neuromuscular sensors, information about an interaction of the user with the physical object in the AR environment generated by the AR system; and instruct the AR system to provide feedback based, at least in part, on the information about the interaction of the user with the physical object.

Abstract: The disclosed computer-implemented method may include presenting, via a user interface, a sensory cue, and receiving, from neuromuscular sensors of a wearable device, various neuromuscular signals generated by a user wearing the wearable device, where the user generates the neuromuscular signals in response to the sensory cue being presented to the user via the user interface;. The method may also include interpreting the received neuromuscular signals as input commands with respect to the sensory cue provided by the user interface, such that the input commands initiate performance of specified tasks within the user interface. The method may also include performing the specified tasks within the user interface according to the interpreted input commands. Various other methods, systems, and computer-readable media are also disclosed.

Abstract: Systems and methods for text input based on neuromuscular information. The system includes a plurality of neuromuscular sensors, arranged on one or more wearable devices, wherein the plurality of neuromuscular sensors is configured to continuously record a plurality of neuromuscular signals from a user, at least one storage device configured to store one or more trained statistical models, and at least one computer processor programmed to obtain the plurality of neuromuscular signals from the plurality of neuromuscular sensors, provide as input to the one or more trained statistical models, the plurality of neuromuscular signals or signals derived from the plurality of neuromuscular signals, and determine based, at least in part, on an output of the one or more trained statistical models, one or more linguistic tokens.

Abstract: Methods and apparatus for controlling a physical object in an environment based, at least in part, on neuromuscular signals. The method comprises recording a plurality of neuromuscular signals from a plurality of neuromuscular sensors arranged on one or more wearable devices worn by a user, receiving a selection of a physical object within the environment, and controlling, based at least in part on the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals, an operation of the selected object within the environment.

Abstract: Computerized systems, methods, and computer-readable storage media storing code for implementing the methods are described for providing dynamically-updated musculoskeletal information. One such system includes a processor is programmed to: provide, as an input to a trained inference model, information based on a plurality of neuromuscular signals from a user and information based on at least one image of the user; determine, based on an output of the trained inference model, position information describing a spatial relationship between two or more connected musculoskeletal segments of the user and/or force information describing a force exerted by at least one musculoskeletal segment of the user; and output the position information and/or the force information.

Abstract: Computerized systems, methods, and computer-readable storage media storing code for implementing the methods are provided for training an inference model used to generate a musculoskeletal representation. One such system includes a processor programmed to: determine, based on information obtained from at least one image, position information describing a spatial relationship between two or more connected musculoskeletal segments of a user; determine, based on a plurality of neuromuscular signals, force information; associate the position information with the force information; train an inference model to output a musculoskeletal representation consistent with the position information and/or the force information when neuromuscular input signals provided to the inference model have at least one predetermined characteristic, to produce an updated inference model; and cause the updated inference model to be stored in a memory.

Abstract: Implementations of the present disclosure include actions of receiving image data of an image capturing a scene, receiving data describing one or more entities determined from the scene, the one or more entities being determined from the scene, determining one or more actions based on the one or more entities, each action being provided at least partly based on search results from searching the one or more entities, and providing instructions to display an action interface comprising one or more action elements, each action element being to induce execution of a respective action, the action interface being displayed in a viewfinder

Abstract: Implementations of the present disclosure include actions of receiving image data of an image capturing a scene, receiving data describing one or more entities determined from the scene, the one or more entities being determined from the scene, determining one or more actions based on the one or more entities, each action being provided at least partly based on search results from searching the one or more entities, and providing instructions to display an action interface comprising one or more action elements, each action element being to induce execution of a respective action, the action interface being displayed in a viewfinder.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating real-time search results. In one aspect, a method includes receiving a search query from a client; determining that real time search results should be included in a user interface document responsive to the search query; generating the user interface document, including client software to cause the client to present real time search results for the query; and sending the user interface document to the client.

Abstract: Methods and apparatus for calibrating performance of one or more statistical models used to generate a musculoskeletal representation. The method comprises controlling presentation of instructions via a user interface to instruct the user to perform the at least one gesture and updating at least one parameter of the one or more statistical models based, at least in part on a plurality of neuromuscular signals recorded by a plurality of neuromuscular sensors during performance of the at least one gesture by the user.

Abstract: Methods and apparatus for enabling a user or third-party to select or adjust parameters of one or more statistical models used to generate a musculoskeletal representation. The method comprises providing as input to the statistical model(s), a plurality of neuromuscular signals recorded by a plurality of neuromuscular sensors during performance of at least one gesture by a user, wherein the at least one gesture is performed by the user while wearing a wearable device having the plurality of neuromuscular sensors arranged thereon, rendering at least one visual representation based on an output of the statistical model(s), and receiving user or third-party input to adjust parameters of the statistical model(s) based on the rendered at least one visual representation, the user input including a selection of a particular statistical model of the statistical model(s) and/or an adjustment of parameters associated with the particular statistical model.

Abstract: Computerized systems, methods, kits, and computer-readable media storing code for implementing the methods are provided for interacting with a physical object in an augmented reality (AR) environment generated by an AR system. One such system includes: a plurality of neuromuscular sensors able to sense a plurality of neuromuscular signals from a user, and at least one computer processor. The neuromuscular sensors are arranged on one or more wearable devices worn by the user to sense the neuromuscular signals. The at least one computer processor is or are programmed to: determine, based at least in part, on the neuromuscular signals sensed by the neuromuscular sensors, information about an interaction of the user with the physical object in the AR environment generated by the AR system; and instruct the AR system to provide feedback based, at least in part, on the information about the interaction of the user with the physical object.

Abstract: Methods and apparatus for controlling a physical object in an environment based, at least in part, on neuromuscular signals. The method comprises recording a plurality of neuromuscular signals from a plurality of neuromuscular sensors arranged on one or more wearable devices worn by a user, receiving a selection of a physical object within the environment, and controlling, based at least in part on the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals, an operation of the selected object within the environment.

Abstract: Methods and systems for providing input to an augmented reality system or an extended reality system based, at least in part, on neuromuscular signals. The methods and systems comprise detecting, using one or more neuromuscular sensors arranged on one or more wearable devices, neuromuscular signals from a user; determining that a computerized system is in a mode configured to provide input including text to the augmented reality system; identifying based, at least in part, on the neuromuscular signals and/or information based on the neuromuscular signals, the input, wherein the input is further identified based, at least in part, on the mode; and providing the identified input to the augmented reality system.

Abstract: Systems and methods for using neuromuscular information to improve speech recognition. The system includes a plurality of neuromuscular sensors, arranged on one or more wearable devices, wherein the plurality of neuromuscular sensors is configured to continuously record a plurality of neuromuscular signals from a user, at least one storage device configured to store one or more trained statistical models, and at least one computer processor programmed to provide as an input to the one or more trained statistical models, the plurality of neuromuscular signals or signals derived from the plurality of neuromuscular signals, determine based, at least in part, on an output of the one or more trained statistical models, at least one instruction for modifying an operation of a speech recognizer, and provide the at least one instruction to the speech recognizer.

Abstract: Computerized systems, methods, and computer-readable storage media storing code for implementing the methods are provided for training an inference model used to generate a musculoskeletal representation. One such system includes a processor programmed to: determine, based on information obtained from at least one image, position information describing a spatial relationship between two or more connected musculoskeletal segments of a user; determine, based on a plurality of neuromuscular signals, force information; associate the position information with the force information; train an inference model to output a musculoskeletal representation consistent with the position information and/or the force information when neuromuscular input signals provided to the inference model have at least one predetermined characteristic, to produce an updated inference model; and cause the updated inference model to be stored in a memory.

Abstract: Computerized systems, methods, and computer-readable storage media storing code for implementing the methods are described for providing dynamically-updated musculoskeletal information. One such system includes a processor is programmed to: provide, as an input to a trained inference model, information based on a plurality of neuromuscular signals from a user and information based on at least one image of the user; determine, based on an output of the trained inference model, position information describing a spatial relationship between two or more connected musculoskeletal segments of the user and/or force information describing a force exerted by at least one musculoskeletal segment of the user; and output the position information and/or the force information.

Abstract: Computerized systems, methods, and computer-readable storage media storing code for implementing the methods are provided, in which camera information is used to calibrate one or more inference models used to generate a musculoskeletal representation. One such system includes at least one camera configured to capture at least one image, a plurality of neuromuscular sensors configured to sense and record a plurality of neuromuscular signals from a user, and at least one computer processor. The plurality of neuromuscular sensors are arranged on one or more wearable devices structured to be worn by the user to obtain the plurality of neuromuscular signals. The at least one computer processor is programmed to calibrate the one or more inference models by updating at least one parameter associated with the one or more inference models based, at least in part, on the plurality of neuromuscular signals and the at least one image.