Abstract: Aspects of the present invention facilitate the customization of binary application user interfaces without modification of the underlying source code. Embodiments described herein can provide for hands-free operation or accessibility services, such as touchless operation of compiled applications, with minimal intervention. Described embodiments can automatically interface with the touch-based operating system to generate hands-free commands that, when detected (e.g., voice detection), can cause corresponding touch-based commands to be executed. Embodiments can utilize audio inputs, by way of example, to facilitate hands-free interaction with the touch-based operating system and applications executing thereon.

Abstract: The invention is directed towards a an audio scrubbing system that allows for scrubbing recognized voice commands from audio data and replacing the recognized voice commands with environment audio data. Specifically, as a user captures video and audio data via a HMD, audio data captured by the HMD may be processed by an audio scrubbing module to identify voice commands in the audio data that are used for controlling the HMD. When a voice command is identified in the audio data, timestamps corresponding to the voice command may be determined. Filler audio data may then be generated to imitate the environment by processing at least a portion of the audio data by a neural network of a machine learning model. The filler audio data may then be used to replace the audio data corresponding to the identified voice commands, thereby scrubbing the voice command from the audio data.

Abstract: The invention is directed towards a an audio scrubbing system that allows for scrubbing recognized voice commands from audio data and replacing the recognized voice commands with environment audio data. Specifically, as a user captures video and audio data via a HMD, audio data captured by the HMD may be processed by an audio scrubbing module to identify voice commands in the audio data that are used for controlling the HMD. When a voice command is identified in the audio data, timestamps corresponding to the voice command may be determined. Filler audio data may then be generated to imitate the environment by processing at least a portion of the audio data by a neural network of a machine learning model. The filler audio data may then be used to replace the audio data corresponding to the identified voice commands, thereby scrubbing the voice command from the audio data.

Abstract: The invention is directed towards a wearable platform system (e.g., head-mounted computing device) that allows for various modular peripheral devices to be interchangeably attached and detached, as well as methods and systems for transitioning an operational mode of the wearable platform between a hard connection mode and a wireless connection mode when a peripheral device is attached to or removed from the wearable platform. Specifically, the wearable platform may include a base member coupled to one or more arm members. A variety of modular peripheral devices may be attached at a hardware interface of the wearable platform to provide varied functionality to the wearable platform. As a user begins to remove a peripheral device, the wearable platform system may determine the peripheral device is being removed from the wearable platform and may transition from a hard connection mode to a wireless connection mode.

Abstract: Aspects of the present invention facilitate the customization of binary application user interfaces without modification of the underlying source code. Embodiments described herein can provide for hands-free operation or accessibility services, such as touchless operation of compiled applications, with minimal intervention. Described embodiments can automatically interface with the touch-based operating system to generate hands-free commands that, when detected (e.g., voice detection), can cause corresponding touch-based commands to be executed. Embodiments can utilize audio inputs, by way of example, to facilitate hands-free interaction with the touch-based operating system and applications executing thereon.

Abstract: Aspects of the present invention allow the customization binary application user interfaces without modification of the underlying source code. Embodiments described herein can provide, for instance, accessibility services such as touchless operation of compiled applications with minimal intervention. Described embodiments can automatically interface with the touch-based operating system to generate hands-free commands that, when detected (e.g., voice detection), can cause corresponding touch-based commands to be executed. Embodiments may utilize audio inputs, by way of example, to facilitate hands-free interaction with the touch-based operating system and applications executing thereon.

Abstract: Systems and methods provide voice activated authentication over time. A user can be registered with a voice authentication system based on a voiceprint profile of common words. This user voiceprint profile can be used in an ongoing secondary authentication as a hands-free head-mounted wearable device is used over time. Upon a user logging into a hands-free head-mounted wearable device voiceprints can be collected during a session. These collected voiceprints can be compared with a user voiceprint profile for a user authorized to operate the hands-free head-mounted wearable device. Such a comparison can include an analysis of frequency, duration, and amplitude for the voiceprints. When the voiceprints match, the login of the user can be maintained based on this secondary authentication using the voiceprints matched to the user voiceprint profile.

Abstract: Aspects of the present invention allow hands-free navigation of touch-based operating systems. The system may automatically interface with a touch-based operating system and generate hands-free commands that are associated with touch-based commands. Embodiments may utilize motion and/or audio inputs to facilitate hands-free interaction with the touch-based operating system.

Abstract: A head-mounted display that is modular and configured for securement to an item of headwear is provided. The head-mounted display may comprise a selection of components that are desired for a particular application, such as a display boom, camera, microphone, position-tracking component, etc., and may include an attachment mechanism for securing the selection of components to the item of headwear. The head-mounted display may include a display module having a corresponding display characteristic that provides a visual display and/or display functionality that is appropriate for a particular application. The display module may include a micro display that is non-transparent and/or a waveguide optic that is at least partially transparent, and may be configured to provide an augmented reality display for a user.

Abstract: The invention is directed towards a an audio scrubbing system that allows for scrubbing recognized voice commands from audio data and replacing the recognized voice commands with environment audio data. Specifically, as a user captures video and audio data via a HMD, audio data captured by the HMD may be processed by an audio scrubbing module to identify voice commands in the audio data that are used for controlling the HMD. When a voice command is identified in the audio data, timestamps corresponding to the voice command may be determined. Filler audio data may then be generated to imitate the environment by processing at least a portion of the audio data by a neural network of a machine learning model. The filler audio data may then be used to replace the audio data corresponding to the identified voice commands, thereby scrubbing the voice command from the audio data.

Abstract: Systems and methods provide for content navigation based on an identified context. A wearable device is initiated and loaded with content from a content repository. The loaded content comprises one or more digital tags. The wearable device is configured to provide for display, content associated with a digital tag. A context is determined for the wearable device, and based on the determined context a digital tag can be presented to a user for selection, the digital tag associate with a piece of content or a portion of content. A command can be received to select the digital tag or perform an action associated with the digital tag. A piece or portion of content associated with the selected digital tag is provided for display. A second digital tag can be presented along with the display of a piece or portion of content to enable further navigation through the content.

Abstract: Aspects of the technology described herein allow a user to add tags to a video as the video is being recorded. The tags can be added by capturing the user's voice as the video is recorded. Aspects can be performed by a head-mounted display. The head-mounted display can include an augmented reality display. In one aspect, a list of tags are displayed. The tags can be selected from a curated list of tags associated with a project the user is filming. For example, a project could comprise a building inspection during construction. In another aspect, the most commonly used tags in a given context can be shown. The most commonly used tags associated with a particular context can be determined through machine learning process. At a high level, a machine learning process can sort through historical tag data and associated contexts to determine a pattern correlating a context to a tag.

Abstract: The invention is directed towards wearable devices and operating wearable devices in a visual mode when a display device of the wearable device is within a line-of-sight (LOS) of a user and operating the wearable device in an audible mode when the wearable device is outside the LOS of the user. The position of the display device, relative to the user's LOS, is automatically determined. The operational mode of the wearable device is automatically transitioned when a transition of position of the display device is detected. The wearable device may be a head-mounted display (HMD) device. A visual mode provides visual-based presentations of UIs, content, information, and data, via the display device. In contrast to a visual mode, an audible mode provides audio-based presentations of UIs, content, information, and data, via one or more audible speakers included in the wearable device.

Abstract: Systems and methods provide for object interaction based on an identified context. A wearable device, such as a head-mounted display device (HMD), is initiated. The HMD can obtain data via one or more sensors that corresponds to one or more non-human-machine interface (HMI) devices or objects. Based on the obtained data, the HMD can recognize and identify the non-HMI devices within an optically-encoded field of view of the HMD. The non-HMI devices can be viewed through the HMD along with virtual objects or identifiers associated therewith, and the virtual objects or identifiers can reference the viewed non-HMI devices. One or more commands associated with the virtual objects or indicators can then be received by the HMD to enable interaction with the viewed non-HMI devices. Based on the commands information can be provided for display or audio output.

Abstract: The invention is directed towards a wearable platform system (e.g., head-mounted computing device) that allows for various modular peripheral devices to be interchangeably attached and detached, as well as methods and systems for transitioning an operational mode of the wearable platform between a hard connection mode and a wireless connection mode when a peripheral device is attached to or removed from the wearable platform. Specifically, the wearable platform may include a base member coupled to one or more arm members. A variety of modular peripheral devices may be attached at a hardware interface of the wearable platform to provide varied functionality to the wearable platform. As a user begins to remove a peripheral device, the wearable platform system may determine the peripheral device is being removed from the wearable platform and may transition from a hard connection mode to a wireless connection mode.

Abstract: Methods and systems provide dynamic selection of noise-cancelling algorithms, and dynamic activation and deactivation of microphones to provide multi-mode noise cancellation for a voice-detecting headset in situations where ambient noise prevents voice navigation from accurately interpreting voice commands. To do so, when an ambient noise is detected that exceeds a threshold, a particular noise-cancelling algorithm best-suited for the situation is selected, and one or more noise-detecting microphones is activated. The noise-detecting microphone(s) receiving the highest level of ambient noise can remain activated while the remaining noise-detecting microphones can be deactivated. A speech signal received by the speech microphone can then be optimized by cancelling the ambient noise signal received from the activated noise-detecting microphone(s) using the selected noise-cancelling algorithm. After the speech signal is optimized, it can be communicated to the voice-detecting headset for interpretation.

Abstract: Systems and methods provide concurrent access to a single input resource. An audio stack of a computing device can receive multiple requests from applications to provide concurrent access to audio data received via an input resource, such as audio data received via an audio card coupled to a microphone. A request to access the resource is received from a first application. Based on the request, a cache memory is instantiated to model a memory buffer of the resource. A direct session between a component of the audio stack and the resource is established. As audio data is encoded, the audio stack component can receive the encoded audio data and write the audio data into the cache. A first session between the first application and the cache is generated, such that the first application interprets the cache as the audio input resource buffer memory.

Abstract: Aspects of the present invention allow hands-free navigation of touch-based operating systems. The system may automatically interface with a touch-based operating system and generate hands-free commands that are associated with touch-based commands. Embodiments may utilize motion and/or audio inputs to facilitate hands-free interaction with the touch-based operating system.

Abstract: The invention is directed towards wearable devices and operating wearable devices in a visual mode when a display device of the wearable device is within a line-of-sight (LOS) of a user and operating the wearable device in an audible mode when the wearable device is outside the LOS of the user. The position of the display device, relative to the user's LOS, is automatically determined. The operational mode of the wearable device is automatically transitioned when a transition of position of the display device is detected. The wearable device may be a head-mounted display (HMD) device. A visual mode provides visual-based presentations of UIs, content, information, and data, via the display device. In contrast to a visual mode, an audible mode provides audio-based presentations of UIs, content, information, and data, via one or more audible speakers included in the wearable device.

Abstract: Enhanced telestrators are presented. Such enhanced telestrators provide visual elements (e.g. annotations and/or visual cues) to a user of a wearable device, such as a head-mounted display device. The visual elements may be superimposed, combined, composited, blended, and/or layered over image data displayed by a display device included in the wearable device. The visual elements may also be projected (by a projection system included in the wearable device), onto physical surfaces of the user's environment. Such projection systems may include lasers and/or light emitting diodes for photon source. The projection systems are enabled to project the visual elements onto surfaces of the environment of the user of the wearable device. More particularly, the various embodiments herein provide annotations and/or visual cues superimposed over image data viewed by the user and/or projected onto (and reflected by) physical surfaces the user's environment, via a photon-emitter.