Abstract: A localization system may include an image input that receives images from a video source and an audio input that receives, from the video source, audio synchronized with the images. The localization system may also include an audio feature disentanglement network that correlates distinct audio elements from the audio input with corresponding visual features from the image input. Additionally, the localization system may include a geometry-based feature aggregation module that estimates a geometric transformation between two or more images from the video source and aggregates the visual features. Various other devices, systems, and methods are also disclosed.

Abstract: An audio-visual scene synthesis system may include a visual neural network, a cross-model bridge, and an audio neural network. Parameters of the audio neural network may be generated by the cross-model bridge based on analysis of a 3-dimensional visual environment modeled by the visual neural network. A coordinate transformation module may apply a transformation to an input camera direction to synthesize a new camera direction. The audio neural network may utilize the new camera direction and the parameters of the audio neural network to synthesize a multi-channel audio signal corresponding to the new camera direction. Various other devices, systems, and methods are also disclosed.

Abstract: A headset designed for inclusion of users with impairments is provided. The headset includes a frame, two eyepieces mounted on the frame, and at least one microphone and a speaker, mounted on the frame. The headset also includes a camera, a memory configured to store multiple instructions, and a processor configured to execute the instructions, wherein the instructions comprise to provide to a user an environmental context from a signal provided by the microphone and the camera. A method for using the above headset and a system for performing the method are also provided.

Abstract: A smart glass for incorporating speech recognition in an immersive reality environment is provided. The smart glass includes an eyepiece mounted on a frame including a transparent optical component to provide a user a view of a scene in a real world. The smart glass also includes a first camera configured to capture an image of a hand gesture from an interlocutor in the real world, and a processor configured to recognize, in the image of the hand gesture, a textual meaning. A system including memories storing instructions and processors to execute the instructions to perform methods for use of the above smart glass, and the methods, are also provided.

Abstract: Embodiments of the present disclosure relate to an audio system for artificial reality applications. One or more transducers of the audio system output, in accordance with audio instructions, one or more ultrasonic pressure waves simulating a virtual audio source near an ear of a user of the headset. A controller of the audio system generates the audio instructions such that the one or more ultrasonic pressure waves form at least a portion of audio content for presentation to the user. An array of microphones of the audio system detects audio signals in a local area. A deep neural network of the audio system processes the detected audio signals to generate enhanced audio content, and the one or more transducers present the enhanced audio content to a user.