Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing multiple modalities to generate accurate two-dimensional floorplans based on sparse digital videos depicting three-dimensional space. In particular, in one or more embodiments, the disclosed systems extract both visual and audio information from sparse digital video coverage of portions of a three-dimensional space and utilize the extracted visual and audio information to generate a two-dimensional floorplan representing both viewed and unviewed portions of the three-dimensional space. For example, the disclosed systems utilize self-attention layers of a specialized machine learning model to maintain and leverage bi-directional relationships among sequences of visual and audio features to generate floorplan predictions associated with the three-dimensional space.

Abstract: A wearable device includes an audio system. In one embodiment, the audio system includes a sensor array that includes a plurality of acoustic sensors. When a user wears the wearable device, the audio system determines an acoustic transfer function for the user based upon detected sounds within a local area surrounding the sensor array. Because the acoustic transfer function is based upon the size, shape, and density of the user's body (e.g., the user's head), different acoustic transfer functions will be determined for different users. The determined acoustic transfer functions are compared with stored acoustic transfer functions of known users in order to authenticate the user of the wearable device.

Abstract: An audio system generates customized head-related transfer functions (HRTFs) for a user. The audio system receives an initial set of estimated HRTFs. The initial set of HRTFs may have been estimated using a trained machine learning and computer vision system and pictures of the user's ears. The audio system generates a set of test locations using the initial set of HRTFs. The audio system presents test sounds at each of the initial set of test locations using the initial set of HRTFs. The audio system monitors user responses to the test sounds. The audio system uses the monitored responses to generate a new set of estimated HRTFs and a new set of test locations. The process repeats until a threshold accuracy is achieved or until a set period of time expires. The audio system presents audio content to the user using the customized HRTFs.

Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing multiple modalities to generate accurate two-dimensional floorplans based on sparse digital videos depicting three-dimensional space. In particular, in one or more embodiments, the disclosed systems extract both visual and audio information from sparse digital video coverage of portions of a three-dimensional space and utilize the extracted visual and audio information to generate a two-dimensional floorplan representing both viewed and unviewed portions of the three-dimensional space. For example, the disclosed systems utilize self-attention layers of a specialized machine learning model to maintain and leverage bi-directional relationships among sequences of visual and audio features to generate floorplan predictions associated with the three-dimensional space.

Abstract: An audio system includes a plurality of transducers, one or more acoustic sensors, and a controller. The plurality of transducers transmits an ultrasonic beam towards an ear of a user. The one or more acoustic sensors detect a reflected signal generated by an interaction of the ultrasonic beam with the ear. The controller updates a three-dimensional geometry of the ear based on the reflected signal. The controller determines a head-related transfer function (HRTF) for the user based in part on the three-dimensional geometry of the ear.

Abstract: In one embodiment, a computing system may capture, by a camera on a headset worn by a user, images that capture a body part of the user. The system may determine, based on the captured images, motion features encoding a motion history of the user. The system may detect, in the images, foreground pixels corresponding to the user's body part. The system may determine, based on the foreground pixels, shape features encoding the body part of the user captured by the camera. The system may determine a three-dimensional body pose and a three-dimensional head pose of the user based on the motion features and shape features. The system may generate a pose volume representation based on foreground pixels and the three-dimensional head pose of the user. The system may determine a refined three-dimensional body pose of the user based on the pose volume representation and the three-dimensional body pose.

Abstract: Embodiments relate to calibrating head-related transfer functions (HRTFs) for a user of an audio system (e.g., as a component of a headset) using cartilage conducted sounds. A test sound is presented to a user using a transducer (e.g., cartilage conduction) and an audio signal is responsively received via a microphone at an entrance to the user's ear canal. The test sound and audio signal combination may be provided to an audio server where a model is used to determine one or more HRTFs for the user. Information describing the one or more HRTFs is provided to the audio system to be used for providing audio to the user. The audio server may also use a model to determine geometric information describing a pinna of the user based on the combination. In one embodiment, the geometric information is used to determine the one or more HRTFs for the user.

Abstract: Embodiments relate to an audio system for various artificial reality applications. The audio system performs large scale filter optimization for audio rendering, preserving spatial and intra-population characteristics using neural networks. Further, the audio system performs adaptive hearing enhancement-aware binaural rendering. The audio includes an in-ear device with an inertial measurement unit (IMU) and a camera. The camera captures image data of a local area, and the image data is used to correct for IMU drift. In some embodiments, the audio system calculates a transducer to ear response for an individual ear using an equalization prediction or acoustic simulation framework. Individual ear pressure fields as a function of frequency are generated. Frequency-dependent directivity patterns of the transducers are characterized in the free field. In some embodiments, the audio system includes a headset and one or more removable audio apparatuses for enhancing acoustic features of the headset.

Abstract: A system for generating individualized HRTFs that are customized to a user of a headset. The system includes a server and an audio system. The server determines the individualized HRTFs based in part on acoustic features data (e.g., image data, anthropometric features, etc.) of the user and a template HRTF. The server provides the individualized HRTFs to the audio system. The audio system presents spatialized audio content to the user using the individualized HRTFs.

Abstract: Embodiments relate to calibrating head-related transfer functions (HRTFs) for a user of an audio system (e.g., as a component of a headset) using cartilage conducted sounds. A test sound is presented to a user using a transducer (e.g., cartilage conduction) and an audio signal is responsively received via a microphone at an entrance to the user's ear canal. The test sound and audio signal combination may be provided to an audio server where a model is used to determine one or more HRTFs for the user. Information describing the one or more HRTFs is provided to the audio system to be used for providing audio to the user. The audio server may also use a model to determine geometric information describing a pinna of the user based on the combination. In one embodiment, the geometric information is used to determine the one or more HRTFs for the user.

Abstract: An audio system generates customized head-related transfer functions (HRTFs) for a user. The audio system receives an initial set of estimated HRTFs. The initial set of HRTFs may have been estimated using a trained machine learning and computer vision system and pictures of the user's ears. The audio system generates a set of test locations using the initial set of HRTFs. The audio system presents test sounds at each of the initial set of test locations using the initial set of HRTFs. The audio system monitors user responses to the test sounds. The audio system uses the monitored responses to generate a new set of estimated HRTFs and a new set of test locations. The process repeats until a threshold accuracy is achieved or until a set period of time expires. The audio system presents audio content to the user using the customized HRTFs.

Abstract: An audio system generates customized head-related transfer functions (HRTFs) for a user. The audio system receives an initial set of estimated HRTFs. The initial set of HRTFs may have been estimated using a trained machine learning and computer vision system and pictures of the user's ears. The audio system generates a set of test locations using the initial set of HRTFs. The audio system presents test sounds at each of the initial set of test locations using the initial set of HRTFs. The audio system monitors user responses to the test sounds. The audio system uses the monitored responses to generate a new set of estimated HRTFs and a new set of test locations. The process repeats until a threshold accuracy is achieved or until a set period of time expires. The audio system presents audio content to the user using the customized HRTFs.

Abstract: A system for generating individualized HRTFs that are customized to a user of a headset. The system includes a server and an audio system. The server determines the individualized HRTFs based in part on acoustic features data (e.g., image data, anthropometric features, etc.) of the user and a template HRTF. The server provides the individualized HRTFs to the audio system. The audio system presents spatialized audio content to the user using the individualized HRTFs.

Abstract: A method for generating an individualized audio output response for a headset based on images of a user's ear. An image of a portion of a user's head including at least the user's ear is received, the user in the image wearing a headset including a plurality of visual markers. One or more features describing the user's ear are identified based at least in part on a position of one of the plurality of visual markers relative to the user's ear that are used to disambiguate orientation and/or scale of the features in the image. The one or more features are input to a model, and the model is configured to determine an audio output response of the user based on the extracted one or more features. An individualized audio output response is generated for the user based on the audio output response, the individualized audio output response configured to adjust one or more acoustic parameters of audio content provided to the user by the headset.

Abstract: Embodiments relate to calibrating head-related transfer functions (HRTFs) for a user of an audio system (e.g., as a component of a headset) using cartilage conducted sounds. A test sound is presented to a user using a transducer (e.g., cartilage conduction) and an audio signal is responsively received via a microphone at an entrance to the user's ear canal. The test sound and audio signal combination may be provided to an audio server where a model is used to determine one or more HRTFs for the user. Information describing the one or more HRTFs is provided to the audio system to be used for providing audio to the user. The audio server may also use a model to determine geometric information describing a pinna of the user based on the combination. In one embodiment, the geometric information is used to determine the one or more HRTFs for the user.

Abstract: An audio system generates customized head-related transfer functions (HRTFs) for a user. The audio system receives an initial set of estimated HRTFs. The initial set of HRTFs may have been estimated using a trained machine learning and computer vision system and pictures of the user's ears. The audio system generates a set of test locations using the initial set of HRTFs. The audio system presents test sounds at each of the initial set of test locations using the initial set of HRTFs. The audio system monitors user responses to the test sounds. The audio system uses the monitored responses to generate a new set of estimated HRTFs and a new set of test locations. The process repeats until a threshold accuracy is achieved or until a set period of time expires. The audio system presents audio content to the user using the customized HRTFs.

Abstract: A system for generating individualized HRTFs that are customized to a user of a headset. The system includes a server and an audio system. The server determines the individualized HRTFs based in part on acoustic features data (e.g., image data, anthropometric features, etc.) of the user and a template HRTF. The server provides the individualized HRTFs to the audio system. The audio system presents spatialized audio content to the user using the individualized HRTFs.

Abstract: A wearable device includes an audio system. In one embodiment, the audio system includes a sensor array that includes a plurality of acoustic sensors. When a user wears the wearable device, the audio system determines an acoustic transfer function for the user based upon detected sounds within a local area surrounding the sensor array. Because the acoustic transfer function is based upon the size, shape, and density of the user's body (e.g., the user's head), different acoustic transfer functions will be determined for different users. The determined acoustic transfer functions are compared with stored acoustic transfer functions of known users in order to authenticate the user of the wearable device.

Abstract: A system generates an output audio signal for an object or virtual object using image data of a room to select a room impulse response from a database. A headset may include a depth camera assembly (DCA) and processing circuitry. The DCA generates depth image data of a room. The processing circuitry determines room parameters such as the dimensions of the room based on the depth image data. A room impulse response for the room is determined based on referencing a database of room impulse responses using the room parameters. An output audio signal is generated by convolving a source audio signal of an object with the room impulse response.

Abstract: A method for generating an individualized audio output response for a headset based on a representation of a user's ear. One or more images of a portion of a user's head including at least the user's ear are received. A representation of the user's ear is generated based in part on the one or more images. A simulation of sound propagation from an audio source to the user's ear is performed based on the representation. An individualized audio output response is generated for the user based on the simulation, the individualized audio output response configured to adjust one or more acoustic parameters of audio content provided to the user by the headset.