Abstract: Various implementations disclosed herein include devices, systems, and methods that display visual content as part of a 3D environment and add audio corresponding to the visual content. The audio may be spatialized to be from one or more audio source locations within the 3D environment. For example, a video may be presented on a virtual surface within an extended reality (XR) environment while audio associated with the video is spatialized to sound as if it is produced from an audio source location corresponding to that virtual surface. How the audio is provided may be determined based on the position of the viewer (e.g., the user or his/her device) relative to the presented visual content.

Abstract: In one implementation, a method of visualizing a combined audio pick-up pattern is performed at a first device in a physical environment, the first device including a display, one or more processors, and non-transitory memory. The method includes determining a first audio pick-up pattern of the first device. The method includes determining one or more second audio pick-up patterns of a respective one or more second devices. The method includes determining a combined audio pick-up pattern of the first device and the one or more second devices based on the first audio pick-up pattern and the one or more second audio pick-up patterns. The method includes displaying, on the display, a representation of the combined audio pick-up pattern. In one implementation, a method of determining an audio emission pattern is performed at a first device at a first location, the first device having a microphone, one or more processors, and non-transitory memory.

Abstract: A method for self-calibrating a sound pickup process that uses a microphone array in a wearable device that also includes a loudspeaker, where the microphone array being in a physical arrangement with respect to the loudspeaker. The method obtains, for each of several microphones of the microphone array, one or more transfer functions that each represent a response of the microphone to sound from a position in an acoustic space. The method determines whether a physical arrangement of the microphone array with respect to the loudspeaker has changed and adjusts the transfer function, for at least one of the microphones of the several microphones, in response to determining that the current physical arrangement of the microphone array with respect to the loudspeaker has changed.

Abstract: Various implementations disclosed herein include devices, systems, and methods that display visual content as part of a 3D environment and add audio corresponding to the visual content. The audio may be spatialized to be from one or more audio source locations within the 3D environment. For example, a video may be presented on a virtual surface within an extended reality (XR) environment while audio associated with the video is spatialized to sound as if it is produced from an audio source location corresponding to that virtual surface. How the audio is provided may be determined based on the position of the viewer (e.g., the user or his/her device) relative to the presented visual content.

Abstract: A method that includes driving a first speaker of a first electronic device using a mix of a first audio signal and a second audio signal. The method determines that the first electronic device is within a threshold distance of a second electronic device within an environment in which the first electronic device is located, where the second electronic device includes a second speaker. Responsive to determining that the first electronic device is within the threshold distance, causing the second electronic device to playback the second audio signal through the speaker and driving the first speaker using the first audio signal instead of the mix.

Abstract: In one implementation, a method of playing audio is performed by a device located in a physical environment, coupled to two or more speakers, and including one or more processors and non-transitory memory. The method includes executing an operating system and an application. The method includes receiving, by the operating system from the application via an application programming interface, audio session parameters including a spatial experience value providing instructions for the spatial playback of audio associated with the application. The method includes receiving, by the operating system from the application, instructions to play audio data. The method includes adjusting, by the operating system, the audio data based on the spatial experience value. The method includes sending, by the operating system to the two or more speakers, the adjusted audio data.

Abstract: A method includes obtaining an acoustic model of an environment. The acoustic model indicates a set of one or more acoustical properties of the environment. The method includes determining a placement location for a microphone within the environment based on the set of one or more acoustical properties of the environment and a pickup pattern of the microphone. The method includes displaying, on the display, a representation of the environment and a visual indicator that is overlaid onto the representation of the environment in order to indicate the placement location for the microphone.

Abstract: A method may include determining a location of a user, determining a virtual playback format based on the location of the user, where the virtual playback format includes a position of a virtual speaker that is fixed in an environment of the user, and determining an acoustic model based on the location of the user. The method also includes rendering audio at the playback device based on the acoustic model and the virtual playback format.

Abstract: Aspects of the subject technology provide improved point-to-point audio communications based on human variable sensitivity to latency differences in multipath communications. In aspects, improved techniques may include measuring a level of ambient noise, and then selecting processing for a received electronic audio based on the measured level of ambient noise before emitting the processed audio signal at a loudspeaker worn by a listener.

Abstract: Various implementations disclosed herein include devices, systems, and methods that provide an improved remote device user interface for enabling accurate text input into a user device. For example, a user device (e.g., an HMD) may be enabled to receive touchscreen input via input controls displayed by a remote device (e.g., a mobile phone, a tablet computer, etc.) using a display mode enhanced for pass-through viewing. For example, an enhanced display mode may enable a larger than normal display to enhance display mode attributes associated with input entry with respect to the user device. In some implementations display mode attributes may be enabled based on context such as distance, lens distortion, lighting, minimum character resolution, etc. Likewise, some implementations may augment input controls (on an HMD) such as 3 dimensional bubbles, auto-correct controls, auto-fill controls, spatialized audio controls, virtual touch/function bar controls, etc.

Abstract: Aspects of the subject technology provide improved point-to-point audio communications based on human variable sensitivity to latency differences in multipath communications. In aspects, improved techniques may include measuring a level of ambient noise, and then selecting processing for a received electronic audio based on the measured level of ambient noise before emitting the processed audio signal at a loudspeaker worn by a listener.

Abstract: Various implementations disclosed herein include devices, systems, and methods that provide video content (e.g., a TV show, a recorded sporting event, a movie, a 3D video, etc.) within a 3D environment using parameters selected based on one or more contextual factors. Such contextual factors may be determined based on an attribute of the content (e.g., its intended purpose or viewing environment), the user (e.g., the user's visual quality, interpupillary distance, etc.), the 3D environment (e.g., current lighting conditions, spatial considerations, etc.), or other context attributes.

Abstract: An audio processing system may obtain a size of a visual object to present to a display. The audio processing system may determine a virtual placement for each of a plurality of virtual speakers at least based on the size of the visual object. Each of the plurality of virtual speakers may be spatially rendered at each virtual placement through binaural audio, for playback through head-worn speakers. Other aspects are also described and claimed.

Abstract: A method for self-calibrating a sound pickup process that uses a microphone array in a wearable device that also includes a loudspeaker, where the microphone array being in a physical arrangement with respect to the loudspeaker. The method obtains, for each of several microphones of the microphone array, one or more transfer functions that each represent a response of the microphone to sound from a position in an acoustic space. The method determines whether a physical arrangement of the microphone array with respect to the loudspeaker has changed and adjusts the transfer function, for at least one of the microphones of the several microphones, in response to determining that the current physical arrangement of the microphone array with respect to the loudspeaker has changed.

Abstract: In one implementation, a method of visualizing a combined audio pick-up pattern is performed at a first device in a physical environment, the first device including a display, one or more processors, and non-transitory memory. The method includes determining a first audio pick-up pattern of the first device. The method includes determining one or more second audio pick-up patterns of a respective one or more second devices. The method includes determining a combined audio pick-up pattern of the first device and the one or more second devices based on the first audio pick-up pattern and the one or more second audio pick-up patterns. The method includes displaying, on the display, a representation of the combined audio pick-up pattern. In one implementation, a method of determining an audio emission pattern is performed at a first device at a first location, the first device having a microphone, one or more processors, and non-transitory memory.

Abstract: A method for self-calibrating a sound pickup process that uses a microphone array in a wearable device that also includes a loudspeaker, where the microphone array being in a physical arrangement with respect to the loudspeaker. The method obtains, for each of several microphones of the microphone array, one or more transfer functions that each represent a response of the microphone to sound from a position in an acoustic space. The method determines whether a physical arrangement of the microphone array with respect to the loudspeaker has changed and adjusts the transfer function, for at least one of the microphones of the several microphones, in response to determining that the current physical arrangement of the microphone array with respect to the loudspeaker has changed.

Abstract: Various implementations disclosed herein include devices, systems, and methods that that modify audio of played back AV content based on context in accordance with some implementations. In some implementations audio-visual content of a physical environment is obtained, and the audio-visual content includes visual content and audio content that includes a plurality of audio portions corresponding to the visual content. In some implementations, a context for presenting the audio-visual content is determined, and a temporal relationship between one or more audio portions of the plurality of audio portions and the visual content is determined based on the context. Then, synthesized audio-visual content is presented based on the temporal relationship.

Abstract: Each of a plurality of virtual loudspeaker arrays and their channels are produced, based on a corresponding microphone array and microphone signals thereof. Channels of a hallucinated loudspeaker array are determined based on the channels of the plurality of virtual loudspeaker arrays. The plurality of virtual loudspeaker arrays and the hallucinated loudspeaker array share a common geometry and orientation. Spatial audio is rendered based on the channels of the hallucinated loudspeaker array.

Abstract: Various implementations disclosed herein include devices, systems, and methods that display visual content as part of a 3D environment and add audio corresponding to the visual content. The audio may be spatialized to be from one or more audio source locations within the 3D environment. For example, a video may be presented on a virtual surface within an extended reality (XR) environment while audio associated with the video is spatialized to sound as if it is produced from an audio source location corresponding to that virtual surface. How the audio is provided may be determined based on the position of the viewer (e.g., the user or his/her device) relative to the presented visual content.

Abstract: Various implementations disclosed herein include devices, systems, and methods that that modify audio of played back AV content based on context in accordance with some implementations. In some implementations audio-visual content of a physical environment is obtained, and the audio-visual content includes visual content and audio content that includes a plurality of audio portions corresponding to the visual content. In some implementations, a context for presenting the audio-visual content is determined, and a temporal relationship between one or more audio portions of the plurality of audio portions and the visual content is determined based on the context. Then, synthesized audio-visual content is presented based on the temporal relationship.