Abstract: Techniques for monitoring devices to use ultrasonic signals to detect and track the locations of moving objects in an environment. To determine distance information, the monitoring devices emit a frequency-modulated continuous wave (FMCW) signal at an ultrasound frequency range. Reflections of the FMCW ultrasonic signal are used to generate time-of-arrival (TOA) profiles that indicate distances between the monitoring device and objects in the environment. The reflections can be processed to suppress undesirable interferences, such as reflections off non-mobile objects in the environment (e.g., walls, furniture, etc.), vibrations off the floorings or the ceilings, etc. After processing the reflections, a heatmap can be used to plot the intensity of the reflections for the different TOAs of the reflections, and depict the movement of the user over time. Finally, a Kalman filter is used to smooth the peaks in the intensity values on the plot, and determine the trajectory of the human.

Abstract: A system configured to detect a tap event on a surface of a device only using microphone audio data. For example, instead of using a physical sensor to detect the tap event, the device may detect a tap event in proximity to a microphone based on a power level difference between two or more microphones. When a power ratio exceeds a threshold, the device may detect a tap event and perform an action. For example, the device may output an alarm and use a detected tap event as an input to delay or end the alarm. In some examples, the device may detect a tap event using a plurality of microphones. Additionally or alternatively, the device may distinguish between multiple tap events based on a location of the tap event, enabling the device to perform two separate actions depending on the location.

Abstract: A system configured to perform deep adaptive acoustic echo cancellation (AEC) to improve audio processing. Due to mechanical noise and continuous echo path changes caused by movement of a device, echo signals are nonlinear and time-varying and not fully canceled by linear AEC processing alone. To improve echo cancellation, deep adaptive AEC processing integrates a deep neural network (DNN) and linear adaptive filtering to perform echo and/or noise removal. The DNN is configured to generate a nonlinear reference signal and step-size data, which the linear adaptive filtering uses to generate output audio data representing local speech. The DNN may generate the nonlinear reference signal by generating mask data that is applied to a microphone signal, such that the reference signal corresponds to a portion of the microphone signal that does not include near-end speech.

Abstract: A computing device may receive audio data from a microphone representing audio in an environment of the device, which may correspond to an utterance and noise. A model may be trained to process the audio data to reduce noise from the audio data. The model may include an encoder that includes one or more dense layers, one or more recurrent layers, and a decoder that includes one or more dense layers.

Abstract: A computer-implemented method includes receiving, at a microphone of a voice-controlled device, a speech input, generating an electrical signal having a first gain level that is below a gain threshold for audible detection by a user, transmitting the electrical signal to the speaker and detecting, by the microphone, an audio signal that includes a combination of ambient noise and a probe audio signal, wherein the probe audio signal is output by the speaker based on the electrical signal. The method further includes determining a power level of the probe audio signal and determining a state of the display based on the power level of the probe audio signal.

Abstract: A computer-implemented method includes receiving, at a microphone of a voice-controlled device, a speech input from a user and determining, by the voice-controlled device, that a power state of an AV display device that is coupled to the voice-controlled device is an ON or OFF state. Based on the user intent determined from the speech input and the power state of the AV display, the voice-controlled device sends data to the AV display device to switch the AV display device ON or OFF. The method can further include receiving, by the voice-controlled device, content from a content source location and sending the content to the AV display device via an AV port.

Abstract: A mobile device capable of capturing voice commands includes a beamformer for determining audio data corresponding to one or more directions and a beam selector for selecting in which direction a source of target audio lies. The device determines, based on data from one or more sensors, an angle through which the device has rotated. Based on the angle and one or more rotation-compensation functions, the device interpolates audio data corresponding to the one or more directions to compensate for the rotation such that the direction corresponding to the source of target audio remains selected.

Abstract: Systems and methods for microphone degradation detection and compensation are disclosed. For example, microphones of an electronic device may capture audio and generate corresponding audio data, such as during a period of time where only ambient noise is present. Sound intensity level value differences between audio data from the various microphones may be determined and when one or more of the sound intensity level value differences satisfies a threshold amount, the microphone associated with the variant sound intensity level value may be determined to be degraded. The sound intensity level value difference may be compensated for, such as by utilizing sound boosting techniques and/or modifying parameters of a beamforming component.

Abstract: A system configured to detect a tap event on a surface of a device only using microphone audio data. For example, instead of using a physical sensor to detect the tap event, the device may detect a tap event in proximity to a microphone based on a power level difference between two or more microphones. When a power ratio exceeds a threshold, the device may detect a tap event and perform an action. For example, the device may output an alarm and use a detected tap event as an input to delay or end the alarm. In some examples, the device may detect a tap event using a plurality of microphones. Additionally or alternatively, the device may distinguish between multiple tap events based on a location of the tap event, enabling the device to perform two separate actions depending on the location.

Abstract: A system that performs wall detection, range estimation, and/or corner detection to determine a position of a device relative to acoustically reflective surfaces. The device generates output audio using a loudspeaker, generates microphone audio data using a microphone array, performs beamforming to generate directional audio data and then generates impulse response data for each of a plurality of directions. The device may detect a peak in the impulse response data and determine a distance and/or direction to a reflective surface based on the peak. Based on a number of reflected surfaces and/or direction(s) of the reflected surfaces detected by the device, the device may classify the different directions and estimate where it is in the room, such as whether the device is in a corner, along one wall, or in an open area. By knowing its position relative to the room surfaces, the device may improve sound equalization.

Abstract: A computer-implemented method includes receiving, at a microphone of a voice-controlled device, a speech input, generating an electrical signal having a first gain level that is below a gain threshold for audible detection by a user, transmitting the electrical signal to the speaker and detecting, by the microphone, an audio signal that includes a combination of ambient noise and a probe audio signal, wherein the probe audio signal is output by the speaker based on the electrical signal. The method further includes determining a power level of the probe audio signal and determining a state of the display based on the power level of the probe audio signal.

Abstract: A system configured to improve audio processing by adaptively selecting target signals based on current system conditions. For example, a device may select a target signal based on a highest signal quality metric when only the local speech is present (e.g., during near-end single-talk conditions), as this maximizes an amount of energy included in the output audio signal. In contrast, the device may select the target signal based on a lowest signal quality metric when only the remote speech is present (e.g., during far-end single-talk conditions), as this minimizes an amount of energy included in the output audio signal. In addition, the device may track positions of the local speech and the remote speech over time, enabling the device to accurately select the target signal when both local speech and remote speech is present (e.g., during double-talk conditions).

Abstract: A system configured to improve echo cancellation for nonlinear systems. The system generate reference audio data by isolating portions of microphone audio data that correspond to playback audio data. For example, the system may determine a correlation between the playback audio data and the microphone audio data in individual time-frequency bands in a frequency domain. In some examples, the system may substitute microphone audio data associated with output audio for the playback audio data. The system may generate the reference audio data based on portions of the microphone audio data that have a strong correlation with the playback audio data. The system may generate the reference audio data by selecting these portions of the microphone audio data or by performing beamforming. This results in precise time alignment between the reference audio data and the microphone audio data, improving performance of the echo cancellation.

Abstract: A system configured to detect a tap event on a surface of a device only using microphone audio data. For example, instead of using a physical sensor to detect the tap event, the device may detect a tap event in proximity to a microphone based on a power level difference between two or more microphones. When a power ratio exceeds a threshold, the device may detect a tap event and perform an action. For example, the device may output an alarm and use a detected tap event as an input to delay or end the alarm. In some examples, the device may detect a tap event using a plurality of microphones. Additionally or alternatively, the device may distinguish between multiple tap events based on a location of the tap event, enabling the device to perform two separate actions depending on the location.

Abstract: A system configured to detect microphone failure and optimize settings using remaining functional microphones. For example, a device may detect a defective (e.g., nonfunctional or malfunctioning) microphone when an energy level for the defective microphone is lower than a threshold value for a period of time. After detecting the defective microphone, the device may update configuration settings by reassigning microphones (e.g., selecting a functional microphone instead of the defective microphone) and/or by calculating new configuration settings accordingly (e.g., excluding a nonfunctional microphone or compensating for an attenuation or phase shift of a malfunctioning microphone). For example, the system may recalculate beamformer coefficients using the remaining microphones, which may improve a performance of the device (e.g., wakeword detection, automatic speech recognition (ASR), etc.) and result in only a marginal performance degradation relative to fully-functioning microphones.

Abstract: A computer-implemented method includes receiving, at a microphone of a voice-controlled device, a speech input, generating an electrical signal having a first gain level that is below a gain threshold for audible detection by a user, transmitting the electrical signal to the speaker and detecting, by the microphone, an audio signal that includes a combination of ambient noise and a probe audio signal, wherein the probe audio signal is output by the speaker based on the electrical signal. The method further includes determining a power level of the probe audio signal and determining a state of the display based on the power level of the probe audio signal.

Abstract: A system that performs wall detection, range estimation, and/or corner detection to determine a position of a device relative to acoustically reflective surfaces. The device generates output audio using loudspeaker(s), generates microphone audio data using a microphone array, and generates impulse response data for each of the microphones. The device may generate the impulse response data using an acoustic echo cancellation (AEC) component or multi-channel AEC (MC-AEC). The device may detect a peak in the impulse response data and determine a distance to a reflective surface based on the peak. Based on a number of reflected surfaces detected by the device, the device may classify a position of the device within the room, such as whether the device is in a corner, along one wall, or in an open area. By knowing the position relative to the room surfaces, the device may improve sound equalization and other processing.

Abstract: A system configured to improve noise cancellation by using portions of multiple reference signals instead of using a complete reference signal. The system divides a frequency spectrum into frequency bands and selects a single reference signal from a group of potential reference signals for every frequency band. For example, a first reference signal is selected for a first frequency band while a second reference signal is selected for a second frequency band. The system may generate a combined reference signal using portions of each of the selected reference signals, such as a portion of the first reference signal corresponding to the first frequency band and a portion of the second reference signal corresponding to the second frequency band. Additionally or alternatively, the system may perform noise cancellation using each of the selected reference signals and filter the outputs based on the corresponding frequency band to generate combined audio output data.

Abstract: This disclosure describes, in part, techniques for detecting that a microphone of a voice-enabled device is unusable due to noise levels around the microphone, and indicating to a user that the device has become unusable due to, for example, buffeting. The device may analyze audio signals to determine that the noise levels are too high to perform speech-processing techniques to detect voice. The device may utilize speakers to output sounds, and/or LEDs to output light, to indicate noise levels and/or that the device is unable to detect voice due to buffeting. In this way, the device may indicate that a user needs to speak loudly to be heard, move their device, and/or reduce sound in the environment to use the speech-processing abilities of the device.

Abstract: A computer-implemented method includes receiving, at a microphone of a voice-controlled device, a speech input from a user and determining, by the voice-controlled device, that a power state of an AV display device that is coupled to the voice-controlled device is an ON or OFF state. Based on the user intent determined from the speech input and the power state of the AV display, the voice-controlled device sends data to the AV display device to switch the AV display device ON or OFF. The method can further include receiving, by the voice-controlled device, content from a content source location and sending the content to the AV display device via an AV port.