Abstract: A system that incorporates contextual entity information when performing automatic speech processing (ASR) using a neural network architecture. The system identifies entities that may be related to the context of an utterance. Text information and pronunciation information related to those entities are encoded and used to determine biasing data that is applied to encoded audio data. The resulting adjusted encoded audio data is processed by the existing neural network architecture to determine ASR data representing a transcription of the utterance.

Abstract: Techniques for performing spoken language understanding (SLU) processing on a device are described. Example embodiments involve a device determining whether a spoken input corresponds to a supported spoken input class, a supported spoken input with dynamic content class or an unsupported spoken input class. For a spoken input corresponding to the supported spoken input with dynamic content class, the device may determine an entity corresponding to the spoken input from a set of entities, which may be determined based on device context data and/or user profile data. For a spoken input corresponding to the supported spoken input class, the device may determine an intent and entity using stored data. For a spoken input corresponding to the unsupported spoken input class, the device may send the audio data to a system for processing.

Abstract: Techniques for biasing for entities during automatic speech recognition (ASR) processing are described. In some embodiments, a system implements a gating component that is configured to switch on and off entity biasing on an audio frame basis when processing a spoken input. The gating component processes an audio frame to determine whether the audio frame likely includes a representation of a custom entity. Based on the determination, a biasing component, which is configured to generate entity embeddings, may be turned on or off. In this manner, entity biasing does not run on every audio frame, but only on the audio frames where it can be helpful in increasing ASR accuracy.

Abstract: Techniques for determining and using relevant prosody information for spoken language understanding (SLU) processing are described. In some embodiments, a system determines local prosody data for individual audio frames of input audio data representing a spoken input. The system also determines global prosody data based on the entire spoken input. A portion of the local prosody data is determined to be relevant for a respective audio frame. A portion of the global prosody data is determined to relevant for the spoken input. The relevant portions are used to determine at least an intent corresponding to the spoken input. Audio features corresponding to the input audio data may be used to determine relevant portions of the prosody data.

Abstract: Techniques for ASR processing using language model (LM)-generated context are described. A LM is prompted to generate words that are relevant for/may be included in a future user input. The prompt to the LM can include words from user interaction history, dialog history, dialog topic, user preferences, etc. The information included in the prompt may focus on rare or unique words rather than words that the ASR model is already confident in recognizing. The techniques can be plugged into an existing/pretrained ASR model and can be used with any existing/pretrained LM, thus saving resources needed to implement and maintain the components.

Abstract: Systems and methods for speech processing utilizing customized embeddings include receiving a first textual representation of first audio data and intent data indicating an intent of a first voice command. A first embedding may be generated from the textual representation and stored on the device. Second audio data representing a second voice command may be received and a second embedding may be generated therefrom. The first embedding may be determined to have at least a threshold similarity to the second embedding, and an intent may be determined that is associated with the second voice command. An action may be performed utilizing the intent.

Abstract: Techniques for reducing latency in processing of audio data, where the latency may be caused in detecting audio of interest in the audio data, are described. A device that captures audio data may include a detection component to determine when the audio data includes audio of interest (e.g., device-directed speech), and an audio embedding generator to generate embedding vectors for the captured audio data while the detection component processes the audio data. The device may generate an embedding vector for audio data captured at the device for a duration of time; determine, at the end of the duration of time, that the audio data represents audio of interest; and send the embedding vector to an audio processing component (e.g., an automatic speech recognition component) for processing.

Abstract: Techniques for using multiple machine learning (ML) models, with varying compute costs, for ASR processing is described. The system may include an arbitrator component configured to determine which ML model is to be used to process an audio frame from a sequence of audio frames representing a spoken natural language input. The arbitrator component may switch between the ML models, on a frame-by-frame basis, to reduce an overall compute cost for the entire spoken natural language input. The outputs of the different ML models may be combined to determine the final output for the entire spoken natural language input.

Abstract: Techniques for determining and storing relevant context information for a user input, such as a spoken input, are described. In some embodiments, context information is determined to be relevant on an audio frame basis. Context scores for different types of context data (e.g., prior dialog turn data, user profile data, device information, etc.) are determined for individual audio frames corresponding to a spoken input. Based on the corresponding context scores, the most relevant context is stored in a local context cache. The local context cache is updated as subsequent audio frames, of the user input, are processed. The data stored in the context cache is provided to downstream components to perform tasks such as ASR, NLU and SLU.

Abstract: Techniques for determining and storing relevant context information for a user input, such as a spoken input, are described. In some embodiments, context information is determined to be relevant on an audio frame basis. Context scores for different types of context data (e.g., prior dialog turn data, user profile data, device information, etc.) are determined for individual audio frames corresponding to a spoken input. Based on the corresponding context scores, the most relevant context is stored in a local context cache. The local context cache is updated as subsequent audio frames, of the user input, are processed. The data stored in the context cache is provided to downstream components to perform tasks such as ASR, NLU and SLU.

Abstract: Some devices may perform processing using machine learning models trained at a centralized system and distributed to the device. The centralized system may update the machine learning model and distribute the update to the device (or devices). To reduce the size of an update, the centralized system may train a model update object, which may be smaller in size than the model itself and thus more suitable for sending to the device(s). A device may receive the model update object and use it to update the on-device machine learning model; for example, by changing some parameters of the model. Parameters left unchanged during the update may retain their previous value. Thus, using the model update object to update the on-device model may result in a more accurate updated model when compared to sending an updated model compressed to a size similar to that of the model update object.

Abstract: Devices and techniques are generally described for compression of natural language processing models. A first index value to a first address of a weight table may be stored in a hash table. The first address may store a first weight associated with a first feature of a natural language processing model. A second index value to a second address of the weight table may be stored in the hash table. The second address may store a second weight associated with a second feature of the natural language processing model. A first code associated with the first feature and comprising a first number of bits may be generated. A second code may be generated associated with the second feature and comprising a second number of bits greater than the first number of bits based on a magnitude of the second weight being greater than a magnitude of the first weight.

Abstract: Devices and techniques are generally described for compression of natural language processing models. A first index value to a first address of a weight table may be stored in a hash table. The first address may store a first weight associated with a first feature of a natural language processing model. A second index value to a second address of the weight table may be stored in the hash table. The second address may store a second weight associated with a second feature of the natural language processing model. A first code associated with the first feature and comprising a first number of bits may be generated. A second code may be generated associated with the second feature and comprising a second number of bits greater than the first number of bits based on a magnitude of the second weight being greater than a magnitude of the first weight.