Abstract: Systems and methods provide for a multi-modal development environment to receive inputs using a variety of different input modalities in different user interfaces (UIs). Multiple user interfaces may be linked within the development environment to maintain state information so that inputs provided to one UI are represented in the other UIs using an appropriate equivalent representation based on the UI modality. Users of the development environment may select a given UI for interaction based on a desired task and then see changes tracked and relayed through the different UIs to verify changes within the development environment. The UIs may also be contextually linked to permit the user to work between both UIs without losing the context due to the switch.

Abstract: Techniques for performing speaker error correction are described. In some examples, speaker error correction is a post-processing task aligned predicted words and predicted one or more speaker identities, wherein the post-processing at least includes: performing predicted speaker error correction on the aligned predicted words and predicted one or more speaker identities to generate a first corrected set of speaker error corrections of the one or more speaker identifiers.

Abstract: This patent application relates to using framework parameters with a large language model to create beams based on a prompt. The beams can be evaluated using multiple criteria of a reward model that can be weighted for importance. The beams can be evaluated according to each of the one or more criteria and compared to determine which beams most closely align with the criteria. The beam that best aligns can be selected to generate a response.

Abstract: Systems and methods provide for modular, iterative framework to resolve inputs provided to an interaction environment. The input may be decomposed into different component parts and then relevant actions for each of the component parts may be predicted. An action may be selected and relevant parameters may be populated based on the input. If parameters remain unresolved, additional queries may be presented in order to resolve the remaining parameters. Multiple actions may be executed and then prepared to generate a combined response responsive to the input. Actions for a given interaction environment may be domain-specific and also may be developer-defined for a given goal or task to restrict one or more underlying language systems.

Abstract: The application is directed to receiving an alignment model and a prompt to evaluate and generate a response using a large language model. The large language model populates tokens for potential responses based on framework parameters and evaluates the potential responses using an alignment model. The alignment model can be utilized or selected for a prompt at decoding-time and can be used to select a potential response to use to create an output.

Abstract: Multi-stage fine-tuning for customizing automatic speech recognition with contextual adapters is performed. Entities to customize automatic speech recognition are received and used in an automatic speech recognition pipeline with a fine-tuned version of a speech recognition machine learning model and a fine-tuned version of a contextual adapter. The speech recognition machine learning model and the contextual adapter are fine-tuned in stages that include an initial stage that fine-tunes the speech recognition machine learning model using training data that includes multiple domains and subsequent tuning stages that freeze the speech recognition machine learning model while tuning the contextual adapter and tune the speech recognition machine learning model again.

Abstract: Techniques for augmenting automated speech recognition neural networks with scalable vocabularies are described. A cluster is selected from a plurality of clusters of similar sounding words based on a score, the score representing a similarity between an embedding of the cluster and an audio embedding of an utterance generated with an automated speech recognition encoder neural network. A bias factor is calculated based on a similarity between an embedding of a word in the selected cluster and the audio embedding. The audio embedding of the utterance is biased by the bias factor.

Abstract: Techniques for performing speaker error correction are described. In some examples, speaker error correction is a post-processing task to be performed on aligned predicted words and predicted one or more speaker identities to jointly perform speaker identities error correction and at least inverse text normalization, wherein the post-processing at least includes: predicting word and speaker contextual features from the aligned predicted words and predicted one or more speaker identities using an encoder, and predicting, from the word and speaker contextual features, inverse text normalization of the aligned predicted words and corrected speaker identities using a decoder.

Abstract: Systems and methods provide for a multi-modal development environment to receive inputs using a variety of different input modalities in different user interfaces (UIs). Multiple user interfaces may be linked within the development environment to maintain state information so that inputs provided to one UI are represented in the other UIs using an appropriate equivalent representation based on the UI modality. Users of the development environment may select a given UI for interaction based on a desired task and then see changes tracked and relayed through the different UIs to verify changes within the development environment. The UIs may also be contextually linked to permit the user to work between both UIs without losing the context due to the switch.

Abstract: A medical audio summarization service receives a medical conversation and an indication of a user preferred summarization style selected from a plurality of available summarization styles to generate a medical summary that conforms to the user preferred summarization style. A transcript is generated via a medical audio transcription service, and the transcript is used by a natural language processing engine (including a large language model) to generate the medical summary. The large language model is trained to be used to generate medical summaries that conform to respective ones of a plurality of user preferred summarization styles. The large language model is trained using training data comprising previously generated summaries and summary interaction metadata generated from user edits and/or feedback.

Abstract: Techniques for customizable latency, from the customer's side, for automatic speech recognition (ASR) are described. In particular, the customer may specify a parameter that controls how fast or how slow the customer's media content will be streamed or processed. Slower processing means higher accuracy, with near real-time latency, while faster processing means lower accuracy, but offers much lower latency (e.g., less than 600 ms). Enabling tuning of the latency-versus-accuracy tradeoff of the ASR system offers customers the flexibility to meet varying needs for different ASR applications.

Abstract: Language models may be dynamically updated for trending entities based on tuning data for particular users. A user may provide specific tuning data associated with trending entities within a class to generate a weight map for a language model. A class based model may be trained using the weight map specific for the user for the trending entities. Additionally, weights may be further boosted using a boosting language model to emphasize the trending entities.

Abstract: Language models may be dynamically updated for trending entities based on tuning data for particular users. A user may provide specific tuning data associated with trending entities within a class to generate a weight map for a language model. A class based model may be trained using the weight map specific for the user for the trending entities. Additionally, weights may be further boosted using a boosting language model to emphasize the trending entities.

Abstract: Systems and methods are described for converting speech to text. In one aspect, a method for converting speech to text includes generating an initial transcript of an audio or video file. At least one named entity may be extracted from the initial transcript using entity recognition. A subset of nodes of a knowledge graph that include the at least one named entity may be selected, where the subset of nodes of the knowledge graph correspond to a set of named entities. The method may further include encoding the set of named entities to generate a set of entity embeddings. The speech in the audio or video file may then be decoded using the set of entity embeddings to produce a final transcript of the audio file.

Abstract: Bias may be removed from automatic speech recognition model predictions using internal language model estimates. Audio data may be received for speech recognition. The audio data may be processed both through an automatic speech recognition model to produce original word token predictions and masked in different portions of the audio data to produce other word token predictions for the masked audio. A comparison of the original word token predictions and the other word token predictions may provide an estimate of an internal language model for the automatic speech recognition model. This estimate can be used to modify the original word token predictions to remove the lexical bias and produce a speech prediction.

Abstract: A set of alternative vocal input styles for specifying a parameter of a dialog-driven application is determined. During execution of the application, an audio prompt requesting input in one of the styles is presented. A value of the parameter is determined by applying a collection of analysis tools to vocal input obtained after the prompt is presented. A task of the application is initiated using the value.

Abstract: Portions of text data generated from inverse text normalization may be redacted. Text data for redaction may be obtained. One or more inverse text normalization models may be applied to the text data to generate normalized text data. A machine learning model, trained to recognize text for redaction, may be applied to identify portions of the normalized text data for redaction. The identified portions may be redacted and the redacted normalized text provided to a destination.

Abstract: Techniques for at least the generation of a chatbot built from a custom vocabulary and to use runtime hints during inference are described. In some examples, the generation of the chatbot includes receiving a request to build a chatbot using a bot definition and a custom vocabulary, wherein the chatbot is to use runtime hints during usage; building the chatbot from the bot definition and custom vocabulary by at least: generating automatic speech recognition (ASR) artifacts to be used in decoding audio input into the chatbot into text for at least one other component of the chatbot to use in determining a next act to be performed, the ASR artifacts including artifacts that use the custom vocabulary and artifacts that do not use the custom vocabulary, and storing the ASR artifacts.

Abstract: Techniques for managing access to sensitive data in transcriptions are described. A method for managing access to sensitive data in transcriptions may include receiving a request to generate a redacted transcript of content, obtaining a transcript of the content, sending at least a portion of the transcript to a model endpoint to identify sensitive entities in the transcript, receiving an inference response identifying one or more sensitive entities in the transcript, and generating the redacted transcript based at least one the transcript and the inference response.

Abstract: Domain specialty instructions may be generated for performing text analysis tasks. An input text may be received for performing a text analysis task. A domain specialty may be identified for the input text. Specialty domain identifiers may be inserted as part of generating instructions to perform the text analysis task using a pre-trained large language model fine-tuned to a domain that includes multiple domain specialties. The pre-trained large language model may perform the text analysis task on the input text using the generated instructions. A result of the text analysis tsk performed on the input text may be provided.