Patents by Inventor Trevor Strohman

Trevor Strohman has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Publication number: 20260350309
    Abstract: A method (500) includes receiving audio data 102 characterizing an utterance (101) including a particular phrase (182) and a subsequent sequence of individual characters (184) that spell the particular phrase. The method also includes processing, using a speech recognition model (142), the audio data to generate an initial transcription (107) including a misrecognition of the particular phrase. The method also includes, in response to detecting a spelling structure (180) in the initial transcription subsequent to the misrecognition of the particular phrase: extracting, from the initial transcription, the sequence of characters; constructing a corrected phrase for the misrecognition of the particular phrase from the extracted sequence of characters; and normalizing the transcription to obtain a final transcription (170) for the utterance by replacing the extracted misrecognition of the particular phrase with the corrected phrase constructed from the extracted sequence of individual characters.
    Type: Application
    Filed: September 7, 2022
    Publication date: November 5, 2026
    Applicant: Google LLC
    Inventors: Khe Chai Sim, Françoise Beaufays, Trevor Strohman, Dan Zivkovic
  • Publication number: 20260351248
    Abstract: An ASR model includes a first encoder configured to receive a sequence of acoustic frames and generate a first higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The ASR model also includes a second encoder configured to receive the first higher order feature representation generated by the first encoder at each of the plurality of output steps and generate a second higher order feature representation for a corresponding first higher order feature frame. The ASR model also includes a decoder configured to receive the second higher order feature representation generated by the second encoder at each of the plurality of output steps and generate a first probability distribution over possible speech recognition hypothesis. The ASR model also includes a language model configured to receive the first probability distribution over possible speech hypothesis and generate a rescored probability distribution.
    Type: Application
    Filed: July 13, 2026
    Publication date: November 5, 2026
    Applicant: Google LLC
    Inventors: Tara N. Sainath, Arun Narayanan, Rami Botros, Yanzhang He, Ehsan Variani, Cyril Allauzen, David Rybach, Ruoming Pang, Trevor Strohman
  • Patent number: 12776316
    Abstract: As part of a dialog session between a user and an automated assistant, implementations can process, using a streaming ASR model, a stream of audio data that captures a portion of a spoken utterance to generate ASR output, process, using an NLU model, the ASR output to generate NLU output, and cause, based on the NLU output, a stream of fulfillment data to be generated. Further, implementations can further determine, based on processing the stream of audio data, audio-based characteristics associated with the portion of the spoken utterance captured in the stream of audio data. Based on the audio-based characteristics and/the stream of NLU output, implementations can determine whether the user has paused in providing the spoken utterance or has completed providing of the spoken utterance. If the user has paused, implementations can cause natural conversation output to be provided for presentation to the user.
    Type: Grant
    Filed: May 24, 2024
    Date of Patent: October 27, 2026
    Assignee: GOOGLE LLC
    Inventors: Jaclyn Konzelmann, Trevor Strohman, Jonathan Bloom, Johan Schalkwyk, Joseph Smarr
  • Publication number: 20260288789
    Abstract: Implementations disclose selecting, in response to receiving a request and from among multiple candidate generative models (e.g., multiple candidate large language models (LLMs)) with differing computational efficiencies, a particular generative model to utilize in generating a response to the request. Those implementations reduce latency and/or conserve computational resource(s) through selection, for various requests, of a more computationally efficient generative model for utilization in lieu of a less computationally efficient generative model. Further, those implementations seek to achieve such benefits, through utilization of more computationally efficient generative models, while also still selectively utilizing less computationally efficient generative models for certain requests to mitigate occurrences of a generated response being inaccurate and/or under-specified.
    Type: Application
    Filed: May 21, 2026
    Publication date: September 24, 2026
    Inventors: Seungyeon Kim, Ankit Singh Rawat, Wittawat Jitkrittum, Hari Narasimhan, Sashank Reddi, Neha Gupta, Srinadh Bhojanapalli, Aditya Menon, Manzil Zaheer, Tal Schuster, Sanjiv Kumar, Toby Boyd, Zhifeng Chen, Emanuel Taropa, Vikram Kasivajhula, Trevor Strohman, Martin Baeuml, Leif Schelin, Yanping Huang
  • Publication number: 20260289126
    Abstract: Implementations relate to handling visual content across a multi-turn dialog. A user input that includes natural language content and visual content is received during the dialog. If the visual content is being received for the first time in the dialog, the visual content is processed to generate a corresponding tokenized representation of the visual content. The corresponding tokenized representation can be cached in a database in association with the dialog, or in association with a user account of a user of the user query. If the visual content is subsequently referenced in the dialog, the corresponding tokenized representation of the visual content is retrieved from the database. The corresponding tokenized representation of the visual content, corresponding tokenized representations of natural language content, and optionally other metadata can be processed, using a generative model, to generate a response responsive to the user input.
    Type: Application
    Filed: May 22, 2026
    Publication date: September 24, 2026
    Inventors: Agoston Weisz, Alessandro Agostini, François-Xavier Aubet, Khalid Salama, Trevor Strohman, Ilia Akolzin, Petre Petrov
  • Patent number: 12731575
    Abstract: A method includes receiving a training example for a listen-attend-spell (LAS) decoder of a two-pass streaming neural network model and determining whether the training example corresponds to a supervised audio-text pair or an unpaired text sequence. When the training example corresponds to an unpaired text sequence, the method also includes determining a cross entropy loss based on a log probability associated with a context vector of the training example. The method also includes updating the LAS decoder and the context vector based on the determined cross entropy loss.
    Type: Grant
    Filed: February 10, 2023
    Date of Patent: September 8, 2026
    Assignee: Google LLC
    Inventors: Tara N. Sainath, Ruoming Pang, Ron Weiss, Yanzhang He, Chung-cheng Chiu, Trevor Strohman
  • Patent number: 12725607
    Abstract: An ASR model includes a first encoder configured to receive a sequence of acoustic frames and generate a first higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The ASR model also includes a second encoder configured to receive the first higher order feature representation generated by the first encoder at each of the plurality of output steps and generate a second higher order feature representation for a corresponding first higher order feature frame. The ASR model also includes a decoder configured to receive the second higher order feature representation generated by the second encoder at each of the plurality of output steps and generate a first probability distribution over possible speech recognition hypothesis. The ASR model also includes a language model configured to receive the first probability distribution over possible speech hypothesis and generate a rescored probability distribution.
    Type: Grant
    Filed: July 15, 2024
    Date of Patent: September 1, 2026
    Assignee: Google LLC
    Inventors: Tara Sainath, Arun Narayanan, Rami Botros, Yanzhang He, Ehsan Variani, Cyril Allauzen, David Rybach, Ruoming Pang, Trevor Strohman
  • Publication number: 20260236495
    Abstract: Implementations relate to reducing latency in generating and/or rendering natural language (NL) output generated using a large language model (LLM). Processor(s) of a system can: receive NL based input associated with a client device, and generate the NL based output utilizing the LLM. The NL based output can be a stream of NL based output in that it includes a plurality of segments, and is generated on a segment-by-segment basis. In some implementations, a first segment of the stream of NL based output is selected for inclusion in the stream of NL based output as a second segment (and any subsequent segment) is being generated to reduce latency in evaluating the NL based output as a whole prior to rendering thereof. In some versions of those implementations, the first segment is rendered as the second segment (and any subsequent segment) is being generated to further reduce latency in rendering thereof.
    Type: Application
    Filed: April 3, 2026
    Publication date: August 13, 2026
    Inventors: Martin Baeuml, Yanping Huang, Wenhao Jia, Chang Lan, Yuanzhong Xu, Junwhan Ahn, Alexander Bailey, Leif Schelin, Trevor Strohman, Emanuel Taropa, Sidharth Mudgal, Yanyan Zheng, Zhifeng Chen, Ahmad Beirami
  • Patent number: 12706095
    Abstract: Implementations described herein relate to reducing latency in automated assistant interactions. In some implementations, a client device can receive audio data that captures a spoken utterance of a user. The audio data can be processed to determine an assistant command to be performed by an automated assistant. The assistant command can be processed, using a latency prediction model, to generate a predicted latency to fulfill the assistant command. Further, the client device (or the automated assistant) can determine, based on the predicted latency, whether to audibly render pre-cached content for presentation to the user prior to audibly rendering content that is responsive to the spoken utterance. The pre-cached content can be tailored to the assistant command and audibly rendered for presentation to the user while the content is being obtained, and the content can be audibly rendered for presentation to the user subsequent to the pre-cached content.
    Type: Grant
    Filed: June 13, 2024
    Date of Patent: August 11, 2026
    Assignee: GOOGLE LLC
    Inventors: Lior Alon, Rafael Goldfarb, Dekel Auster, Dan Rasin, Michael Andrew Goodman, Trevor Strohman, Nino Tasca, Valerie Nygaard, Jaclyn Konzelmann
  • Patent number: 12706086
    Abstract: A method includes obtaining an ASR model trained to recognize speech in a first language and receiving transcribed training utterances in a second language. The method also includes integrating the ASR model with an input reprogramming module and a latent reprogramming module. The method also includes adapting the ASR model to learn how to recognize speech in the second language by training the input reprogramming module and the latent reprogramming module while parameters of the ASR model are frozen.
    Type: Grant
    Filed: October 20, 2023
    Date of Patent: August 11, 2026
    Assignee: Google LLC
    Inventors: Bo Li, Yu Zhang, Nanxin Chen, Rohit Prakash Prabhavalkar, Chao-Han Huck Yang, Tara N. Sainath, Trevor Strohman
  • Patent number: 12681964
    Abstract: Implementations relate to reducing latency in generating and/or rendering a given stream of natural language (NL) based output generated using a large language model (LLM). Processor(s) of a system can: receive NL based input associated with a client device, generate the stream of NL based output utilizing the LLM that is responsive to the NL based input and that is for a given dialog context of an ongoing dialog, and cause the stream of NL based output to be rendered at the client device. Notably, the processor(s) can employ attribute classifier(s) and a multi-objective scorer to implement a blockwise controlled decoding technique in generating the stream of NL based output utilizing the LLM. By implementing the blockwise controlled decoding technique in generating the stream of NL based output utilizing the LLM, the processor(s) can reduce latency in generating and/or of the stream of NL based output generated utilizing the LLM.
    Type: Grant
    Filed: July 25, 2023
    Date of Patent: July 14, 2026
    Assignee: GOOGLE LLC
    Inventors: Sidharth Mudgal, Ahmad Beirami, Jilin Chen, Alex Beutel, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Trevor Strohman
  • Patent number: 12664187
    Abstract: Implementations disclose selecting, in response to receiving a request and from among multiple candidate generative models (e.g., multiple candidate large language models (LLMs)) with differing computational efficiencies, a particular generative model to utilize in generating a response to the request. Those implementations reduce latency and/or conserve computational resource(s) through selection, for various requests, of a more computationally efficient generative model for utilization in lieu of a less computationally efficient generative model. Further, those implementations seek to achieve such benefits, through utilization of more computationally efficient generative models, while also still selectively utilizing less computationally efficient generative models for certain requests to mitigate occurrences of a generated response being inaccurate and/or under-specified.
    Type: Grant
    Filed: June 19, 2023
    Date of Patent: June 23, 2026
    Assignee: GOOGLE LLC
    Inventors: Seungyeon Kim, Ankit Singh Rawat, Wittawat Jitkrittum, Hari Narasimhan, Sashank Reddi, Neha Gupta, Srinadh Bhojanapalli, Aditya Menon, Manzil Zaheer, Tal Schuster, Sanjiv Kumar, Toby Boyd, Zhifeng Chen, Emanuel Taropa, Vikram Kasivajhula, Trevor Strohman, Martin Baeuml, Leif Schelin, Yanping Huang
  • Patent number: 12651122
    Abstract: Implementations relate to handling visual content across a multi-turn dialog. A user input that includes natural language content and visual content is received during the dialog. If the visual content is being received for the first time in the dialog, the visual content is processed to generate a corresponding tokenized representation of the visual content. The corresponding tokenized representation can be cached in a database in association with the dialog, or in association with a user account of a user of the user query. If the visual content is subsequently referenced in the dialog, the corresponding tokenized representation of the visual content is retrieved from the database. The corresponding tokenized representation of the visual content, corresponding tokenized representations of natural language content, and optionally other metadata can be processed, using a generative model, to generate a response responsive to the user input.
    Type: Grant
    Filed: June 26, 2024
    Date of Patent: June 9, 2026
    Assignee: GOOGLE LLC
    Inventors: Ágoston Weisz, Alessandro Agostini, François-Xavier Aubet, Khalid Salama, Trevor Strohman, Ilia Akolzin, Petre Petrov
  • Patent number: 12632657
    Abstract: A method includes receiving training data that includes a set of unspoken textual utterances. For each respective unspoken textual utterance, the method includes, tokenizing the respective textual utterance into a sequence of sub-word units, generating a first higher order textual feature representation for a corresponding sub-word unit tokenized from the respective unspoken textual utterance, receiving the first higher order textual feature representation generated by a text encoder, and generating a first probability distribution over possible text units. The method also includes training an encoder based on the first probability distribution over possible text units generated by a first-pass decoder for each respective unspoken textual utterance in the set of unspoken textual utterances.
    Type: Grant
    Filed: July 1, 2023
    Date of Patent: May 19, 2026
    Assignee: Google LLC
    Inventors: Tara N. Sainath, Zhouyuan Huo, Zhehuai Chen, Yu Zhang, Weiran Wang, Trevor Strohman, Rohit Prakash Prabhavalkar, Bo Li, Ankur Bapna
  • Patent number: 12620390
    Abstract: A method includes processing, using a speech recognizer, a first portion of audio data to generate a first lattice, and generating a first partial transcription for an utterance based on the first lattice. The method includes processing, using the recognizer, a second portion of the data to generate, based on the first lattice, a second lattice representing a plurality of partial speech recognition hypotheses for the utterance and a plurality of corresponding speech recognition scores. For each particular partial speech recognition hypothesis, the method includes generating a corresponding re-ranked score based on the corresponding speech recognition score and whether the particular partial speech recognition hypothesis shares a prefix with the first partial transcription.
    Type: Grant
    Filed: July 13, 2023
    Date of Patent: May 5, 2026
    Assignee: Google LLC
    Inventors: Antoine Jean Bruguier, David Qiu, Yanzhang He, Trevor Strohman
  • Patent number: 12602408
    Abstract: Implementations relate to reducing latency in generating and/or rendering natural language (NL) output generated using a large language model (LLM). Processor(s) of a system can: receive NL based input associated with a client device, and generate the NL based output utilizing the LLM. The NL based output can be a stream of NL based output in that it includes a plurality of segments, and is generated on a segment-by-segment basis. In some implementations, a first segment of the stream of NL based output is selected for inclusion in the stream of NL based output as a second segment (and any subsequent segment) is being generated to reduce latency in evaluating the NL based output as a whole prior to rendering thereof. In some versions of those implementations, the first segment is rendered as the second segment (and any subsequent segment) is being generated to further reduce latency in rendering thereof.
    Type: Grant
    Filed: April 19, 2023
    Date of Patent: April 14, 2026
    Assignee: GOOGLE LLC
    Inventors: Martin Baeuml, Yanping Huang, Wenhao Jia, Chang Lan, Yuanzhong Xu, Junwhan Ahn, Alexander Bailey, Leif Schelin, Trevor Strohman, Emanuel Taropa, Sidharth Mudgal, Yanyan Zheng, Zhifeng Chen, Ahmad Beirami
  • Patent number: 12548561
    Abstract: A method includes receiving a sequence of acoustic frames as input to a multilingual automated speech recognition (ASR) model configured to recognize speech in a plurality of different supported languages and generating, by an audio encoder of the multilingual ASR, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The method also includes generating, by a language identification (LID) predictor of the multilingual ASR, a language prediction representation for a corresponding higher order feature representation. The method also includes generating, by a decoder of the multilingual ASR, a probability distribution over possible speech recognition results based on the corresponding higher order feature representation, a sequence of non-blank symbols, and a corresponding language prediction representation. The decoder includes monolingual output layer having a plurality of output nodes each sharing a plurality of language-specific wordpiece models.
    Type: Grant
    Filed: October 11, 2023
    Date of Patent: February 10, 2026
    Assignee: Google LLC
    Inventors: Chao Zhang, Bo Li, Tara N. Sainath, Trevor Strohman, Shuo-yiin Chang
  • Publication number: 20260038489
    Abstract: Two-pass automatic speech recognition (ASR) models can be used to perform streaming on-device ASR to generate a text representation of an utterance captured in audio data. Various implementations include a first-pass portion of the ASR model used to generate streaming candidate recognition(s) of an utterance captured in audio data. For example, the first-pass portion can include a recurrent neural network transformer (RNN-T) decoder. Various implementations include a second-pass portion of the ASR model used to revise the streaming candidate recognition(s) of the utterance and generate a text representation of the utterance. For example, the second-pass portion can include a listen attend spell (LAS) decoder. Various implementations include a shared encoder shared between the RNN-T decoder and the LAS decoder.
    Type: Application
    Filed: October 13, 2025
    Publication date: February 5, 2026
    Inventors: Tara N. Sainath, Ruoming Pang, David Rybach, Yanzhang He, Rohit Prabhavalkar, Wei Li, Mirkó Visontai, Qiao Liang, Trevor Strohman, Yonghui Wu, Ian C. McGraw, Chung-Cheng Chiu
  • Publication number: 20260004071
    Abstract: Implementations relate to handling visual content across a multi-turn dialog. A user input that includes natural language content and visual content is received during the dialog. If the visual content is being received for the first time in the dialog, the visual content is processed to generate a corresponding tokenized representation of the visual content. The corresponding tokenized representation can be cached in a database in association with the dialog, or in association with a user account of a user of the user query. If the visual content is subsequently referenced in the dialog, the corresponding tokenized representation of the visual content is retrieved from the database. The corresponding tokenized representation of the visual content, corresponding tokenized representations of natural language content, and optionally other metadata can be processed, using a generative model, to generate a response responsive to the user input.
    Type: Application
    Filed: June 26, 2024
    Publication date: January 1, 2026
    Inventors: Ágoston Weisz, Alessandro Agostini, François-Xavier Aubet, Khalid Salama, Trevor Strohman, Ilia Akolzin, Petre Petrov
  • Patent number: 12482453
    Abstract: A method includes obtaining a set of training samples, wherein each training sample includes a corresponding sequence of speech segments corresponding to a training utterance and a corresponding sequence of ground-truth transcriptions for the sequence of speech segments, and wherein each ground-truth transcription includes a start time and an end time of a corresponding speech segment. For each training sample in the set of training samples, the method includes processing, using a speech recognition model, the corresponding sequence of speech segments to obtain one or more speech recognition hypotheses for the training utterance; and, for each speech recognition hypothesis obtained for the training utterance, identifying a respective number of word errors relative to the corresponding sequence of ground-truth transcriptions.
    Type: Grant
    Filed: September 27, 2022
    Date of Patent: November 25, 2025
    Assignee: Google LLC
    Inventors: Zhiyun Lu, Thibault Doutre, Yanwei Pan, Liangliang Cao, Rohit Prabhavalkar, Trevor Strohman, Chao Zhang