Patents by Inventor Dmitry Lepikhin

Dmitry Lepikhin has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Publication number: 20230222318
    Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing machine learning task on a network input to generate a network output. In one aspect, one of the systems includes an attention neural network configured to perform the machine learning task, the attention neural network including one or more attention layers, each attention layer comprising an attention sub-layer and a feed-forward sub-layer. Some or all of the attention layers have a feed-forward sub-layer that applies conditional computation to the inputs to the sub-layer.
    Type: Application
    Filed: June 30, 2021
    Publication date: July 13, 2023
    Inventors: Dmitry Lepikhin, Yanping Huang, Orhan Firat, Maxim Krikun, Dehao Chen, Noam M. Shazeer, HyoukJoong Lee, Yuanzhong Xu, Zhifeng Chen
  • Publication number: 20220237435
    Abstract: Systems and methods for routing in mixture-of-expert models. In some aspects of the technology, a transformer may have at least one Mixture-of-Experts (“MoE”) layer in each of its encoder and decoder, with the at least one MoE layer of the encoder having a learned gating function configured to route each token of a task to two or more selected expert feed-forward networks, and the at least one MoE layer of the decoder having a learned gating function configured to route each task to two or more selected expert feed-forward networks.
    Type: Application
    Filed: January 27, 2021
    Publication date: July 28, 2022
    Applicant: Google LLC
    Inventors: Yanping Huang, Dmitry Lepikhin, Maxim Krikun, Orhan Firat, Ankur Bapna, Thang Luong, Sneha Kudugunta