Patents by Inventor Bryan Catanzaro

Bryan Catanzaro has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Publication number: 20210067735
    Abstract: Apparatuses, systems, and techniques to enhance video. In at least one embodiment, one or more neural networks are used to create, from a first video, a second video having a higher frame rate, higher resolution, or reduced number of missing or corrupt video frames.
    Type: Application
    Filed: September 3, 2019
    Publication date: March 4, 2021
    Inventors: Fitsum Reda, Deqing Sun, Aysegul Dundar, Mohammad Shoeybi, Guilin Liu, Kevin Shih, Andrew Tao, Jan Kautz, Bryan Catanzaro
  • Publication number: 20210064925
    Abstract: Apparatuses, systems, and techniques to enhance video are disclosed. In at least one embodiment, one or more neural networks are used to create, from a first video, a second video having one or more additional video frames.
    Type: Application
    Filed: September 3, 2019
    Publication date: March 4, 2021
    Inventors: Kevin Shih, Aysegul Dundar, Animesh Garg, Robert Pottorff, Andrew Tao, Bryan Catanzaro
  • Publication number: 20200394994
    Abstract: Systems and methods to help synthesize a second audio signal based, at least in part, on one or more neural networks trained using one or more characteristics of a first audio signal. Systems and methods to train one or more neural networks to synthesize a second audio signal based, at least in part, on one or more characteristics of a first audio signal.
    Type: Application
    Filed: June 12, 2019
    Publication date: December 17, 2020
    Inventors: Ryan Prenger, Rafael Valle, Bryan Catanzaro
  • Patent number: 10832120
    Abstract: Systems and methods for a multi-core optimized Recurrent Neural Network (RNN) architecture are disclosed. The various architectures affect communication and synchronization operations according to the Multi-Bulk-Synchronous-Parallel (MBSP) model for a given processor. The resulting family of network architectures, referred to as MBSP-RNNs, perform similarly to a conventional RNNs having the same number of parameters, but are substantially more efficient when mapped onto a modern general purpose processor. Due to the large gain in computational efficiency, for a fixed computational budget, MBSP-RNNs outperform RNNs at applications such as end-to-end speech recognition.
    Type: Grant
    Filed: April 5, 2016
    Date of Patent: November 10, 2020
    Assignee: Baidu USA LLC
    Inventors: Gregory Diamos, Awni Hannun, Bryan Catanzaro, Dario Amodei, Erich Elsen, Jesse Engel, Shubhabrata Sengupta
  • Patent number: 10769533
    Abstract: Disclosed are systems and methods that implement efficient engines for computation-intensive tasks such as neural network deployment. Various embodiments of the invention provide for high-throughput batching that increases throughput of streaming data in high-traffic applications, such as real-time speech transcription. In embodiments, throughput is increased by dynamically assembling into batches and processing together user requests that randomly arrive at unknown timing such that not all the data is present at once at the time of batching. Some embodiments allow for performing steaming classification using pre-processing. The gains in performance allow for more efficient use of a compute engine and drastically reduce the cost of deploying large neural networks at scale, while meeting strict application requirements and adding relatively little computational latency so as to maintain a satisfactory application experience.
    Type: Grant
    Filed: July 13, 2016
    Date of Patent: September 8, 2020
    Assignee: Baidu USA LLC
    Inventors: Christopher Fougner, Bryan Catanzaro
  • Patent number: 10540957
    Abstract: Presented herein are embodiments of state-of-the-art speech recognition systems developed using end-to-end deep learning. In embodiments, the model architecture is significantly simpler than traditional speech systems, which rely on laboriously engineered processing pipelines; these traditional systems also tend to perform poorly when used in noisy environments. In contrast, embodiments of the system do not need hand-designed components to model background noise, reverberation, or speaker variation, but instead directly learn a function that is robust to such effects. A phoneme dictionary, nor even the concept of a “phoneme,” is needed. Embodiments include a well-optimized recurrent neural network (RNN) training system that can use multiple GPUs, as well as a set of novel data synthesis techniques that allows for a large amount of varied data for training to be efficiently obtained.
    Type: Grant
    Filed: June 9, 2015
    Date of Patent: January 21, 2020
    Assignee: BAIDU USA LLC
    Inventors: Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Gregory Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubhabrata Sengupta, Adam Coates, Andrew Y. Ng
  • Publication number: 20190371298
    Abstract: Presented herein are embodiments of state-of-the-art speech recognition systems developed using end-to-end deep learning. In embodiments, the model architecture is significantly simpler than traditional speech systems, which rely on laboriously engineered processing pipelines; these traditional systems also tend to perform poorly when used in noisy environments. In contrast, embodiments of the system do not need hand-designed components to model background noise, reverberation, or speaker variation, but instead directly learn a function that is robust to such effects. A phoneme dictionary, nor even the concept of a “phoneme,” is needed. Embodiments include a well-optimized recurrent neural network (RNN) training system that can use multiple GPUs, as well as a set of novel data synthesis techniques that allows for a large amount of varied data for training to be efficiently obtained.
    Type: Application
    Filed: August 15, 2019
    Publication date: December 5, 2019
    Applicant: BAIDU USA LLC
    Inventors: Awni HANNUN, Carl CASE, Jared Casper, Bryan Catanzaro, Gregory Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubhabrata Sengupta, Adam Coates, Andrew Ng
  • Publication number: 20190295228
    Abstract: A neural network architecture is disclosed for performing image in-painting using partial convolution operations. The neural network processes an image and a corresponding mask that identifies holes in the image utilizing partial convolution operations, where the mask is used by the partial convolution operation to zero out coefficients of the convolution kernel corresponding to invalid pixel data for the holes. The mask is updated after each partial convolution operation is performed in an encoder section of the neural network. In one embodiment, the neural network is implemented using an encoder-decoder framework with skip links to forward representations of the features at different sections of the encoder to corresponding sections of the decoder.
    Type: Application
    Filed: March 21, 2019
    Publication date: September 26, 2019
    Inventors: Guilin Liu, Fitsum A. Reda, Kevin Shih, Ting-Chun Wang, Andrew Tao, Bryan Catanzaro
  • Publication number: 20190297326
    Abstract: A neural network architecture is disclosed for performing video frame prediction using a sequence of video frames and corresponding pairwise optical flows. The neural network processes the sequence of video frames and optical flows utilizing three-dimensional convolution operations, where time (or multiple video frames in the sequence of video frames) provides the third dimension in addition to the two-dimensional pixel space of the video frames. The neural network generates a set of parameters used to predict a next video frame in the sequence of video frames by sampling a previous video frame utilizing spatially-displaced convolution operations. In one embodiment, the set of parameters includes a displacement vector and at least one convolution kernel per pixel. Generating a pixel value in the next video frame includes applying the convolution kernel to a corresponding patch of pixels in the previous video frame based on the displacement vector.
    Type: Application
    Filed: March 21, 2019
    Publication date: September 26, 2019
    Inventors: Fitsum A. Reda, Guilin Liu, Kevin Shih, Robert Kirby, Jonathan Barker, David Tarjan, Andrew Tao, Bryan Catanzaro
  • Patent number: 10332509
    Abstract: Embodiments of end-to-end deep learning systems and methods are disclosed to recognize speech of vastly different languages, such as English or Mandarin Chinese. In embodiments, the entire pipelines of hand-engineered components are replaced with neural networks, and the end-to-end learning allows handling a diverse variety of speech including noisy environments, accents, and different languages. Using a trained embodiment and an embodiment of a batch dispatch technique with GPUs in a data center, an end-to-end deep learning system can be inexpensively deployed in an online setting, delivering low latency when serving users at scale.
    Type: Grant
    Filed: November 21, 2016
    Date of Patent: June 25, 2019
    Assignee: Baidu USA, LLC
    Inventors: Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Erich Elsen, Jesse Engel, Christopher Fougner, Xu Han, Awni Hannun, Ryan Prenger, Sanjeev Satheesh, Shubhabrata Sengupta, Dani Yogatama, Chong Wang, Jun Zhan, Zhenyao Zhu, Dario Amodei
  • Patent number: 10319374
    Abstract: Embodiments of end-to-end deep learning systems and methods are disclosed to recognize speech of vastly different languages, such as English or Mandarin Chinese. In embodiments, the entire pipelines of hand-engineered components are replaced with neural networks, and the end-to-end learning allows handling a diverse variety of speech including noisy environments, accents, and different languages. Using a trained embodiment and an embodiment of a batch dispatch technique with GPUs in a data center, an end-to-end deep learning system can be inexpensively deployed in an online setting, delivering low latency when serving users at scale.
    Type: Grant
    Filed: November 21, 2016
    Date of Patent: June 11, 2019
    Assignee: Baidu USA, LLC
    Inventors: Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Erich Elsen, Jesse Engel, Christopher Fougner, Xu Han, Awni Hannun, Ryan Prenger, Sanjeev Satheesh, Shubhabrata Sengupta, Dani Yogatama, Chong Wang, Jun Zhan, Zhenyao Zhu, Dario Amodei
  • Patent number: 10223333
    Abstract: In one embodiment of the present invention a convolution engine configures a parallel processing pipeline to perform multi-convolution operations. More specifically, the convolution engine configures the parallel processing pipeline to independently generate and process individual image tiles. In operation, for each image tile, the pipeline calculates source locations included in an input image batch. Notably, the source locations reflect the contribution of the image tile to an output tile of an output matrix—the result of the multi-convolution operation. Subsequently, the pipeline copies data from the source locations to the image tile. Similarly, the pipeline copies data from a filter stack to a filter tile. The pipeline then performs matrix multiplication operations between the image tile and the filter tile to generate data included in the corresponding output tile. To optimize both on-chip memory usage and execution time, the pipeline creates each image tile in on-chip memory as-needed.
    Type: Grant
    Filed: August 27, 2015
    Date of Patent: March 5, 2019
    Assignee: NVIDIA CORPORATION
    Inventors: Sharanyan Chetlur, Bryan Catanzaro
  • Publication number: 20170169326
    Abstract: Systems and methods for a multi-core optimized Recurrent Neural Network (RNN) architecture are disclosed. The various architectures affect communication and synchronization operations according to the Multi-Bulk-Synchronous-Parallel (MBSP) model for a given processor. The resulting family of network architectures, referred to as MBSP-RNNs, perform similarly to a conventional RNNs having the same number of parameters, but are substantially more efficient when mapped onto a modern general purpose processor. Due to the large gain in computational efficiency, for a fixed computational budget, MBSP-RNNs outperform RNNs at applications such as end-to-end speech recognition.
    Type: Application
    Filed: April 5, 2016
    Publication date: June 15, 2017
    Applicant: Baidu USA LLC
    Inventors: Gregory Diamos, Awni Hannun, Bryan Catanzaro, Dario Amodei, Erich Elsen, Jesse Engel, Shubhabrata Sengupta
  • Publication number: 20170148433
    Abstract: Embodiments of end-to-end deep learning systems and methods are disclosed to recognize speech of vastly different languages, such as English or Mandarin Chinese. In embodiments, the entire pipelines of hand-engineered components are replaced with neural networks, and the end-to-end learning allows handling a diverse variety of speech including noisy environments, accents, and different languages. Using a trained embodiment and an embodiment of a batch dispatch technique with GPUs in a data center, an end-to-end deep learning system can be inexpensively deployed in an online setting, delivering low latency when serving users at scale.
    Type: Application
    Filed: November 21, 2016
    Publication date: May 25, 2017
    Applicant: Baidu USA LLC
    Inventors: Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Erich Elsen, Jesse Engel, Christopher Fougner, Xu Han, Awni Hannun, Ryan Prenger, Sanjeev Satheesh, Shubhabrata Sengupta, Dani Yogatama, Chong Wang, Jun Zhan, Zhenyao Zhu, Dario Amodei
  • Publication number: 20170148431
    Abstract: Embodiments of end-to-end deep learning systems and methods are disclosed to recognize speech of vastly different languages, such as English or Mandarin Chinese. In embodiments, the entire pipelines of hand-engineered components are replaced with neural networks, and the end-to-end learning allows handling a diverse variety of speech including noisy environments, accents, and different languages. Using a trained embodiment and an embodiment of a batch dispatch technique with GPUs in a data center, an end-to-end deep learning system can be inexpensively deployed in an online setting, delivering low latency when serving users at scale.
    Type: Application
    Filed: November 21, 2016
    Publication date: May 25, 2017
    Applicant: Baidu USA LLC
    Inventors: Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Erich Elsen, Jesse Engel, Christopher Fougner, Xu Han, Awni Hannun, Ryan Prenger, Sanjeev Satheesh, Shubhabrata Sengupta, Dani Yogatama, Chong Wang, Jun Zhan, Zhenyao Zhu, Dario Amodei
  • Publication number: 20170068889
    Abstract: Disclosed are systems and methods that implement efficient engines for computation-intensive tasks such as neural network deployment. Various embodiments of the invention provide for high-throughput batching that increases throughput of streaming data in high-traffic applications, such as real-time speech transcription. In embodiments, throughput is increased by dynamically assembling into batches and processing together user requests that randomly arrive at unknown timing such that not all the data is present at once at the time of batching. Some embodiments allow for performing steaming classification using pre-processing. The gains in performance allow for more efficient use of a compute engine and drastically reduce the cost of deploying large neural networks at scale, while meeting strict application requirements and adding relatively little computational latency so as to maintain a satisfactory application experience.
    Type: Application
    Filed: July 13, 2016
    Publication date: March 9, 2017
    Applicant: Baidu USA LLC
    Inventors: Christopher Fougner, Bryan Catanzaro
  • Publication number: 20160171974
    Abstract: Presented herein are embodiments of state-of-the-art speech recognition systems developed using end-to-end deep learning. In embodiments, the model architecture is significantly simpler than traditional speech systems, which rely on laboriously engineered processing pipelines; these traditional systems also tend to perform poorly when used in noisy environments. In contrast, embodiments of the system do not need hand-designed components to model background noise, reverberation, or speaker variation, but instead directly learn a function that is robust to such effects. A phoneme dictionary, nor even the concept of a “phoneme,” is needed. Embodiments include a well-optimized recurrent neural network (RNN) training system that can use multiple GPUs, as well as a set of novel data synthesis techniques that allows for a large amount of varied data for training to be efficiently obtained.
    Type: Application
    Filed: June 9, 2015
    Publication date: June 16, 2016
    Applicant: BAIDU USA LLC
    Inventors: Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Gregory Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubhabrata Sengupta, Adam Coates, Andrew Y. Ng
  • Publication number: 20160062947
    Abstract: In one embodiment of the present invention a convolution engine configures a parallel processing pipeline to perform multi-convolution operations. More specifically, the convolution engine configures the parallel processing pipeline to independently generate and process individual image tiles. In operation, for each image tile, the pipeline calculates source locations included in an input image batch. Notably, the source locations reflect the contribution of the image tile to an output tile of an output matrix—the result of the multi-convolution operation. Subsequently, the pipeline copies data from the source locations to the image tile. Similarly, the pipeline copies data from a filter stack to a filter tile. The pipeline then performs matrix multiplication operations between the image tile and the filter tile to generate data included in the corresponding output tile. To optimize both on-chip memory usage and execution time, the pipeline creates each image tile in on-chip memory as-needed.
    Type: Application
    Filed: August 27, 2015
    Publication date: March 3, 2016
    Inventors: Sharanyan CHETLUR, Bryan CATANZARO
  • Patent number: 8275805
    Abstract: A method of decreasing a total computation time for a visual simulation loop includes sharing a common data structure across each phase of the visual simulation loop by adapting the common data structure to a requirement for each particular phase prior to performing a computation for that particular phase.
    Type: Grant
    Filed: March 26, 2010
    Date of Patent: September 25, 2012
    Assignee: Intel Corporation
    Inventors: Jatin Chhugani, Bryan Catanzaro, Sanjeev Kumar, Changkyu Kim, Nadathur Rajagopalan Satish
  • Publication number: 20110238680
    Abstract: A method of decreasing a total computation time for a visual simulation loop includes sharing a common data structure across each phase of the visual simulation loop by adapting the common data structure to a requirement for each particular phase prior to performing a computation for that particular phase.
    Type: Application
    Filed: March 26, 2010
    Publication date: September 29, 2011
    Inventors: Jatin Chhugani, Bryan Catanzaro, Sanjeev Kumar, Changkyu Kim, Nadathur Rajagopalan Satish