Patents by Inventor William Rae
William Rae has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).
-
Publication number: 20240046103Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input that is a sequence to generate a network output. In one aspect, one of the methods includes, for each particular sequence of layer inputs: for each attention layer in the neural network: maintaining episodic memory data; maintaining compressed memory data; receiving a layer input to be processed by the attention layer; and applying an attention mechanism over (i) the compressed representation in the compressed memory data for the layer, (ii) the hidden states in the episodic memory data for the layer, and (iii) the respective hidden state at each of the plurality of input positions in the particular network input to generate a respective activation for each input position in the layer input.Type: ApplicationFiled: October 12, 2023Publication date: February 8, 2024Inventors: Jack William Rae, Anna Potapenko, Timothy Paul Lillicrap
-
Patent number: 11836596Abstract: A system including one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to implement a memory and memory-based neural network is described. The memory is configured to store a respective memory vector at each of a plurality of memory locations in the memory. The memory-based neural network is configured to: at each of a plurality of time steps: receive an input; determine an update to the memory, wherein determining the update comprising applying an attention mechanism over the memory vectors in the memory and the received input; update the memory using the determined update to the memory; and generate an output for the current time step using the updated memory.Type: GrantFiled: November 30, 2020Date of Patent: December 5, 2023Assignee: DeepMind Technologies LimitedInventors: Mike Chrzanowski, Jack William Rae, Ryan Faulkner, Theophane Guillaume Weber, David Nunes Raposo, Adam Anthony Santoro
-
Patent number: 11829884Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input that is a sequence to generate a network output. In one aspect, one of the methods includes, for each particular sequence of layer inputs: for each attention layer in the neural network: maintaining episodic memory data; maintaining compressed memory data; receiving a layer input to be processed by the attention layer; and applying an attention mechanism over (i) the compressed representation in the compressed memory data for the layer, (ii) the hidden states in the episodic memory data for the layer, and (iii) the respective hidden state at each of the plurality of input positions in the particular network input to generate a respective activation for each input position in the layer input.Type: GrantFiled: September 25, 2020Date of Patent: November 28, 2023Assignee: DeepMind Technologies LimitedInventors: Jack William Rae, Anna Potapenko, Timothy Paul Lillicrap
-
Publication number: 20230124177Abstract: A computer-implemented method of training a neural network. The method comprises repeatedly determining a forward-pass set of network parameters by selecting a first sub-set of parameters of the neural network and setting all other parameters of the forward-pass set of network parameters to zero. The method then processes a training data item using the neural network in accordance with the forward-pass set of network parameters to generate a neural network output, determines a value of an objective function from the neural network output and the training data item, selects a second sub-set of network parameters, determines a backward-pass set of network parameters comprising the first and second sub-sets of parameters, and updates parameters corresponding to the backward-pass set of network parameters using a gradient estimate determined from the value of the objective function.Type: ApplicationFiled: June 4, 2021Publication date: April 20, 2023Inventors: Siddhant Madhu Jayakumar, Razvan Pascanu, Jack William Rae, Simon Osindero, Erich Konrad Elsen
-
Publication number: 20230061411Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for selecting actions to be performed by an agent to interact with an environment using an action selection neural network. In one aspect, a method comprises, at each time step in a sequence of time steps: generating a current representation of a state of a task being performed by the agent in the environment as of the current time step as a sequence of data elements; autoregressively generating a sequence of data elements representing a current action to be performed by the agent at the current time step; and after autoregressively generating the sequence of data elements representing the current action, causing the agent to perform the current action at the current time step.Type: ApplicationFiled: August 24, 2021Publication date: March 2, 2023Inventors: Tom Erez, Alexander Novikov, Emilio Parisotto, Jack William Rae, Konrad Zolna, Misha Man Ray Denil, Joao Ferdinando Gomes de Freitas, Oriol Vinyals, Scott Ellison Reed, Sergio Gomez, Ashley Deloris Edwards, Jacob Bruce, Gabriel Barth-Maron
-
Publication number: 20220366218Abstract: A system including an attention neural network that is configured to receive an input sequence and to process the input sequence to generate an output is described. The attention neural network includes: an attention block configured to receive a query input, a key input, and a value input that are derived from an attention block input. The attention block includes an attention neural network layer configured to: receive an attention layer input derived from the query input, the key input, and the value input, and apply an attention mechanism to the query input, the key input, and the value input to generate an attention layer output for the attention neural network layer; and a gating neural network layer configured to apply a gating mechanism to the attention block input and the attention layer output of the attention neural network layer to generate a gated attention output.Type: ApplicationFiled: September 7, 2020Publication date: November 17, 2022Inventors: Emilio Parisotto, Hasuk Song, Jack William Rae, Siddhant Madhu Jayakumar, Maxwell Elliot Jaderberg, Razvan Pascanu, Caglar Gulcehre
-
Publication number: 20220180147Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for implementing associative memory. In one aspect a system comprises an associative memory neural network to process an input to generate an output that defines an energy corresponding to the input. A reading subsystem retrieves stored information from the associative memory neural network. The reading subsystem performs operations including receiving a given, i.e. query, input and retrieving a data element from the associative memory neural network that is associated with the given input. The retrieving is performed by iteratively adjusting the given input using the associative memory neural network.Type: ApplicationFiled: May 19, 2020Publication date: June 9, 2022Inventors: Sergey Bartunov, Jack William Rae, Timothy Paul Lillicrap, Simon Osindero
-
Patent number: 11302446Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting future adverse health events using neural networks. One of the methods includes receiving electronic health record data for a patient; generating, from the electronic health record data, an input sequence comprising a respective feature representation at each of a plurality of time window time steps, comprising, for each time window time step: determining, for each of the possible numerical features, whether the numerical feature occurred during the time window; and generating, for each of the possible numerical features, one or more presence features that identify whether the numerical feature occurred during the time window; and processing the input sequence using a neural network to generate a neural network output that characterizes a predicted likelihood that an adverse health event will occur to the patient.Type: GrantFiled: November 13, 2019Date of Patent: April 12, 2022Assignee: Google LLCInventors: Nenad Tomasev, Xavier Glorot, Jack William Rae, Michal Zielinski, Anne Mottram, Harry Askham, Andre Saraiva Nobre Dos Santos, Clemens Ludwig Meyer, Suman Ravuri, Ivan Protsyuk, Trevor Back, Joseph R. Ledsam, Shakir Mohamed
-
Patent number: 11151443Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for augmenting neural networks with an external memory. One of the systems includes a sparse memory access subsystem that is configured to perform operations comprising generating a sparse set of reading weights that includes a respective reading weight for each of the plurality of locations in the external memory using the read key, reading data from the plurality of locations in the external memory in accordance with the sparse set of reading weights, generating a set of writing weights that includes a respective writing weight for each of the plurality of locations in the external memory, and writing the write vector to the plurality of locations in the external memory in accordance with the writing weights.Type: GrantFiled: February 3, 2017Date of Patent: October 19, 2021Assignee: DeepMind Technologies LimitedInventors: Ivo Danihelka, Gregory Duncan Wayne, Fu-min Wang, Edward Thomas Grefenstette, Jack William Rae, Alexander Benjamin Graves, Timothy Paul Lillicrap, Timothy James Alexander Harley, Jonathan James Hunt
-
Publication number: 20210150314Abstract: A system for compressed data storage using a neural network. The system comprises a memory comprising a plurality of memory locations configured to store data; a query neural network configured to process a representation of an input data item to generate a query; an immutable key data store comprising key data for indexing the plurality of memory locations; an addressing system configured to process the key data and the query to generate a weighting associated with the plurality of memory locations; a memory read system configured to generate output memory data from the memory based upon the generated weighting associated with the plurality of memory locations and the data stored at the plurality of memory locations; and a memory write system configured to write received write data to the memory based upon the generated weighting associated with the plurality of memory locations.Type: ApplicationFiled: November 23, 2020Publication date: May 20, 2021Inventors: Jack William Rae, Timothy Paul Lillicrap, Sergey Bartunov
-
Publication number: 20210089829Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input that is a sequence to generate a network output. In one aspect, one of the methods includes, for each particular sequence of layer inputs: for each attention layer in the neural network: maintaining episodic memory data; maintaining compressed memory data; receiving a layer input to be processed by the attention layer; and applying an attention mechanism over (i) the compressed representation in the compressed memory data for the layer, (ii) the hidden states in the episodic memory data for the layer, and (iii) the respective hidden state at each of the plurality of input positions in the particular network input to generate a respective activation for each input position in the layer input.Type: ApplicationFiled: September 25, 2020Publication date: March 25, 2021Inventors: Jack William Rae, Anna Potapenko, Timothy Paul Lillicrap
-
Publication number: 20210081795Abstract: A system including one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to implement a memory and memory-based neural network is described. The memory is configured to store a respective memory vector at each of a plurality of memory locations in the memory. The memory-based neural network is configured to: at each of a plurality of time steps: receive an input; determine an update to the memory, wherein determining the update comprising applying an attention mechanism over the memory vectors in the memory and the received input; update the memory using the determined update to the memory; and generate an output for the current time step using the updated memory.Type: ApplicationFiled: November 30, 2020Publication date: March 18, 2021Inventors: Mike Chrzanowski, Jack William Rae, Ryan Faulkner, Theophane Guillaume Weber, David Nunes Raposo, Adam Anthony Santoro
-
Patent number: 10853725Abstract: A system including one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to implement a memory and memory-based neural network is described. The memory is configured to store a respective memory vector at each of a plurality of memory locations in the memory. The memory-based neural network is configured to: at each of a plurality of time steps: receive an input; determine an update to the memory, wherein determining the update comprising applying an attention mechanism over the memory vectors in the memory and the received input; update the memory using the determined update to the memory; and generate an output for the current time step using the updated memory.Type: GrantFiled: May 17, 2019Date of Patent: December 1, 2020Assignee: DeepMind Technologies LimitedInventors: Mike Chrzanowski, Jack William Rae, Ryan Faulkner, Theophane Guillaume Weber, David Nunes Raposo, Adam Anthony Santoro
-
Patent number: 10846588Abstract: A system for compressed data storage using a neural network. The system comprises a memory comprising a plurality of memory locations configured to store data; a query neural network configured to process a representation of an input data item to generate a query; an immutable key data store comprising key data for indexing the plurality of memory locations; an addressing system configured to process the key data and the query to generate a weighting associated with the plurality of memory locations; a memory read system configured to generate output memory data from the memory based upon the generated weighting associated with the plurality of memory locations and the data stored at the plurality of memory locations; and a memory write system configured to write received write data to the memory based upon the generated weighting associated with the plurality of memory locations.Type: GrantFiled: September 27, 2019Date of Patent: November 24, 2020Assignee: DeepMind Technologies LimitedInventors: Jack William Rae, Timothy Paul Lillicrap, Sergey Bartunov
-
Publication number: 20200285940Abstract: There is described herein a computer-implemented method of processing an input data item. The method comprises processing the input data item using a parametric model to generate output data, wherein the parametric model comprises a first sub-model and a second sub-model. The processing comprises processing, by the first sub-model, the input data to generate a query data item, retrieving, from a memory storing data point-value pairs, at least one data point-value pair based upon the query data item and modifying weights of the second sub-model based upon the retrieved at least one data point-value pair. The output data is then generated based upon the modified second sub-model.Type: ApplicationFiled: October 29, 2018Publication date: September 10, 2020Inventors: Pablo Sprechmann, Siddhant Jayakumar, Jack William Rae, Alexander Pritzel, Adrià Puigdomènech Badia, Oriol Vinyals, Razvan Pascanu, Charles Blundell
-
Publication number: 20200152333Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting future adverse health events using neural networks. One of the methods includes receiving electronic health record data for a patient; generating, from the electronic health record data, an input sequence comprising a respective feature representation at each of a plurality of time window time steps, comprising, for each time window time step: determining, for each of the possible numerical features, whether the numerical feature occurred during the time window; and generating, for each of the possible numerical features, one or more presence features that identify whether the numerical feature occurred during the time window; and processing the input sequence using a neural network to generate a neural network output that characterizes a predicted likelihood that an adverse health event will occur to the patient.Type: ApplicationFiled: November 13, 2019Publication date: May 14, 2020Inventors: Nenad Tomasev, Xavier Glorot, Jack William Rae, Michal Zielinski, Anne Mottram, Harry Askham, Andre Saraiva Nobre Dos Santos, Clemens Ludwig Meyer, Suman Ravuri, Ivan Protsyuk, Trevor Back, Joseph R. Ledsam, Shakir Mohamed
-
Publication number: 20200104677Abstract: A system for compressed data storage using a neural network. The system comprises a memory comprising a plurality of memory locations configured to store data; a query neural network configured to process a representation of an input data item to generate a query; an immutable key data store comprising key data for indexing the plurality of memory locations; an addressing system configured to process the key data and the query to generate a weighting associated with the plurality of memory locations; a memory read system configured to generate output memory data from the memory based upon the generated weighting associated with the plurality of memory locations and the data stored at the plurality of memory locations; and a memory write system configured to write received write data to the memory based upon the generated weighting associated with the plurality of memory locations.Type: ApplicationFiled: September 27, 2019Publication date: April 2, 2020Inventors: Jack William Rae, Timothy Paul Lillicrap, Sergey Bartunov
-
Publication number: 20190354858Abstract: A system including one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to implement a memory and memory-based neural network is described. The memory is configured to store a respective memory vector at each of a plurality of memory locations in the memory. The memory-based neural network is configured to: at each of a plurality of time steps: receive an input; determine an update to the memory, wherein determining the update comprising applying an attention mechanism over the memory vectors in the memory and the received input; update the memory using the determined update to the memory; and generate an output for the current time step using the updated memory.Type: ApplicationFiled: May 17, 2019Publication date: November 21, 2019Inventors: Mike Chrzanowski, Jack William Rae, Ryan Faulkner, Theophane Guillaume Weber, David Nunes Raposo, Adam Anthony Santoro
-
Publication number: 20170228638Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for augmenting neural networks with an external memory. One of the systems includes a sparse memory access subsystem that is configured to perform operations comprising generating a sparse set of reading weights that includes a respective reading weight for each of the plurality of locations in the external memory using the read key, reading data from the plurality of locations in the external memory in accordance with the sparse set of reading weights, generating a set of writing weights that includes a respective writing weight for each of the plurality of locations in the external memory, and writing the write vector to the plurality of locations in the external memory in accordance with the writing weights.Type: ApplicationFiled: February 3, 2017Publication date: August 10, 2017Inventors: Ivo Danihelka, Gregory Duncan Wayne, Fu-min Wang, Edward Thomas Grefenstette, Jack William Rae, Alexander Benjamin Graves, Timothy Paul Lillicrap, Timothy James Alexander Harley, Jonathan James Hunt
-
Publication number: 20090169317Abstract: A hole saw holder comprising an elongate body having a first end and a second end, wherein the first end has an external thread (6) of sufficient length t receive a plurality of hole saws (16), a support formation (8) at the end of the external thread for supporting one or more hole saws received by the external thread, and a central axial drill bit receiving bore (4) which extends throughout the length of the elongate body. The hole saw holder can used to conveniently store a plurality of hole saws and a hole saw arbour. T hole saw holder can be used with two hole saws of different diameters to enlarge a hole which has already been drilled in a surface.Type: ApplicationFiled: October 24, 2006Publication date: July 2, 2009Inventor: Scott William Rae