Abstract: The present disclosure provides a computer-implemented method for determining an optimized list of sets of hyperparameter values for application to an additional machine learning task. The method includes obtaining data describing a plurality of different machine learning tasks. The method includes obtaining a plurality of candidate sets of hyperparameter values. The method includes determining an ordered list of sets of hyperparameters selected from the plurality of candidate sets of hyperparameter values, wherein the ordered list of sets of hyperparameters minimizes an aggregate loss over the plurality of different machine learning tasks. The method includes storing the ordered list of sets of hyperparameters for use in training an additional machine learning model to perform an additional machine learning task.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training neural networks using learned optimizers. One of method is for training a neural network layer comprising a plurality of network parameters having a plurality of dimensions each having a plurality of indices, the method comprising: maintaining a set of values corresponding to respective sets of indices of each dimension, each value representing a measure of central tendency of past gradients of the network parameters having an index in the dimension that is in the set of indices; performing a training step to obtain a new gradient for each network parameter; updating each set of values using the new gradients; and for each network parameter: generating an input from the updated sets of values; processing the input using an optimizer neural network to generate an output defining an update for the network parameter; and applying the update.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing optimization using an optimizer neural network. One of the methods includes for each optimizer network parameter, randomly sampling a perturbation value; generating a plurality of sets of candidate values for the optimizer network parameters, for each set of candidate values of the optimizer network parameters: determining a respective loss value representing a performance of the optimizer neural network in updating one or more sets of inner parameters in accordance with the set of candidate of values of the optimizer network parameters; and updating the current values of the optimizer network parameters based on the loss values for the plurality of sets of candidate values of the optimizer network parameters.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network. One of the methods includes performing, using a plurality of training examples, a training step to obtain respective gradients of a loss function with respect to each of the parameters in the parameter tensors; obtaining a validation loss for a plurality of validation examples that are different from the plurality of training examples generating an optimizer input from at least the respective gradients and the validation loss; processing the optimizer input using an optimizer neural network to generate an output defining a respective update for each of the parameters in the parameter tensors of the neural network; and for each of the parameters in the parameter tensors, applying the respective update to a current value of the parameter to generate an updated value for the parameter.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network. One of the methods includes training the neural network for one or more training steps in accordance with a current learning rate; generating a training dynamics observation characterizing the training of the trainee neural network on the one or more training steps; providing the training dynamics observation as input to a controller neural network that is configured to process the training dynamics observation to generate a controller output that defines an updated learning rate; obtaining as output from the controller neural network the controller output that defines the updated learning rate; and setting the learning rate to the updated learning rate.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a base neural network that generates numeric representations of network inputs.

Abstract: A computer-implemented method for training an optimizer neural network having optimizer parameters is described. The optimizer neural network is configured to generate an output that defines updated values of target parameters of a target neural network in a set of target neural networks during training of the target neural network to perform one or more neural network tasks. The optimizer neural network is associated with an outer loss function that measures how well the optimizer neural network generates updated values of target parameters for the target neural network.