Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media for training machine learning models. One method includes obtaining a machine learning model, wherein the machine learning model comprises one or more model parameters, and the machine learning model is trained using gradient descent techniques to optimize an objective function; determining an update rule for the model parameters using a recurrent neural network (RNN); and applying a determined update rule for a final time step in a sequence of multiple time steps to the model parameters.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media for training machine learning models. One method includes obtaining a machine learning model, wherein the machine learning model comprises one or more model parameters, and the machine learning model is trained using gradient descent techniques to optimize an objective function; determining an update rule for the model parameters using a recurrent neural network (RNN); and applying a determined update rule for a final time step in a sequence of multiple time steps to the model parameters.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media for training machine learning models. One method includes obtaining a machine learning model, wherein the machine learning model comprises one or more model parameters, and the machine learning model is trained using gradient descent techniques to optimize an objective function; determining an update rule for the model parameters using a recurrent neural network (RNN); and applying a determined update rule for a final time step in a sequence of multiple time steps to the model parameters.

Abstract: Systems, methods, devices, and other techniques are disclosed for using an augmented neural network system to generate a sequence of outputs from a sequence of inputs. An augmented neural network system can include a controller neural network, a hierarchical external memory, and a memory access subsystem. The controller neural network receives a neural network input at each of a series of time steps processes the neural network input to generate a memory key for the time step. The external memory includes a set of memory nodes arranged as a binary tree. To provide an interface between the controller neural network and the external memory, the system includes a memory access subsystem that is configured to, for each of the series of time steps, perform one or more operations to generate a respective output for the time step. The capacity of the neural network system to account for long-range dependencies in input sequences may be extended.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media for training machine learning models. One method includes obtaining a machine learning model, wherein the machine learning model comprises one or more model parameters, and the machine learning model is trained using gradient descent techniques to optimize an objective function; determining an update rule for the model parameters using a recurrent neural network (RNN); and applying a determined update rule for a final time step in a sequence of multiple time steps to the model parameters.

Abstract: Systems, methods, devices, and other techniques are disclosed for using an augmented neural network system to generate a sequence of outputs from a sequence of inputs. An augmented neural network system can include a controller neural network, a hierarchical external memory, and a memory access subsystem. The controller neural network receives a neural network input at each of a series of time steps processes the neural network input to generate a memory key for the time step. The external memory includes a set of memory nodes arranged as a binary tree. To provide an interface between the controller neural network and the external memory, the system includes a memory access subsystem that is configured to, for each of the series of time steps, perform one or more operations to generate a respective output for the time step. The capacity of the neural network system to account for long-range dependencies in input sequences may be extended.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a system output from a system input. In one aspect, a neural network system includes a memory storing a set of register vectors and data defining modules, wherein each module is a respective function that takes as input one or more first vectors and outputs a second vector. The system also includes a controller neural network configured to receive a neural network input for each time step and process the neural network input to generate a neural network output. The system further includes a subsystem configured to determine inputs to each of the modules, process the input to the module to generate a respective module output, determine updated values for the register vectors, and generate a neural network input for the next time step from the updated values of the register vectors.

Abstract: A floating-point unit and a method of identifying exception cases in a floating-point unit. In one embodiment, the floating-point unit includes: (1) a floating-point computation circuit having a normal path and an exception path and operable to execute an operation on an operand and (2) a decision circuit associated with the normal path and the exception path and configured to employ a flush-to-zero mode of the floating-point unit to determine which one of the normal path and the exception path is appropriate for carrying out the operation on the operand.

Abstract: A floating-point unit and a method of identifying exception cases in a floating-point unit. In one embodiment, the floating-point unit includes: (1) a floating-point computation circuit having a normal path and an exception path and operable to execute an operation on an operand and (2) a decision circuit associated with the normal path and the exception path and configured to employ a flush-to-zero mode of the floating-point unit to determine which one of the normal path and the exception path is appropriate for carrying out the operation on the operand.